uberAgent 7 Preview: EDR & Antivirus Performance Monitoring

While we’re finalizing version 7.0 of our digital employee experience monitoring & endpoint security analytics products uberAgent UXM and uberAgent ESA, let’s take a look at another cool new feature: protected process performance monitoring (applies to EDR, antivirus, and other types of security products).

What Are Protected Processes?

Endpoint security products need to protect themselves from tampering and deactivation by malware. One of the techniques to help with such protection is built into the Windows OS: protected processes light (PPL).

The PPL model represents a security boundary. When a vendor chooses to run its security agent as a protected process light, other user-mode processes cannot modify or terminate the agent any more – this includes elevated processes and processes running as SYSTEM (access right details).

Pros & Cons of Protected Processes Light (PPL)

Pros

• Security hardening.

Cons

When a security product makes use of the PPL security model, some operations become more difficult, e.g.:

• Administrators cannot stop or restart the product’s system services.
• Monitoring software cannot query the OS for CPU or RAM performance metrics of the product’s processes.

Process Protection Levels

A process’ protection level is stored in the Protection member of the EPROCESS kernel structure. The Protection member is a PS_PROTECTION structure containing the following three components (docs):

• Type
• Audit (unused)
• Signer

Type can be one of the following:

• PsProtectedTypeNone: the process is not protected.
• PsProtectedTypeProtectedLight: the process is a protected process light (PPL).
• PsProtectedTypeProtected: the process is a protected process.

The actual protection level is a combination of Type and Signer. The following table lists the valid protection levels from high to low power:

Protection level	Type	Signer
PsProtectedSignerWinSystem	Protected	WinSystem
PsProtectedSignerWinTcb	Protected	WinTcb
PsProtectedSignerWinTcb-Light	Protected light	WinTcb
PsProtectedSignerWindows	Protected	Windows
PsProtectedSignerWindows-Light	Protected light	Windows
PsProtectedSignerLsa-Light	Protected light	LSA
PsProtectedSignerAntimalware-Light	Protected light	Antimalware
PsProtectedSignerAuthenticode	Protected	Authenticode
PsProtectedSignerAuthenticode-Light	Protected light	Authenticode
PsProtectedSignerNone	None	None

How to Monitor the CPU & RAM Performance of Protected Processes?

The Windows security model does not allow regular user-mode monitoring products to query the CPU or RAM usage of a protected process. Only processes with equal or higher protection level can access protected processes, even if it’s only to collect performance data. There are two ways to solve this:

1. Configure the monitoring product to run as a protected process itself, at a protection level that is higher than the processes it needs to monitor. This is impractical given that some OS processes like wininit.exe or lsass.exe run at very high protection levels (PsProtectedSignerWinTcb-Light and PsProtectedSignerLsa-Light, respectively).
2. Collect process performance data from a kernel-mode driver. This is exactly what uberAgent 7.0 does and which enables it to collect performance metrics for all processes independent of their protection level.

Example: Windows Defender CPU & RAM Performance

The following screenshots demonstrate uberAgent’s new capability to monitor the performance of security agents and antivirus products. We’re using Windows Defender as an example. Typical for antivirus products, Defender’s main process MsMpEng.exe has the following protection properties:

• Type: Protected light
• Signer: Antimalware
• Protection level: PsProtectedSignerAntimalware-Light

Please note that the screenshots were taken in a lab environment and show average values of multiple endpoints.

Windows Defender CPU & RAM Usage Over Time

As you can see in the screenshot below, the baseline RAM usage is approximately 200 MB. Interestingly, the RAM usage increases significantly to almost 600 MB over a period of several hours once per day. These increases do not coincide with CPU spikes (blue line).

Windows Defender CPU Usage per Process

The average CPU usage of Windows Defender is very low. The most active process is MsMpEng.exe. All Defender processes combined generate an average CPU load of less than 0.2%.

Windows Defender RAM Usage per Process

The average RAM usage of Windows Defender is a little more than 300 MB. Again, the most active process is MsMpEng.exe.

Windows Defender CPU & Disk IO per Process

As with CPU and RAM usage, MsMpEng.exe generates most of the disk IO. The value of 36,712 in the screenshot below was measured over a period of three full days. Disk operations per second (IOPS) are, therefore, low.

Windows Defender Network Connections

Defender talks to multiple hosts in the Azure Cloud. Actual hostnames and IP addresses will vary between geographic regions.

Resources

• Windows Internals, 7th edition, part 1
• itm4n: Do You Really Know About LSA Protection (RunAsPPL)?
• Microsoft docs: Protecting anti-malware services