Threat Detection Rules: Performance, Readability & Operations Improvements

This post provides a preview of some exciting improvements we’re making to uberAgent ESA’s Threat Detection rules. These upcoming changes increase the rule engine’s performance, enhance the readability of rule queries, and simplify day-to-day operations.

Rule Post-Processing: Handle False Positives Elegantly

False Positives Before uberAgent 7.2

Before uberAgent 7.2, dealing with false positives was a bit difficult. If a threat detection rule matched a legitimate operation, you had one of two choices:

1. Exclude the rule generating the false positives in our configuration automation script.
2. Exclude the rule generating the false positives in an ActivityMonitoringRule_Filter stanza (docs) in your uberAgent configuration.

Both of the above approaches come with the downside that the rule generating the false positives is disabled completely. If you wanted to keep it but have a few items added to a denylist, you had to follow up one of the above choices by making a modified copy of the original rule. A modified copy that, unfortunately, wasn’t updated via uberAgent’s configuration repository anymore. uberAgent 7.2 offers something much better.

Post-Processing Stanzas

With the new post-processing stanza type, you can modify existing rules, either individually or in groups by event type or tag.

Consider the following default rule, which, unfortunately, also matches certain Splunk processes:

 [ThreatDetectionRule]
RuleId = bdc64095-d59a-42a2-8588-71fd9c9d9abc
RuleName = Suspicious Unsigned Dbghelp/Dbgcore DLL Loaded
EventType = Image.Load
Tag = suspicious-unsigned-dbghelp/dbgcore-dll-loaded
RiskScore = 75
Annotation = {" mitre_attack" : [" T1003.001" ]}
Query = ((Image.Path like r" %\\dbghelp.dll"  or Image.Path like r" %\\dbgcore.dll" ) and Image.IsSigned == false)

To prevent the above rule from matching your Splunk processes, add the following post-processing stanza to any uberAgent configuration file:

 [ThreatDetectionRuleExtension RuleId=bdc64095-d59a-42a2-8588-71fd9c9d9abc]
Query = Rule.Result and Parent.Name != " Splunkd.exe" 

The above rule extension does what its name implies: it extends the original rule’s uAQL query with additional statements, which is ideal for adding exclusions.

One of the many cool things about the way we implemented rule post-processing is that extension stanzas needn’t apply to individual rules only; they can apply to entire sets of rules as well. The following extension stanza definition, for example, would apply to all rules matching network events:

 [ThreatDetectionRuleExtension EventType=Net.Any]
Query = Rule.Result and Parent.Name != " Splunkd.exe" 

Regex Performance

Regular expressions, the Swiss army knife for pattern matching, allow for concise specifications of expected result sets. Their versatility makes regexes a powerful ally in malware detection as well as in countless other use cases.

In previous versions of uberAgent, regular expressions were implemented as functions, i.e., regex_match(VALUE, r"PATTERN"). uberAgent 7.2 implements regexes as operators instead, e.g., VALUE regex r"PATTERN". This comes with two significant benefits that both stem from the fact that regular expressions are now compiled early while the agent configuration is read:

1. Performance: compiling regexes is costly in terms of performance. Doing it only once and then re-using the pre-compiled expression dramatically increases matching performance.
2. User-friendliness: errors in regexes are now logged early when the agent configuration is processed, right when you expect them to be. You don’t have to trigger a rule match anymore to make sure a regex is syntactically correct.

Rules Spanning Multiple Lines

Threat detection rules can become quite lengthy and complicated. When they do, you want to be able to format them so that readability is maximized. This often involves splitting the rule across multiple lines. Rules spanning multiple lines is precisely what we’re making possible with uberAgent 7.2.

Consider the following rule that ships with uberAgent 7.1:

 [ActivityMonitoringRule platform=Windows]
# Detects suspicious use of an .exe extension after a non-executable file extension like .pdf.exe, a set of spaces or underlines to cloak the executable file in spear phishing campaigns
# Author: Florian Roth (Nextron Systems), @blu3_team (idea), Nasreddine Bencherchali (Nextron Systems)
RuleId = 1cdd9a09-06c9-4769-99ff-626e2b3991b8
RuleName = Suspicious Double Extension File Execution
EventType = Process.Start
Tag = proc-start-suspicious-double-extension-file-execution
RiskScore = 100
Annotation = {" mitre_attack" : [" T1566.001" ]}
Query = (Process.Path like r" %.doc.exe"  or Process.Path like r" %.docx.exe"  or Process.Path like r" %.xls.exe"  or Process.Path like r" %.xlsx.exe"  or Process.Path like r" %.ppt.exe"  or Process.Path like r" %.pptx.exe"  or Process.Path like r" %.rtf.exe"  or Process.Path like r" %.pdf.exe"  or Process.Path like r" %.txt.exe"  or Process.Path like r" %      .exe"  or Process.Path like r" %\_\_\_\_\_\_.exe"  or Process.Path like r" %.doc.js"  or Process.Path like r" %.docx.js"  or Process.Path like r" %.xls.js"  or Process.Path like r" %.xlsx.js"  or Process.Path like r" %.ppt.js"  or Process.Path like r" %.pptx.js"  or Process.Path like r" %.rtf.js"  or Process.Path like r" %.pdf.js"  or Process.Path like r" %.txt.js" ) and (Process.CommandLine like r" %.doc.exe%"  or Process.CommandLine like r" %.docx.exe%"  or Process.CommandLine like r" %.xls.exe%"  or Process.CommandLine like r" %.xlsx.exe%"  or Process.CommandLine like r" %.ppt.exe%"  or Process.CommandLine like r" %.pptx.exe%"  or Process.CommandLine like r" %.rtf.exe%"  or Process.CommandLine like r" %.pdf.exe%"  or Process.CommandLine like r" %.txt.exe%"  or Process.CommandLine like r" %      .exe%"  or Process.CommandLine like r" %\_\_\_\_\_\_.exe%"  or Process.CommandLine like r" %.doc.js%"  or Process.CommandLine like r" %.docx.js%"  or Process.CommandLine like r" %.xls.js%"  or Process.CommandLine like r" %.xlsx.js%"  or Process.CommandLine like r" %.ppt.js%"  or Process.CommandLine like r" %.pptx.js%"  or Process.CommandLine like r" %.rtf.js%"  or Process.CommandLine like r" %.pdf.js%"  or Process.CommandLine like r" %.txt.js%" )

In uberAgent 7.2, you can improve readability significantly by splitting the rule’s query and having it span multiple lines as follows:

 [ActivityMonitoringRule platform=Windows]
# Detects suspicious use of an .exe extension after a non-executable file extension like .pdf.exe, a set of spaces or underlines to cloak the executable file in spear phishing campaigns
# Author: Florian Roth (Nextron Systems), @blu3_team (idea), Nasreddine Bencherchali (Nextron Systems)
RuleId = 1cdd9a09-06c9-4769-99ff-626e2b3991b8
RuleName = Suspicious Double Extension File Execution
EventType = Process.Start
Tag = proc-start-suspicious-double-extension-file-execution
RiskScore = 100
Annotation = {" mitre_attack" : [" T1566.001" ]}
QueryStart
   Process.Path like r" %.doc.exe"  or
   Process.Path like r" %.docx.exe"  or
   Process.Path like r" %.xls.exe"  or
   Process.Path like r" %.xlsx.exe"  or
   Process.Path like r" %.ppt.exe"  or
   Process.Path like r" %.pptx.exe"  or
   Process.Path like r" %.rtf.exe"  or
   Process.Path like r" %.pdf.exe"  or
   Process.Path like r" %.txt.exe"  or
   Process.Path like r" %      .exe"  or
   Process.Path like r" %\_\_\_\_\_\_.exe"  or
   Process.Path like r" %.doc.js"  or
   Process.Path like r" %.docx.js"  or
   Process.Path like r" %.xls.js"  or
   Process.Path like r" %.xlsx.js"  or
   Process.Path like r" %.ppt.js"  or
   Process.Path like r" %.pptx.js"  or
   Process.Path like r" %.rtf.js"  or
   Process.Path like r" %.pdf.js"  or
   Process.Path like r" %.txt.js" ) and
   (Process.CommandLine like r" %.doc.exe%"  or
   Process.CommandLine like r" %.docx.exe%"  or
   Process.CommandLine like r" %.xls.exe%"  or
   Process.CommandLine like r" %.xlsx.exe%"  or
   Process.CommandLine like r" %.ppt.exe%"  or
   Process.CommandLine like r" %.pptx.exe%"  or
   Process.CommandLine like r" %.rtf.exe%"  or
   Process.CommandLine like r" %.pdf.exe%"  or
   Process.CommandLine like r" %.txt.exe%"  or
   Process.CommandLine like r" %      .exe%"  or
   Process.CommandLine like r" %\_\_\_\_\_\_.exe%"  or
   Process.CommandLine like r" %.doc.js%"  or
   +Process.CommandLine like r" %.docx.js%"  or
   Process.CommandLine like r" %.xls.js%"  or
   Process.CommandLine like r" %.xlsx.js%"  or
   Process.CommandLine like r" %.ppt.js%"  or
   Process.CommandLine like r" %.pptx.js%"  or
   Process.CommandLine like r" %.rtf.js%"  or
   Process.CommandLine like r" %.pdf.js%"  or
   Process.CommandLine like r" %.txt.js%" 
QueryEnd