One of our customers wanted to identify users who launch a specific executable more often than n times in a given time range. That is easy to accomplish, given that uberAgent monitors process starts. This article describes multiple ways to query for that information.
The following walkthroughs illustrate how to identify processes by executable or by application name, counting process/application launches per user for any time range. We will start with a “traditional” Splunk SPL search, followed by a second implementation that makes use of the uberAgent data model.
Splunk SPL Search
Let’s build the Splunk SPL search step by step.
We start with all events from uberAgent’s index.
Note: to facilitate changing the index name used by uberAgent, all our dashboards make use of the uberAgent_index macro which contains the actual index name. The macro is defined in macros.conf of the uberAgent searchhead app.
We filter for the process startup sourcetype which contains one event per started process.
Note: The documentation of uberAgent’s sourcetypes and fields can be found here.
We ignore processes started by SYSTEM, LOCAL SERVICE and NETWORK SERVICE.
Note: The pseudo-users sys, lvc and nvc are defined in the lookup table systemusers.csv of the uberAgent searchhead app. They are auto-expanded to the proper user names SYSTEM, LOCAL SERVICE and NETWORK SERVICE in uberAgent’s data model.
We add a filter for the name of the process we are interested in, Winword.exe in this example.
We count the number of (start) events per user.
Note: The only purpose of adding the field ProcName to the stats command is to make it part of the results table, too.
We only keep users with more than five starts in the results list.
We rename fields to make them look nicer.
We sort the results so that the user with the highest number of starts is listed first. The 0 in the sort command ensures that the output is not truncated after the 10,000th result.
This is what the final Splunk SPL search for users with more than five starts of Winword.exe looks like. The screenshot below shows the search being run over the past 30 days. In practice, you would adjust the time range to any relevant time interval.
Accelerated Data Model Search
uberAgent comes with an accelerated data model. Searching an accelerated data model is a lot faster than searching the underlying index (by “a lot” we mean at least 50x), but requires a different search syntax based on the pivot or tstats commands. We are using pivot because of the easier syntax compared to tstats.
In this second example, we demonstrate how to search for starts of a “modern” UWP app, specifically the weather app that is part of Windows. Most UWP apps cannot be identified by process name – which is simply backgroundTaskHost.exe. Luckily uberAgent determines the real app name automatically.
We start with a count of all process starts.
Note: The documentation for the Splunk pivot command can be found here.
We filter for the weather app.
Note: An easy way to identify the name of the weather app is to dig around with a search like the following: index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup AppName=*Weather*
We split by user so that we get a count of process starts per user (renaming the ProcUser field to User in the process).
We only keep users with more than five starts in the results list. We also sort the results so that the user with the highest number of starts is listed first. The 0 in the sort command ensures that the output is not truncated after the 10,000th result.
We add the application name as a row to the results table.
The resulting output is very similar to the first example above: a table with the application, the users and the number of starts that can easily be exported to CSV or otherwise be processed further.