uberAgent https://uberagent.com Citrix, VMware & Windows monitoring on Splunk, optimized for XenApp/XenDesktop, Horizon View, RDS, VDI and physical PCs Tue, 13 Feb 2018 07:55:46 +0000 en-US hourly 1 https://wordpress.org/?v=4.9.4 Monitoring RDP Session Hijacking https://uberagent.com/blog/monitoring-rdp-session-hijacking/ https://uberagent.com/blog/monitoring-rdp-session-hijacking/#respond Thu, 18 Jan 2018 15:48:13 +0000 https://uberagent.com/?p=3540 In addition to all the fuss around Spectre and Meltdown, there are several other security flaws which are worth mentioning. One of these is RDP session hijacking. How RDP Session Hijacking Works In his excellent article, Kevin Beaumont explains in great detail what RDP session hijacking is and how to do it. Here is a...

The post Monitoring RDP Session Hijacking appeared first on uberAgent.

]]>
In addition to all the fuss around Spectre and Meltdown, there are several other security flaws which are worth mentioning. One of these is RDP session hijacking.

How RDP Session Hijacking Works

In his excellent article, Kevin Beaumont explains in great detail what RDP session hijacking is and how to do it.
Here is a summary:

  • Windows lets you connect to other user’s RDP sessions via tscon.exe. You typically need the other user’s password for this.
  • If you run tscon.exe with SYSTEM privileges, you can connect to any other RDP session without a password.
  • There are several ways to get SYSTEM privileges if you have administrator permissions
    • PSEXEC from the Sysinternals suite
    • Create and start a service
    • Use a scheduled task
  • There are even RDP backdoor methods to get SYSTEM privileges. Mimikatz is probably the best-known example.
  • It is hard to monitor because there isn’t a specific Windows event log entry

How to Monitor Session Hijacking

Eric from XenAppBlog.com asked if it is possible to monitor RDP session hijacking with uberAgent. It is. Here is one way to do it.

Requirements

uberAgent uses Splunk to visualize collected data from your endpoints. As Splunk is licensed by daily indexed traffic, we have a strong incentive for keeping the amount of indexed data as small as possible. Therefore, we do not store command line information about each started process by default. One can enable this via the setting EnableExtendedInfo=true in our configuration. Keep in mind that your daily indexed traffic will increase.

The Simplest Solution

As stated above, you have to run tscon.exe with SYSTEM privileges to connect to another RDP session without the need to enter the user’s password. That is easy to accomplish with uberAgent and Splunk.

First, we have a look at all process startups and choose the fields host, SessionID, and ProcCmdline.

| pivot uberAgent Process_ProcessStartup
   values(ProcCmdline) as "Command line(s)" 
   splitrow 
      host as "Machine name"
   splitrow 
      SessionID as "Session ID"

In the second step, we filter the events for the process name tscon.exe and the process user SYSTEM. We do some sorting, too.

| pivot uberAgent Process_ProcessStartup
   values(ProcCmdline) as "Command line(s)" 
   splitrow 
      host as "Machine name"
   splitrow       
      SessionID as "Session ID"
   filter ProcName is "tscon.exe"
   filter ProcUser is "SYSTEM" 
| eval sortfield=lower('Machine name')
| sort limit=0 sortfield
| table
   "Machine name"
   "Session ID"
   "Command line(s)"

The result may look like this.

Goal achieved! But we only see session IDs, not usernames. It would be laborious to manually figure out the right username for the ID.

The More Sophisticated Solution

Not only does uberAgent give you details like name or id for the tscon.exe process but also for its parent. With this information, you can see how tscon.exe was started.

| pivot uberAgent Process_ProcessStartup
   values(ProcCmdline) as "Command line(s)"   
   splitrow 
      host as "Machine name"
   splitrow 
      SessionID as "Session ID"
   splitrow 
      ProcParentName as "Parent process name"
   filter ProcName is "tscon.exe"
   filter ProcUser is "SYSTEM" 
| eval sortfield=lower('Machine name')
| sort limit=0 sortfield
| table
   "Machine name"
   "Session ID"
   "Parent process name"   
   "Command line(s)"

Additionally, uberAgent can combine different datasets. By combining process startup data with session detail data, we can find out the usernames of the attacking and target sessions.

| pivot uberAgent Process_ProcessStartup
   latest(_time) as Time
   splitrow 
      host as "Machine name"
   splitrow 
      SessionID as "Attacking session ID"
   splitrow 
      ProcParentName as "Attacking session process name"
   splitrow
      ProcCmdline as "Command line"
   filter ProcName is "tscon.exe"
   filter ProcUser is "SYSTEM"
| eval Time = strftime (strptime (Time, "%Y-%m-%dT%H:%M:%S.%Q%z"), "%Y-%m-%d %H:%M:%S")
| rex field="Command line" "(?<temp>\d+)"
| rename temp as "Target session ID"
| join type=outer "Machine name" "Target session ID"
[
   | pivot uberAgent Session_SessionDetail_Users 
      latest(User) as "Target session username"
      splitrow 
         host as "Machine name"
      splitrow 
         SessionID as "Target session ID"
   | fields + "Target session username" "Machine name" "Target session ID"
]
| join type=outer "Machine name" "Attacking session ID"
[
   | pivot uberAgent Session_SessionDetail_Users 
      latest(User) as "Attacking session username"
      splitrow 
         host as "Machine name"
      splitrow 
         SessionID as "Attacking session ID"
   | fields + "Attacking session username" "Machine name" "Attacking session ID" 
]
| eval sortfield=lower('Machine name')
| sort limit=0 sortfield
| table
   Time
   "Machine name"
   "Attacking session ID"
   "Attacking session username"
   "Attacking session process name"   
   "Command line"   
   "Target session ID"
   "Target session username"

The result may look like this.

Conclusion

RDP session hijacking really is a thing. It could do massive damage, cannot be prevented and is hard to monitor. At least with the monitoring part, uberAgent may help.

The post Monitoring RDP Session Hijacking appeared first on uberAgent.

]]>
https://uberagent.com/blog/monitoring-rdp-session-hijacking/feed/ 0
Creating a Foreground Application Usage Report https://uberagent.com/blog/creating-foreground-application-usage-report/ https://uberagent.com/blog/creating-foreground-application-usage-report/#respond Wed, 13 Dec 2017 13:39:34 +0000 https://uberagent.com/?p=3422 Would you like to find out which applications your users spend most of their time with? Which application versions are being run? How long ago applications were last used? You have come to the right place. Application Inventory, Usage – and the Foreground App Of the hundreds of applications that may be installed on a...

The post Creating a Foreground Application Usage Report appeared first on uberAgent.

]]>
Would you like to find out which applications your users spend most of their time with? Which application versions are being run? How long ago applications were last used? You have come to the right place.

Application Inventory, Usage – and the Foreground App

Of the hundreds of applications that may be installed on a machine, only a few dozen are typically running concurrently. Of those applications that are running there is always one that has a special role: it receives the keyboard input. That is the active foreground application. That is the application the user is currently interacting with.

uberAgent determines application inventory and usage on several different levels. It regularly runs an inventory scan, identifying all applications that are installed. uberAgent also monitors all running applications, for which it collects detailed information, e.g., the version number. Last, but not least, uberAgent reports which application currently is the foreground app.

In this post, we are going to make use of the latter two information sets to create a report that lists all running applications along with their versions, when they were last seen and how often each application was the foreground app.

uberAgent’s Pre-Built Dashboards

Before we dive into custom searches let’s take a look at what uberAgent’s dashboards have to offer. The Application Usage dashboard, for example, shows which applications I used most recently:

It seems I am spending a lot of time – more than two thirds – in the browser. When I don’t, I am using an editor, keeping the screen locked, playing video or organizing files, to name the most relevant applications for yours truly.

The above is only a fraction of what is available out of the box, of course. uberAgent’s dashboards let you dive right in to figure out how applications are used over time, which application versions are used by how many people, and much more.

A Custom Application Usage Report

Requirements

This blog post was inspired by a customer who asked how to create an application usage report that also showed for each app whether it was being run in the foreground. We have tweaked those requirements slightly and are going to present a Splunk search that generates a report with the following fields:

  • Application name
  • Application version(s)
  • Application last run date
  • Application in foreground (%)

The Splunk Search

uberAgent collects detailed information on all running applications every 30 seconds in the ProcessDetail sourcetype. The currently active foreground application, however, is determined per user session in the sourcetype SessionDetail.

Translated to the accelerated data model which we typically use for searches due to the vastly superior performance, that corresponds to the datasets Process_ProcessDetail and Session_SessionDetail_Users.

The search we are going to build consists of three parts:

  1. For each running application, get us the name, version(s) and last run time
  2. For each of the resulting applications, determine how often they were in the foreground
  3. Format the output nicely

Without further ado, here is the search:

| pivot uberAgent Process_ProcessDetail
   latest(_time) as LastRun
   values(AppVersion) as AppVersions
   splitrow
      AppName
| eval LastRun=strftime (strptime (LastRun, "%Y-%m-%dT%H:%M:%S.%Q%z"), "%Y-%m-%d %H:%M:%S")
| join type=outer AppName
[
   | pivot uberAgent Session_SessionDetail_Users
      count(SessionFgAppName) as CountSessionFgAppName
      splitrow
         SessionFgAppName
   | eventstats sum(CountSessionFgAppName) as EventCount
   | eval "Application in foreground (%)"= round(CountSessionFgAppName / EventCount * 100, 1)
   | rename SessionFgAppName as AppName
   | fields AppName "Application in foreground (%)"
]
| eval sortfield = lower('AppName')
| sort limit=0 sortfield
| rename
   AppName as "Application name"
   AppVersions as "Application version(s)"
   LastRun as "Application last run date"
| table
  "Application name"
  "Application version(s)"
  "Application last run date"
  "Application in foreground (%)"

The Output

The resulting table can easily be exported in various formats or scheduled for regular automated report generation:

The post Creating a Foreground Application Usage Report appeared first on uberAgent.

]]>
https://uberagent.com/blog/creating-foreground-application-usage-report/feed/ 0
Getting Started with uberAgent & Splunk https://uberagent.com/blog/getting-started-uberagent-splunk/ https://uberagent.com/blog/getting-started-uberagent-splunk/#comments Wed, 06 Dec 2017 15:26:56 +0000 https://uberagent.com/?p=3381 A new hire’s journey into the world of user experience monitoring. Hi, my name is Dominik Britz. I’m from Cologne, Germany and the latest team member of vast limits, the uberAgent company. I’d like to introduce myself, tell you why I’m here and what I’m going to do. I started my career as a consultant...

The post Getting Started with uberAgent & Splunk appeared first on uberAgent.

]]>
A new hire’s journey into the world of user experience monitoring.

Hi, my name is Dominik Britz. I’m from Cologne, Germany and the latest team member of vast limits, the uberAgent company. I’d like to introduce myself, tell you why I’m here and what I’m going to do.

dominik

I started my career as a consultant in the end user computing space, building Citrix, Microsoft and VMware implementations from small to big. My focus was the automation of these deployments with PowerShell (you should always automate your environments, of course! Only automation guarantees quality and reproducibility). I also worked as a Citrix administrator for some time.

A few weeks ago I started as a customer success engineer at vast limits. I’ll do support, partner onboarding, webinars and pretty much anything that will make our customers’ and partners’ implementations successful.

This Is Why I Am at vast limits

Two things have to fit for me when I start a new career path: the people and the technology. As I know most of my new colleagues from my former consultant life, they aren’t “new” colleagues per se. They are all awesome and brilliant people, and I’m thrilled to work with them, again.

On to the second point: technology. uberAgent is such a great product! It collects all the relevant metrics from your endpoints without drowning you in numbers. And that’s not what marketing is saying, that is what our customers are telling me and what I am hearing from our partners as I am getting in touch with them in my first days at vast limits.

Learning Splunk as an EUC Guy

We use Splunk to visualize the data collected by uberAgent (we have experimental support for Elasticsearch and Microsoft OMS Log Analytics, too). As an end user computing guy, I had not worked with Splunk before. It is better known in the security space, e.g., to collect firewall logs. However, it can do so much more, and one learns it fast. My PowerShell obsession helped me a lot because the principle of piping was known to me.

Here is an example of how piping works. You want to know the top five hosts where Mozilla Firefox is crashing.

Top five hosts were firefox.exe is crashing

You start a search in the data collected by uberAgent (index=uberAgent) but you only want application error information (sourcetype=uberAgent:Application:Errors). In addition, you only want data for Mozilla Firefox (ProcName=firefox.exe).

In the next step, you pipe the resulting data from the previous commands to the Splunk top command, which displays the most common values of a field. You limit the output to five results (limit=5) and group it by the host field. The timeframe for the search is configurable with the time picker next to the search field.

Easy, isn’t it?

That was my two cents about my first weeks at vast limits. I’m happy to get in touch with you to talk about user experience monitoring with uberAgent and to discuss, how to make our product even better.

The post Getting Started with uberAgent & Splunk appeared first on uberAgent.

]]>
https://uberagent.com/blog/getting-started-uberagent-splunk/feed/ 1
Listing Users who are Launching Applications Often https://uberagent.com/blog/listing-users-launching-applications-often/ https://uberagent.com/blog/listing-users-launching-applications-often/#respond Wed, 25 Oct 2017 12:54:15 +0000 https://uberagent.com/?p=3216 One of our customers wanted to identify users who launch a specific executable more often than n times in a given time range. That is easy to accomplish, given that uberAgent monitors process starts. This article describes multiple ways to query for that information. The following walkthroughs illustrate how to identify processes by executable or...

The post Listing Users who are Launching Applications Often appeared first on uberAgent.

]]>
One of our customers wanted to identify users who launch a specific executable more often than n times in a given time range. That is easy to accomplish, given that uberAgent monitors process starts. This article describes multiple ways to query for that information.

The following walkthroughs illustrate how to identify processes by executable or by application name, counting process/application launches per user for any time range. We will start with a “traditional” Splunk SPL search, followed by a second implementation that makes use of the uberAgent data model.

Splunk SPL Search

Let’s build the Splunk SPL search step by step.

Step 1

We start with all events from uberAgent’s index.

Note: to facilitate changing the index name used by uberAgent, all our dashboards make use of the uberAgent_index macro which contains the actual index name. The macro is defined in macros.conf of the uberAgent searchhead app.

index=`uberAgent_index`

Step 2

We filter for the process startup sourcetype which contains one event per started process.

Note: The documentation of uberAgent’s sourcetypes and fields can be found here.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup

Step 3

We ignore processes started by SYSTEM, LOCAL SERVICE and NETWORK SERVICE.

Note: The pseudo-users sys, lvc and nvc are defined in the lookup table systemusers.csv of the uberAgent searchhead app. They are auto-expanded to the proper user names SYSTEM, LOCAL SERVICE and NETWORK SERVICE in uberAgent’s data model.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc

Step 4

We add a filter for the name of the process we are interested in, Winword.exe in this example.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe

Step 5

We count the number of (start) events per user.

Note: The only purpose of adding the field ProcName to the stats command is to make it part of the results table, too.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe | stats count as Starts by ProcName ProcUser

Step 6

We only keep users with more than five starts in the results list.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe | stats count as Starts by ProcName ProcUser | where Starts > 5

Step 7

We rename fields to make them look nicer.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe | stats count as Starts by ProcName ProcUser | where Starts > 5 | rename ProcUser as User ProcName as Process

Step 8

We sort the results so that the user with the highest number of starts is listed first. The 0 in the sort command ensures that the output is not truncated after the 10,000th result.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe | stats count as Starts by ProcName ProcUser | where Starts > 5 | rename ProcUser as User ProcName as Process | sort 0 -Starts

The Result

This is what the final Splunk SPL search for users with more than five starts of Winword.exe looks like. The screenshot below shows the search being run over the past 30 days. In practice, you would adjust the time range to any relevant time interval.

Accelerated Data Model Search

uberAgent comes with an accelerated data model. Searching an accelerated data model is a lot faster than searching the underlying index (by “a lot” we mean at least 50x), but requires a different search syntax based on the pivot or tstats commands. We are using pivot because of the easier syntax compared to tstats.

In this second example, we demonstrate how to search for starts of a “modern” UWP app, specifically the weather app that is part of Windows. Most UWP apps cannot be identified by process name – which is simply backgroundTaskHost.exe. Luckily uberAgent determines the real app name automatically.

Step 1

We start with a count of all process starts.

Note: The documentation for the Splunk pivot command can be found here.

| pivot uberAgent Process_ProcessStartup count(Process_ProcessStartup) as Starts

Step 2

We filter for the weather app.

Note: An easy way to identify the name of the weather app is to dig around with a search like the following: index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup AppName=*Weather*

| pivot uberAgent Process_ProcessStartup count(Process_ProcessStartup) as Starts filter AppName is “Microsoft.BingWeather”

Step 3

We split by user so that we get a count of process starts per user (renaming the ProcUser field to User in the process).

| pivot uberAgent Process_ProcessStartup count(Process_ProcessStartup) as Starts filter AppName is “Microsoft.BingWeather” splitrow ProcUser as User

Step 4

We only keep users with more than five starts in the results list. We also sort the results so that the user with the highest number of starts is listed first. The 0 in the sort command ensures that the output is not truncated after the 10,000th result.

| pivot uberAgent Process_ProcessStartup count(Process_ProcessStartup) as Starts filter AppName is “Microsoft.BingWeather” splitrow ProcUser as User | where Starts > 5 | sort 0 -Starts

Step 5

We add the application name as a row to the results table.

| pivot uberAgent Process_ProcessStartup count(Process_ProcessStartup) as Starts latest(AppName) as Application filter AppName is “Microsoft.BingWeather” splitrow ProcUser as User | where Starts > 5 | sort 0 -Starts | table Application User Starts

The Result

The resulting output is very similar to the first example above: a table with the application, the users and the number of starts that can easily be exported to CSV or otherwise be processed further.

The post Listing Users who are Launching Applications Often appeared first on uberAgent.

]]>
https://uberagent.com/blog/listing-users-launching-applications-often/feed/ 0
uberAgent 4.2: Splunk 7.0, Simplified Operations, Elasticsearch Improvements https://uberagent.com/blog/uberagent-4-2-splunk-7-0-simplified-operations-elasticsearch-improvements/ https://uberagent.com/blog/uberagent-4-2-splunk-7-0-simplified-operations-elasticsearch-improvements/#respond Mon, 16 Oct 2017 20:19:31 +0000 https://uberagent.com/?p=3190 We are happy to announce the newest version of our user experience and application performance monitoring product. uberAgent 4.2 brings new features and improvements for any kind of device. For a full list of all improvements and bugfixes please consult the change log. As always, upgrading is highly recommended (instructions). Splunk 7.0 uberAgent 4.2 adds...

The post uberAgent 4.2: Splunk 7.0, Simplified Operations, Elasticsearch Improvements appeared first on uberAgent.

]]>
We are happy to announce the newest version of our user experience and application performance monitoring product. uberAgent 4.2 brings new features and improvements for any kind of device.

For a full list of all improvements and bugfixes please consult the change log. As always, upgrading is highly recommended (instructions).

Splunk 7.0

uberAgent 4.2 adds full support for the latest version 7.0 of Splunk Enterprise and Splunk Cloud.

Central License File Management

Before uberAgent 4.2 license files had to be distributed along with the agent installation package. That was easy in environments with a single master image file, like Citrix PVS. Customers with tens of thousands of laptops, however, found it a lot harder to roll out new licenses to their endpoints.

uberAgent 4.2 greatly simplifies license file management. All you have to do is set up a file share with read permissions for computer accounts, drop your uberAgent license there and configure the share path in uberAgent’s new configuration option LicenseFilePath. Endpoints will periodically check LicenseFilePath for new licenses. If any are found that are not yet cached locally, the new license files are copied to the local license cache directory.

There is no need for the configured license file path to be available all of the time. Endpoints always use the local license cache for license validation. Of course, there is no requirement to set up a central license file path, either. If deploying the license file along with the agent installation package works for you: great, you are all set.

Elasticsearch Improvements

uberAgent 4.2 comes with important improvements for our Elasticsearch customers.

Security

uberAgent now supports X-Pack authentication. This ensures that only validated endpoints can send data to the backend.

Ingest Pipelines

The ability to modify and enrich data before the indexing stage adds a great deal of flexibility to the Elasticsearch platform. Customers can new reference their ingest pipelines from uberAgent so that uberAgent data traverses the pipeline before being indexed.

More Improvements

Custom Scripts

uberAgent’s custom script execution engine, capable of running user-supplied scripts in machine or user context, now supports Elasticsearch as well as Splunk HTTP Event Collector (HEC) backends.

Monitor Inventory

uberAgent’s hardware inventory now collects information about the physical monitors attached to the machine along with their respective resolutions.

About uberAgent

uberAgent is a Windows user experience analytics and application performance monitoring product. Its highlights include detailed information about boot and logon duration (showing why and when boots/logons are slow), application unresponsiveness detection, network reliability drilldowns, process startup duration, application usage metering, browser performance per website and remoting protocol insights.

About vast limits

vast limits GmbH is the company behind uberAgent, the innovative user experience and application performance monitoring product. Our customer list includes organizations from industries like finance, healthcare, professional services and education, ranging from medium-sized businesses to global enterprises. vast limits’ network of qualified solution partners ensures best-in-class service and support anywhere in the world.

Our founder, Helge Klein, is an experienced consultant and developer who architected the user profile management product whose successor is now available as Citrix Profile Management. In 2009 Helge received the Citrix Technology Professional (CTP) award, in 2011 he was nominated a Microsoft Most Valuable Professional (MVP), in 2014 he was a Splunk Revolution Award Winner, in 2015 he became a VMware vExpert. Helge frequently presents at conferences and user group events like Citrix Synergy, Splunk .conf, BriForum or E2EVC.

The post uberAgent 4.2: Splunk 7.0, Simplified Operations, Elasticsearch Improvements appeared first on uberAgent.

]]>
https://uberagent.com/blog/uberagent-4-2-splunk-7-0-simplified-operations-elasticsearch-improvements/feed/ 0
User Session Script Collecting Custom Metrics (Citrix ICA RTT) https://uberagent.com/blog/user-session-script-collecting-custom-metrics-citrix-ica-rtt/ https://uberagent.com/blog/user-session-script-collecting-custom-metrics-citrix-ica-rtt/#comments Wed, 13 Sep 2017 13:57:51 +0000 https://uberagent.com/?p=3064 uberAgent is often used in conjunction with Universal Forwarder, Splunk’s generic agent that monitors logs and collects the output from custom scripts. The combination of the two agents is a powerful one, as it allows customers to add any metric they require to uberAgent’s already rich dataset. However, running two agents side by side has...

The post User Session Script Collecting Custom Metrics (Citrix ICA RTT) appeared first on uberAgent.

]]>
uberAgent is often used in conjunction with Universal Forwarder, Splunk’s generic agent that monitors logs and collects the output from custom scripts. The combination of the two agents is a powerful one, as it allows customers to add any metric they require to uberAgent’s already rich dataset.

However, running two agents side by side has drawbacks, too: the administrative overhead increases as do the hardware resources required on the endpoints.

As of uberAgent 4.1 there exists an attractive alternative: what started out with the intention of providing a way of collecting custom metrics from individual user sessions turned into a generic script execution engine. It runs any type of script at any desired interval, either per machine or per user session.

Overview: How to Configure Custom Script Execution

The execution of custom scripts is handled by uberAgent’s endpoint agent. Scripts are configured as part of timer stanzas in uberAgent’s configuration. The following lists the relevant configuration options:

#   Setting name: Name
#   Description: Arbitrary name for the timer.
#   Valid values: any string
#   Default: empty
#   Required: yes
#
#   Setting name: Interval
#   Description: How long to wait before collecting data again. Unit: milliseconds.
#   Valid values: any number
#   Default: [none]
#   Required: yes
#
#   Setting name: Script
#   Description: Run a script once or periodically, depending on the configured Interval (0 = run only once). The script's output to stdout is sent to Splunk, each line as a new event. Can be specified more than once per timer.
#   Valid values: Any valid command line, optionally including command line parameters.
#   Default: empty
#   Required: no
#
#   Setting name: ScriptContext
#   Description: The user context to run a script in.
#   Valid values: Session0AsSystem | UserSessionAsSystem | UserSessionAsUser
#   Default: Session0AsSystem
#   Required: no

Please note that the uberAgent service on the endpoint is running in the context of LocalSystem, so the referenced script must be accessible by and executable to the LocalSystem account. This is particularly relevant when running scripts stored on a network file share.

The Script

You can use any script written in your preferred scripting language, e.g. PowerShell or VBScript. Our agent will capture all script output sent to standard output (stdout), i.e. printed to the console. Every line of output is sent as its own event to the backend. Script output must be formatted as key-value pairs (e.g. key=value).

Please keep in mind that any data collected in addition to our default dataset has an impact on the generated data volume. Running custom scripts generates additional load on the endpoint the amount of which depends on the executed process (e.g. powershell.exe or cscript.exe) and the underlying data source. Especially Windows Management Instrumentation (WMI) can cause a significant load.

Additionally, please choose an appropriate timer interval for your script. Data that does not change often, like inventory information, probably only needs to be collected once a day whereas volatile metrics like network throughput might have to be collected once per minute.

uberAgent does not manage the deployment process of custom scripts to the endpoints. Please feel free to use either your existing software distribution system or Splunk’s Deployment Server.

The Script Context

Custom scripts can be executed in three different contexts:

  • Session0AsSystem: the script runs in session 0 as LocalSystem
  • UserSessionAsSystem: the script runs in every interactive user session as LocalSystem
  • UserSessionAsUser: the script runs in every interactive user session as the user logged on to the session

The script context is configured per timer. Of course, you can configure multiple timers for independent execution of different scripts.

Example: Querying WMI Data Using a PowerShell Script

This example shows how to collect the ICA RTT metric in every user session by way of a custom script running every 30s. The ICA protocol round trip time (RTT) is an important metric supplementing uberAgent’s remoting protocol latency in Citrix XenApp / XenDesktop environments. You can find a detailed description of the ICA RTT metric here.

The following PowerShell script queries the ICA RTT as a property of a WMI class:

$Citrix_Euem_RoundTrip = Get-WmiObject -Namespace root\Citrix\euem -Class Citrix_Euem_RoundTrip
$CurrentSessionID = [System.Diagnostics.Process]::GetCurrentProcess().SessionId

foreach ($Session in $Citrix_Euem_RoundTrip)
{
   if ($Session.SessionID -eq $CurrentSessionID)
   {
      [Hashtable]$Output = @{
         'RoundtripTime'=$($Session.RoundtripTime) 
         'SessionID'=$($Session.SessionID)
         'SessionUser'=[System.Security.Principal.WindowsIdentity]::GetCurrent().Name
      }
      Write-Output $($Output.Keys.ForEach({"$_=$($Output.$_)"}) -join ' ')
   }
}

All properties (SessionUser, SessionID and RoundtripTime) are written to stdout as key-value pairs:

SessionUser=AD\timmtest02 SessionID=5 RoundtripTime=28

This script (named Citrix_Euem_RoundTrip.ps1) is located in the %ProgramFiles%\vast limits\uberAgent\Scripts directory on the endpoint.

I configured the script execution as follows in uberAgent’s configuration:

############################################
# Timer 10
############################################
[Timer]
Name           = PowerShell Citrix Euem RoundTrip
Interval       = 30000
Script         = powershell.exe -executionpolicy bypass -file "C:\Program Files\vast limits\uberAgent\Scripts\Citrix_Euem_RoundTrip.ps1"
ScriptContext  = UserSessionAsUser

As you can see, this script is executed every 30 seconds (30,000 milliseconds) as user inside every interactive session.

The Splunk search results look as follows:

By default, all collected data is sent to the Splunk index uberagent. The Splunk sourcetype used for the script’s output is a concatenation of uberAgent:Script: and the timer name specified in uberAgent’s configuration.

The post User Session Script Collecting Custom Metrics (Citrix ICA RTT) appeared first on uberAgent.

]]>
https://uberagent.com/blog/user-session-script-collecting-custom-metrics-citrix-ica-rtt/feed/ 2
uberAgent 4.1: Blue Screens & Hangs, Custom Scripts, Optimized Config https://uberagent.com/blog/blue-screens-hangs-custom-scripts-optimized-config-data-volume/ https://uberagent.com/blog/blue-screens-hangs-custom-scripts-optimized-config-data-volume/#respond Wed, 05 Jul 2017 17:55:35 +0000 https://uberagent.com/?p=2816 We are happy to announce the newest version of our user experience and application performance monitoring product. uberAgent 4.1 brings exciting new features and improvements for any kind of device. For a full list of all improvements and bugfixes please consult the change log. As always, upgrading is highly recommended (instructions). Blue Screens, Power Losses,...

The post uberAgent 4.1: Blue Screens & Hangs, Custom Scripts, Optimized Config appeared first on uberAgent.

]]>
We are happy to announce the newest version of our user experience and application performance monitoring product. uberAgent 4.1 brings exciting new features and improvements for any kind of device.

For a full list of all improvements and bugfixes please consult the change log. As always, upgrading is highly recommended (instructions).

Blue Screens, Power Losses, Hard Hangs

Stability and reliability are important cornerstones for good user experience. Machines that crash, hang or reboot randomly cause data loss and user frustration. That is why uberAgent’s new blue screen & hang reporting is so important. It shows you far more than just which kinds of stop errors happened how often. It lets you drill down by site, machine type or OS version to get to the root cause of where the errors are coming from.

Various conditions can lead to a stop error: if the system detects an unrecoverable error it collects rudimentary information and then halts execution in a controlled manner displaying a blue screen. Sometimes, however, that is not possible. The machine might reboot seemingly randomly or just freeze (“hard hang”). Those types of errors typically indicate hardware failure or power loss. Sometimes users perform a hard power off by pressing and holding the power button for four seconds, trying to fix an error condition.

uberAgent detects all of the above, providing visibility and helping to improve productivity.

Running Custom Scripts in User Context

Splunk’s Universal Forwarder is very good at running scripts while capturing output and sending it to Splunk for indexing. It does, however, have a limitation that is sometimes relevant in end-user computing: it cannot run scripts per user session.

uberAgent can. It executes custom scripts either per machine or per user session, optionally running them in user context. This provides the unique ability to run queries or perform actions as the user. This works very well on RDS hosts like Citrix XenApp, of course. Needless to say that flexible scheduling options are available.

More Improvements

Configuration Optimized for Data Volume

uberAgent now comes with an alternative configuration that has been optimized for low data volume, minimizing Splunk licensing costs by approximately 10x (details).

Standby/Resume Duration

The data collected when the system switches to a low-power state like suspend or hibernation is now much more detailed. It not only includes the duration of the standby or resume operation, but also the effective power state and the reason for waking up. And it works on Windows 10, too.

Citrix Site Monitoring Performance

We have been able to significantly reduce CPU usage of the Citrix Site Monitoring feature to a point where uberAgent’s footprint on the Delivery Controller is almost negligible (details).

About uberAgent

uberAgent is a Windows user experience analytics and application performance monitoring product. Its highlights include detailed information about boot and logon duration (showing why and when boots/logons are slow), application unresponsiveness detection, network reliability drilldowns, process startup duration, application usage metering, browser performance per website and remoting protocol insights.

About vast limits

vast limits GmbH is the company behind uberAgent, the innovative user experience and application performance monitoring product. Our customer list includes organizations from industries like finance, healthcare, professional services and education, ranging from medium-sized businesses to global enterprises. vast limits’ network of qualified solution partners ensures best-in class service and support around the globe.

Our founder, Helge Klein, is an experienced consultant and developer who architected the user profile management product whose successor is now available as Citrix Profile Management. In 2009 Helge received the Citrix Technology Professional (CTP) award, in 2011 he was nominated a Microsoft Most Valuable Professional (MVP), in 2014 he was a Splunk Revolution Award Winner, in 2015 he became a VMware vExpert. Helge frequently presents at conferences and user group events like Citrix Synergy, Splunk .conf, BriForum or E2EVC.

The post uberAgent 4.1: Blue Screens & Hangs, Custom Scripts, Optimized Config appeared first on uberAgent.

]]>
https://uberagent.com/blog/blue-screens-hangs-custom-scripts-optimized-config-data-volume/feed/ 0
Identifying Console Window Popups by Monitoring Process Starts https://uberagent.com/blog/identifying-console-window-popups-monitoring-process-starts/ https://uberagent.com/blog/identifying-console-window-popups-monitoring-process-starts/#respond Tue, 06 Jun 2017 14:31:00 +0000 https://uberagent.com/?p=2717 I recently noticed a console window pop up for a fraction of a second on my desktop. About an hour later, it happened again. Random windows appearing out of nowhere is not something I like to see on my machines. Read on to learn how I investigated the issue. Monitoring Process Starts Obviously, something was...

The post Identifying Console Window Popups by Monitoring Process Starts appeared first on uberAgent.

]]>
I recently noticed a console window pop up for a fraction of a second on my desktop. About an hour later, it happened again. Random windows appearing out of nowhere is not something I like to see on my machines. Read on to learn how I investigated the issue.

Monitoring Process Starts

Obviously, something was starting processes on my machine at a certain schedule. I wanted for find out what that was.

uberAgent, our user experience and application performance monitoring product, detects any process start on a monitored endpoint. Process starts are recorded with the exact timestamp, process name, PID and the name of the parent process.

Detecting Console Process Starts

Identifying starts of console processes is easy if you have uberAgent. For every console process there is a console host child process called conhost.exe. All we need to do is look for conhost.exe processes and then list the names of the parent process. The following Splunk search does just that:

index=uberagent sourcetype=uberAgent:Process:ProcessStartup host=hkx1c ProcName=conhost.exe ProcParentName=* | table _time ProcParentName

Above search looks for process start events on my machine where the process name is conhost.exe. For every matching process start it lists the time the process was started and the name of the parent process. Please note that uberAgent’s configuration setting EnableExtendedInfo needs to be enabled in order for this to work. A listing of all sourcetypes and field names collected by uberAgent can be found here.

Finding the Console Process

When that offending console window popped up the next time I took note of the time and then ran the search from above. The result looked like this:

The console process started at around the time I noticed the window pop up was officebackgroundtaskhandler.exe. A while later I repeated the process, and again officebackgroundtaskhandler.exe turned up. So what’s the matter whith this process apparently belonging to the Microsoft Office suite?

Why is officebackgroundtaskhandler.exe Flashing Console Windows?

A quick search for officebackgroundtaskhandler.exe led me to this Microsoft Answers post. Apparently the flashing console window is a know bug that will be fixed in a future update.

Phew, looks like this one is harmless and going to go away on its own. Unfortunately that is not always the case. Make sure you understand the processes running on your machines. uberAgent can help with that.

The post Identifying Console Window Popups by Monitoring Process Starts appeared first on uberAgent.

]]>
https://uberagent.com/blog/identifying-console-window-popups-monitoring-process-starts/feed/ 0
Heard Splunk is Expensive? What About $17 / user / year? https://uberagent.com/blog/splunk-expensive/ https://uberagent.com/blog/splunk-expensive/#respond Wed, 31 May 2017 13:02:05 +0000 https://uberagent.com/?p=2665 uberAgent’s primary backend for data storage, search and visualization is Splunk, the leading big data platform. Some people, especially when they are new to Splunk, are worried about the additional cost. Apparently Splunk is rumored to be expensive. Let me debunk that myth for the uberAgent use case. No Database Required Many products use a...

The post Heard Splunk is Expensive? What About $17 / user / year? appeared first on uberAgent.

]]>
uberAgent’s primary backend for data storage, search and visualization is Splunk, the leading big data platform. Some people, especially when they are new to Splunk, are worried about the additional cost. Apparently Splunk is rumored to be expensive. Let me debunk that myth for the uberAgent use case.

No Database Required

Many products use a database for data storage which can incur significant additional costs, especially when clustered. This is not the case with uberAgent. There is no database, nor are there any other required infrastructure components. uberAgent only needs Splunk. This is the first good news.

The second good news is that Splunk is actually very affordable when used with uberAgent. Let’s take a look at an example to find out exactly what the numbers may look like.

Splunk Pricing Example Calculation

Acme Corporation need visibility into user experience and application performance on their laptops. They also want to be prepared for the upcoming migration to Windows 10. They choose uberAgent as their end-user computing monitoring and analytics product. Before they install the agent on all 1,000 of their Windows machines they need to determine which Splunk license is right for them.

Splunk Enterprise annual pricing

They know from our data volume calculation guidelines that the average data volume per client is 15 MB / day. Multiplied by 1,000 machines this amounts to a daily volume of 15 GB to be indexed by Splunk.

Splunk licenses are based on the amount of new data added to the Splunk index per day. A 15 GB license is considered a very small license; customers that are using Splunk for security or log analytics often have licenses in the range of terabytes per day.

Splunk’s website lists a price of $1,150 per GB for a yearly 15 GB license including maintenance. Divided by 1,000 users this amounts to $17.25 per user per year. Compare that number to the price of pretty much any SaaS application (e.g. Salesforce or GoToMeeting) which cost many times that per user per month.

Splunk Cloud

If you do not want to maintain Splunk servers on premises Splunk Cloud might be the right choice. Looking at the 20 GB/day tier, the annual price per user is only $20.7.

Given that Splunk manages the entire backend and even guarantees 100% availability the markup compared to on-premises Splunk Enterprise is surprisingly low (20%), making Splunk Cloud a compelling offer.

Benefits of Splunk

Splunk has a number of features and capabilities that put it far ahead of the competition.

Practically Unlimited Scalability

A single Splunk server can handle between 100 and 250 GB of incoming new data per data. When that is not sufficient just add more machines, the load is balanced automatically. Optionally configure replication if you want to be safe in case individual servers become unavailable.

Data Retention Only Limited by Disk Space

Being a big data platform, Splunk can store and search vast amounts of data. This means visibility is not cut off after a week, a month or any other arbitrary period of time. In fact, you can keep the collected data for as long as you like. Disk space is the only limiting factor.

There is also no averaging of historical data. Other products need to replace the collected data with less detailed averages after a certain time in order to keep the sizes of their databases from ballooning. This is not the case with Splunk. All collected data can be retrieved in full fidelity even years later.

Easy Custom Dashboard Creation

uberAgent comes with a rich set of dashboards that are well suited for many requirements and use cases. Whenever that is not enough, Splunk’s dashboard editor allows for easy creation of custom visualizations which can be arranged into dashboards or used in scheduled reports.

Operational Intelligence

Although uberAgent is our preferred Splunk use case 😉 there is a lot more you can do with it. Splunk’s app store lists hundreds of apps for a wide range of products and technologies. All of these can be used at the same time, and data from multiple sources can be brought together in queries and visualizations. Imagine the possibilities – metrics and events from all your relevant IT components in one place, ranging from networking appliances to end-user computing machines. That is what Splunk calls operational intelligence.

The post Heard Splunk is Expensive? What About $17 / user / year? appeared first on uberAgent.

]]>
https://uberagent.com/blog/splunk-expensive/feed/ 0
Why You Should Monitor Applications, Not Processes https://uberagent.com/blog/monitor-applications-not-processes/ https://uberagent.com/blog/monitor-applications-not-processes/#respond Thu, 04 May 2017 21:23:54 +0000 https://uberagent.com/?p=2598 One of the many things you come to love once you have used uberAgent for a little while is the fact that it is very application-centric. Applications are what really matters, not the individual processes they are comprised of. I will give you a example of that in this post. Automatic Application Identification For every...

The post Why You Should Monitor Applications, Not Processes appeared first on uberAgent.

]]>
One of the many things you come to love once you have used uberAgent for a little while is the fact that it is very application-centric. Applications are what really matters, not the individual processes they are comprised of. I will give you a example of that in this post.

Automatic Application Identification

For every running process uberAgent determines the name of the application the process is a part of. That happens automatically and works for regular Win32 applications just as well as it does for UWP apps (AKA “modern” Windows apps) or Java applications.

Why is this special? The Windows operating system has no concept of applications as logical entities. There is no API to map processes to applications or vice versa. The magic is happening right in uberAgent.

With knowledge about applications all of uberAgent’s rich performance and user experience metrics can be presented at the process level as well as on the application level. Read on for an example where that distinction is crucial.

Example: Firefox Browser Performance

When I recently analyzed browser video playback performance I was surprised by Firefox’ apparent efficiency. According to Task Manager it only used 1.1% CPU and 121 MB of RAM while playing back a 1080p movie on Netflix:

uberAgent, on the other hand, reported very different numbers: 8.3% CPU and 533 MB of RAM:

Why this difference?

Why Task Manager can be Misleading

Task Manager and most monitoring tools display performance metrics per process. This often makes things unnecessarily difficult.

For starters, Firefox uses multiple firefox.exe processes. In order to determine Firefox’ resource utilization one has to add up their numbers:

But even when combining the three firefox.exe instances there is still a missing piece:

Firefox plays protected web video by way of a plugin – and plugins are hosted in a dedicated process for reasons of security and stability. With that in mind one needs to identify instances of Firefox’ plugin container executable, plugin-container.exe and include their performance numbers in the calculation:

As you can see in the screenshot above, plugin-container.exe adds quite significant CPU load, so one would not want to miss it.

Automating the Process

Even this rather simple example shows that coming up with accurate performance numbers for an application of interest can be quite cumbersome. The good news is: you do not have to. uberAgent does it all for you, fully automatically. Try it for yourself: downloading and installing only take a few minutes but might save you hours very quickly.

The post Why You Should Monitor Applications, Not Processes appeared first on uberAgent.

]]>
https://uberagent.com/blog/monitor-applications-not-processes/feed/ 0