New features

• Dashboards [B461]: new Experience Score dashboard providing a high-level overview of the whole environment. Scores are also available in the dashboards Single Machine Detail, Single Application Detail, and Single User Detail.
• macOS: new macOS endpoint agent.
• Security: new product: uberAgent ESA (endpoint security analytics). It shares the binaries with the existing UXM (user experience monitoring) product but must be licensed separately.
• uAQL [B439]: uberAgent ESA includes uAQL, a powerful query language for use with ESA’s Activity Monitoring rules.
• Hashing [B280]: uberAgent ESA calculates hash values of PE files (executables). Supported hash types: SHA-1, SHA-256, MD5, ImpHash.
• Registry monitoring [B340]: the new registry monitoring feature extends ESA’s Activity Monitoring to registry events.
• Network communication [B209]: the new network monitoring driver adds features like jitter, packet loss, source address, and latency accuracy.
• Browsers [B231]: configurable URL detail in web app monitoring. Specify the level of URL detail individually per site and collect the path (up to a given depth or full) and query parameters (either all params or those specified).
• Browsers [B329]: support for Microsoft Edge (Chromium).
• Splunk CIM [B476]: uberAgent UXM and ESA offer compliance with the Splunk Common Information Model, supporting multiple CIM data models (CIM version 4.13.0 or later required).
• Configuration [B279]: the configuration file now supports includes from other files.
• Configuration [B290]: the configuration file now supports reusable blocks.

Improvements

• Service [B376]: new architecture guarantees type safety for all sourcetype fields.
• Service [I262]: improved the IP address to DNS name lookup and the internal name/address cache. The new algorithm does not perform network DNS queries any more, is more efficient and provides better results. It also favors CNAMEs over A/AAAA names.
• Service [I89][I258]: AD user/host tags now support multi-value attributes, numbers, GUIDs and SIDs.
• Service [B466]: the metrics ProcessStartup, ProcessStop, and ActivityMonitoring are now timer-based. This makes it possible to specify the data transfer frequency.
• Application errors [B429]: uberAgent now records the type of application hang events (requires Windows 10 1909).
• Configuration [B428]: the application ID to name mapping data collection interval is now configurable via the new AppNameIdMapping timer metric.
• Configuration [B420]: new configuration setting ConfigFlags for altering specific aspects of uberAgent’s functionality.
• Configuration [I407]: an additional configuration file can be included via Group Policy. This can be used to set a base configuration via Group Policy and include ESA rules via config files.
• Browsers/IE add-on [B451]: if the config flag IEIgnoreFrames is set, the IE add-on only tracks the main page, ignoring frames.
• Browsers [I301]: added possibility to record URIs with special characters in Chrome/Firefox.
• Driver [B265]: timestamps now have a higher resolution (to within a microsecond).
• Logging [B437]: the configuration is now written to a separate log file (uberAgentConfiguration.log).
• Dashboards [B422]: the Application Errors dashboard now shows the affected hosts/users.
• Dashboards [B272]: new dashboard Application GPU.
• Dashboards [B272]: the Process GPU dashboard now shows usage data over time.
• Dashboards [B426]: the Data Volume dashboard now shows ESA sourcetypes, too, and lists the data volume per product (UXM/ESA).
• Dashboards [I468]: added calculated fields to make the use of ProcName consistent across all dashboards.
• Dashboards [B419]: the dashboards Process startup (UXM) and Process Tagging Events (ESA) are enriched with the Hash and Hash type fields.
• Dashboards [B167]: the dashboards Application Network Communication and Application Network Issues are enriched with the NetTargetSourceAddress field.
• Dashboards [I162]: the network monitoring dashboard shows the latency for TCP handshakes as Initial TCP send latency (ms).
• Dashboards [B478]: added an OsPlatform filter to all dashboards, for easy selection of Windows and macOS devices.
• Dashboards [B285]: the dashboards Single Machine Detail and User Sessions Overview are enriched with the HwManufacturer and HwModel fields.
• Network configuration [I140]: some VPN adapters (e.g., Cisco AnyConnect) were excluded because they present themselves not as VPN but as Ethernet interfaces to the OS. The new config flag NoGatewayCheck can be used to adjust uberAgent’s detection algorithm.
• Network communication: uberAgent now measures TCP send latency accurately. Previously, the measurements had a high margin of error.
• Backend: process start and stop events are now sent in bulk API calls to HTTP(S) receivers. This significantly reduces the number of API calls and the load on the endpoint.
• Performance [I98]: reduced agent CPU & memory usage.
• Sourcetype uberAgent:System:SystemPerformanceSummary2 field NetUtilizationPercent: calculation now only includes active physical network adapters. Before, virtual adapters were counted, too, which could skew the result.
• Splunk [B276]: improved lookup performance in large environments by switching from CSV to KV Store.
• Splunk [B404]: improved scheduled searches performance by switching from raw to data model searches.

Bugfixes

• Session Details [I228]: session protocol name and connection state contained wrong values if the protocol could not be determined.
• Service [I176]: fixed a handle leak in logon and logoff monitoring.
• Service [I222]: fixed inheriting handles to child processes.
• Service [I238]: fixed a memory leak causing an internal list grow unnecessarily.
• Service [I135]: fixed a memory leak in public key cryptography code.
• Service [I260]: when shutting down a Windows 7 OS the service was not shut down in time.
• Service [I200]: zero-only GUIDs are now suppressed.
• Service [I96]: a BSOD or power loss does not cause multiple bugcheck events anymore.
• Service [I205]: in rare cases involving Citrix PVS the OS boot time reported by uberAgent would reflect the master image’s boot time.
• Service [I90]: logon metrics sometimes contained wrong fields.
• Dashboards [I150]: the timechart values within the Citrix XA/XD Licensing dashboard now match the details table.
• Dashboards [I248]: filter expressions with special characters resulted in unsuccessful searches.
• uAInSessionHelper [I142]: reduced the CPU usage while collecting per-process GPU metrics.
• uAInSessionHelper [I22]: fixed rare crash during with faulting module KERNEL32.DLL_unloaded during the startup phase of the helper.
• Backend [I156]: if Kafka/Confluent schema ID caching is disabled, the log is flooded with: Did not find the value_schema_id in the server’s response.
• Backend [I3]: on-demand metrics ProcessStartup and ProcessStop were always sent to all receivers.
• Backend [I201]: Splunk HTTP Event Collector (HEC) falls back to HTTP/1.0 if clients don’t specify a user agent string. This breaks persistent HTTP connections resulting in high connection counts.
• IE add-on [I93]: SessionFgBrowserActiveTabHost is not sent if BrowserPerformanceIE is disabled or the URL must be ignored due to configuration.
• Splunk [I88]: moved configuration settings in props.conf from the search head to the indexer app.
• Browsers [I48]: in rare cases, multiple concurrent communications with the Chrome/Firefox browser extensions would get mixed up. uberAgent would stop processing extension data and log “BrowserExtTransact,Response protocol type does not match requested data”.
• Browsers/IE add-on [I97]: the field SessionFgBrowserType could be empty even though IE was in the foreground. This happened with a blank page as the active tab, for example.
• Logon monitoring [I99]: in rare cases, it could happen that uberAgent started logon monitoring for a session while the service was still starting up. When this happened for session 0, the service’s memory usage would slowly grow with every new process started in the session because logon monitoring for session 0 had no timeout.
• Logoff monitoring [I190]: when logoff monitoring is aborted due to a false positive signal, the list storing information about the processes involved in the logoff is not reset, causing a growth in memory usage.
• Citrix ADC [I105]: if the hostname of Citrix ADC Gateway Edition was missing, dashboards would be empty.
• Splunk [I143]: changed the scheduled search populate_hostinfo so that it returns data even if the sub-search terminates.

Release notes

• Sourcetype: new sourcetype uberAgentESA:Process:ProcessStop with fields: Timestamp, ProcName, ProcUser, ProcLifetimeMs, AppId, ProcID, ProcParentID, SessionID, ProcGUID, SessionGUID, ProcParentName, ProcPath, ProcCmdline, IsElevated, AppVersion, ProcParentGUID, ProcHash, HashType, IsProtected.
• Sourcetype: new sourcetype uberAgentESA:ActivityMonitoring:ProcessTagging with fields: Timestamp, EventType, ProcName, ProcParentName, ProcUser, ProcLifetimeMs, ProcID, ProcParentID, ProcGUID, ProcParentGUID, ProcPath, ProcCmdline, ProcTag1, ProcRiskScore1, ProcHash, IsElevated, SessionID, SessionGUID, AppId, AppVersion, HashType, ImageName, ImagePath, ImageHash, NetTargetIp, NetTargetName, NetTargetPort, NetProtocol, IsProtected, EventCount.
• Sourcetype: new sourcetype uberAgentESA:System:ScheduledTasks with fields: Timestamp, TaskEventType, TaskFolder, TaskName, TaskUserName, TaskPrincipal, LogonType, Elevated, TaskAuthor, TaskHidden, WakeToRun, HasActions.
• Sourcetype: new sourcetype uberAgentESA:System:ScheduledTaskActions with fields: Timestamp, TaskPath, IsDeprecated, ActionType, ActionListIndex, ExePath, ExeArguments, ExeWorkingDir, ComClsid, ComData, ComBinary, ComHandlerDescription, ComRemoteComputer, ComServiceName, AutoElevated, EmailBcc, EmailCc, EmailFrom, EmailServer, EmailSubject, EmailTo, MsgTitle, MsgContent.
• Sourcetype: new sourcetype uberAgentESA:System:ScheduledTaskTriggers with fields: Timestamp, TaskPath, TriggerId, TriggerType, EventTriggerSubscription, EventTriggerNumValues, TriggerUserId, LogonTriggerPossiblyGroup, TriggerEnabled, TriggerStartBoundary, TriggerEndBoundary, TriggerRepetitionDuration, TriggerRepetitionInterval, TriggerRepetitionStopAtDurationEnd, TriggerListIndex, DayDisplayName, WeekDisplayName, MonthDisplayName, DayOfMonthDisplayName, DailyTriggerDaysInterval, WeeklyTriggerWeeksInterval, MonthlyTriggerRunOnLastDayOfMonth, MonthlyDowTriggerRunOnLastWeekOfMonth, StateChangeId, WnfTriggerStateName.
• Sourcetype: uberAgent:Citrix:Applications has new field(s): ApplicationGroupId, ApplicationGroupName.
• Sourcetype: uberAgent:Citrix:Applications has removed fields: LifecycleState, CreatedDate and ModifiedDate.
• Sourcetype: uberAgent:Process:ProcessStartup has new field(s): IsProtected.
• Sourcetype: uberAgent:Process:ProcessStartup has new field(s): ProcHash, HashType and ProcParentGUID (these requires ESA to be enabled).
• Sourcetype: uberAgent:Process:NetworkTargetPerformance has new fields: NetTargetSendLatencyInitialMs and NetTargetSendLatencyInitialCount.
• Sourcetype: uberAgent:OnOffTransition:BootProcesses has new fields: SortOrder2 replaces SortOrder because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:Session:SessionDetail has new fields: SessionClientHwIdCtx2 replaces SessionClientHwIdCtx because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:Session:SessionDetail has new fields: SessionRpLatencyMs2 replaces SessionRpLatencyMs because the Kafka data type was incorrect (int instead of double).
• Sourcetype: uberAgent:CitrixADC:AppliancePerformance has new fields: NumCpus2, MemSizeMB2, and MemUseInMB2 replaces NumCpus, MemSizeMB, and MemUseInMB because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:CitrixADC:Gateway has new fields: TotalRequests2, TotalResponses2, and SessionTimeout2 replaces TotalRequests, TotalResponses, and SessionTimeout because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:CitrixADC:vServer has new fields: ActSvcs2, TotHits2, TotalRequests2, TotalResponses2, VSLBHealth2, and SessionTimeout2 replaces ActSvcs, TotHits, TotalRequests, TotalResponses, VSLBHealth, and SessionTimeout because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:CitrixADC:ApplianceInventory has new fields: SSLCards2 and SSLCardsUp2 replaces SSLCards and SSLCardsUp because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:Application:Errors has new field(s): HangType.
• Sourcetype: uberAgent:System:MachineInventory field BatteryWearLevelPercent does not report negative numbers anymore if the full charged capacity is higher than designed capacity.
• Sourcetype: merged the following KV sourcetypes into the new CSV sourcetype uberAgent:Logon:LogonDetail: uberAgent:Logon:SessionLogonTime, uberAgent:Logon:ProfileLoadTimeMs, uberAgent:Logon:GroupPolicyProcessingTimes, uberAgent:Logon:GroupPolicyLogonScriptTimeMs, uberAgent:Logon:ADLogonScriptTimeMs, uberAgent:Logon:ResWmProcessingTimeMs, uberAgent:Logon:ShellStartupTimeMs, uberAgent:Logon:TotalLogonTimeMs, uberAgent:Logon:LogonPerformance.
• Sourcetype: merged the following KV sourcetypes into the new CSV sourcetype uberAgent:Logoff:LogoffDetail: uberAgent:Logoff:SessionLogoffTime, uberAgent:Logoff:ProfileUnloadTimeMs, uberAgent:Logoff:GroupPolicyLogoffScriptTimeMs, uberAgent:Logoff:TotalLogoffTimeMs, uberAgent:Logon:SessionEnd, uberAgent:Logoff:LogoffPerformance.
• Sourcetype: replaced KV sourcetype uberAgent:Logon:GroupPolicyCSEDetail with CSV sourcetype uberAgent:Logon:GroupPolicyCSEDetail2. No changes to the fields.
• Sourcetype: uberAgent:Process:NetworkTargetPerformance has new fields: NetTargetSendJitterMs and NetTargetSendJitterCount.
• Sourcetype: uberAgent:Process:ProcessDetail has new fields: ProcIOLatencyReadMs2 and ProcIOLatencyWriteMs2 replaces ProcIOLatencyReadMs and ProcIOLatencyWriteMs because the Kafka data type was incorrect (int instead of double).
• Sourcetype: uberAgent:Process:LogonProcesses has new fields: ProcIOLatencyReadMs2 and ProcIOLatencyWriteMs2 replaces ProcIOLatencyReadMs and ProcIOLatencyWriteMs because the Kafka data type was incorrect (int instead of double).
• Sourcetype: uberAgent:Process:LogonProcesses has new fields: SortOrder2 replaces SortOrder because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:Process:LogoffProcesses has new fields: SortOrder2 replaces SortOrder because the Kafka data type was incorrect (string instead of int).
• Sourcetype: uberAgent:Process:LogoffProcesses has new fields: ProcIOLatencyReadMs2 and ProcIOLatencyWriteMs2 replaces ProcIOLatencyReadMs and ProcIOLatencyWriteMs because the Kafka data type was incorrect (int instead of double).
• Sourcetype: uberAgent:Logoff:ProfileUnloadTimeMs (now merged into uberAgent:Logoff:LogoffDetail) has new field: ProfileUnloadTimeMs2 replaces ProfileUnloadTimeMs because the Kafka data type was incorrect (string instead of number).
• Sourcetype: uberAgent:Citrix::Licenses has new fields: LicenseEdition2 replaces LicenseEdition because the Kafka data type was incorrect (int instead of string).
• Sourcetype: uberAgent:System:GpuUsage has removed fields: ComputeUsagePercentEngine0 through ComputeUsagePercentEngine11 because a much more useful alternative exists with the sourcetype uberAgent:System:GpuUsageEngine.
• Sourcetype: uberAgent:Session:SessionCount has been removed.
• Configuration: now uses the terms allowlist and denylist instead of whitelist and blacklist. Older terms remain supported.
• Configuration: the NetworkTargetPerformanceProcess metric was moved from its own timer #5 to the shared default timer #1 because it does not perform network requests any more.
• Splunk: the minimum required Splunk version is now 6.6 (formerly 6.3).
• Azure Monitor (formerly OMS Log Analytics): events are now assigned to log type tables by sourcetype instead of by index.
• Performance counters: changed the sourcetype names from uberAgent:System:PerformanceCounter to uberAgent:PerformanceCounter:TimerName (where TimerName is the timer name from uberAgent’s configuration).
• Splunk [B276]: changed the type of the following lookups from CSV to KV Store: lookup_hostinfo, lookup_hostinfo2, lookup_processstartup_processlist, lookup_networktargetperformance_targetlist.
• Splunk [B404]: changed the following scheduled searches from raw to data model searches: populate_appnameidmapping, populate_hostinfo, populate_hostinfo2.

Known issues

• Dashboards [I338]: the new Experience Score dashboard makes use of several saved searches, whose results are saved in the Splunk KVStore. A modification to two searches is needed, to avoid high disk usage by collection_score_historic_per_machine and collection_score_historic_per_session.
  Within the savedsearches.conf file ($SPLUNK_HOME/etc/apps/uberAgent/default/) remove append=true from line 278 and line 486.

• System CPU usage [I336]: PE hash calculation is enabled by default what results in a higher system CPU usage. As a workaround, explicitly disable the hash calculation by adding the following to uberAgent’s configuration:

  [ProductComponents]
  EnableESA = true
  [ProcessStartupSettings]
  EnableCalculateHash = false

• Process driver [I323]: stop error (blue screen) on server operating systems when certain modifications are in-place. As a workaround, disable ESA.

  [ProductComponents]
  EnableESA = false

• VPN bandwidth: reduced VPN performance when uberAgent’s new network monitoring driver is enabled in conjunction with some types of VPN. This was observed with Palo Alto Networks Global Protect and WireGuard. As a workaround, switch back to uberAgent’s previous network monitoring data source, ETW, by adding the following to uberAgent’s configuration:

  [NetworkTargetPerformanceProcess_Config]
  NetworkDriverEnabled = false

• Internet Explorer [I265]: uberAgent’s Internet Explorer extension does not read the configuration correctly. As a workaround, run REG COPY "HKLM\SOFTWARE\vast limits\uberAgent\ConfigCache" "HKLM\SOFTWARE\WOW6432Node\vast limits\uberAgent\ConfigCache" /s /f in an elevated command prompt after installation.
• Network communication [I197]: latency metrics may be not accurate for delayed TCP acknowledgements.
• Network communication [I291]: in rare cases sending data to Kafka results in a SEC_E_BUFFER_TOO_SMALL error message in the logfile. This should have no affect; the transmission is repeated and succeeds on the second try.
• GPU [I33]: values for the fields ComputeUsagePercentAllEngines, ComputeUsagePercentEngine0 and similar can be higher than 100 with Intel Iris GPUs on Windows Server 2016 1607.
• Citrix ADC: in very rare cases the content of the Virtual Server Performance field vServerName contains spaces in wrong places.
• Citrix XA/XD Machines: when running the Citrix VDA on a Citrix Delivery Controller, some per-machine information is missing.
• Browsers/IE add-on: metrics are not collected on page reload.
• Browsers/IE add-on: metrics are collected incompletely for the configured start page.
• IE browser performance monitoring does not work if IE is running as a Citrix XenApp published application. It does work from published desktops, however.
• Boot duration: the metrics TotalBootTimeMs, MainPathBootTimeMs and PostBootTimeMs cannot be determined for every system boot.
• Browser web app performance: websites may modify the JavaScript performance variable. When that happens, uberAgent cannot determine the page load duration.
• Update inventory: not all installed Windows updates may be reported due to API limitations.