How We Achieved Effectively Unlimited Scalability
Our largest customers have deployed uberAgent to hundreds of thousands of endpoints – each. This article explains which aspects of the product’s architecture make that kind of (effectively unlimited) scalability possible.
uberAgent’s preferred backend, Splunk, is a big data platform that has been designed to handle huge amounts of data. If architected well, Splunk has no built-in limitations and scales practically infinitely.
Splunk’s data index is typically spread across multiple servers, each of which stores only a subset of all data. Search queries are executed in parallel by all indexers, whose partial results are combined into a complete result set. Increasing the capacity of a Splunk installation is as easy as adding more servers. See this article for an overview of how Splunk works.
Even very large numbers of endpoints do not need to be segmented in any way, as is necessary with some other products. Splunk searches always cover the entire fleet of endpoints. It is, of course, possible to search only a subset of machines or users, e.g., a specific site.
uberAgent and Splunk have no limit on how much historical data can be stored or for how long data can be kept. Neither are historical metrics compressed in any way: all data is always available in full resolution, in the same way as it was collected by the endpoint agent. With Splunk, you’re only limited by the available disk space.
If you only have a few hundred endpoints, the amount of data that each endpoint generates does not really matter. Things are very different, though, when hundreds of thousands of endpoints collect and send detailed information on a regular basis. At scale, any optimization that reduces the data volume or the number of API calls over the network proves to be invaluable.
uberAgent’s data volume optimizations include, but are not limited to:
- Most data types are transmitted in the most compact form possible, as comma-separated text. Even headers are omitted; Splunk knows about the field structure.
- Common text constants, such as the name of a browser (e.g.,
Chrome), are not transmitted as text strings, but represented by single-digit numbers, e.g.
1. The human-readable names are added via automatic lookups in Splunk.
- Numeric fields very often have the value zero. E.g., most processes’ CPU usage effectively amounts to zero most of the time. Instead of transmitting zero values as
0.0, as most applications would do, uberAgent shortens them to
0, which reduces the data’s length from 3 bytes to 1, saving 66%.
Dashboards visualizing uberAgent’s data and the searches that are powering them cannot just query and attempt to display data about individual machines. Tables with more than a few hundred (or rather: dozen) rows are unusable. Inefficient searches that have not been optimized for huge result sets take too long to execute.
uberAgent’s Experience Score dashboard is the perfect entry point. It gives an overview of the state of the environment and offers drill-downs to any relevant KPI or metric one might want to inspect.
uberAgent’s dashboards never attempt to list all entities individually. They always start with overview metrics or charts that provide grouped information per machine type, for example. Drilldown capabilities offer quick navigation down to individual endpoints, users, applications, and the like.
Organizations with hundreds of thousands of endpoints tend to have heterogeneous networks and complex requirements. A product that thrives in such an environment needs to be inherently flexible in the way it can be configured and deployed.
uberAgent’s installation packages can be rolled out with any software deployment tool a customer might have in place. License files need not be deployed: they can be centrally hosted and updated. Agents occasionally connect to the license repository and update their cached copies.
The agent can be configured via config files or via Active Directory Group Policy (Windows only). uberAgent ships with two configuration sets: the default configuration and a second configuration optimized for data volume. The data collection frequency of any of uberAgent’s metrics can be configured freely, from very high resolution (mere seconds) to minute or even hour or day intervals (reducing the data volume).
Allowlists and denylists make it possible to only collect what is needed. Event data filtering allows for powerful manipulation of the collected data before it leaves the endpoint, e.g., to remove sensitive data or further reduce the data volume.
uberAgent is an innovative Windows and macOS user experience monitoring (UXM) and endpoint security analytics (ESA) product. UXM highlights include detailed information about boot and logon duration, application unresponsiveness detection, network reliability drill-downs, process startup duration, application usage metering, browser performance, web app metrics, and Citrix insights. ESA comes with a sophisticated activity monitoring engine, the uAQL query language, detection of risky activity, DNS query monitoring, hash calculation, registry monitoring, and Authenticode signature verification.
About vast limits
vast limits GmbH is the company behind uberAgent, the innovative user experience monitoring and endpoint security analytics product. vast limits’ customer list includes organizations from industries like finance, healthcare, professional services, and education, ranging from medium-sized businesses to global enterprises. vast limits’ network of qualified solution partners ensures best-in-class service and support anywhere in the world.