uberAgent 6.2 Preview: Persistent Output Queue (Disk Buffering)
While we’re finalizing version 6.2 of our user experience monitoring & endpoint security analytics products uberAgent UXM and uberAgent ESA, let’s take a look at another cool new feature: persistent output queues.
What Are Persistent Output Queues?
Persistent output queues (POQs) buffer the generated events on the endpoint’s disk before uberAgent attempts to send them to the backend. Only when an event has been delivered successfully is it removed from the POQ’s buffer.
uberAgent’s persistent output queues ensure that no data is lost even in situations where the backend is unavailable for prolonged periods of time. The most important use case for POQs is with laptops.
On mobile devices, uberAgent was traditionally coupled with Splunk’s Universal Forwarder due to UF’s persistent queue functionality. With uberAgent’s new built-in persistent output queues, it’s not necessary anymore to deploy Universal Forwarder just for its disk buffering feature.
Behavior Without POQs
Without persistent output queues, events are buffered in memory. If events cannot be sent to the backend, uberAgent keeps them in the in-memory buffer so that they can be sent at a later time. The memory buffer is limited to 10 MB per receiver by default. If the size limit is reached, new events are discarded until the buffer size drops below the limit.
uberAgent’s in-memory buffer protects well against data loss during shorter periods of network unavailability. It meets its limits, however, when endpoints might be rebooted while offline.
That is where persistent output queues come in.
POQ Internals
With POQs enabled, all events are stored in SQLite database files. A background thread reads the data from the database and sends it to the backend servers. Events that were transmitted successfully are removed from the database. uberAgent’s POQ code has been carefully optimized to minimize disk IO.
POQs are configured per receiver (backend). If multiple receivers are configured, persistent output queues can be enabled for each receiver independently.
uberAgent’s persistent output queues are fully compatible with all supported backend types: Splunk, Elasticsearch, Apache Kafka, and Microsoft Azure Monitor.
POQ Configuration Options
As expected from uberAgent, persistent output queues offer a lot of flexibility. Configuration options include the maximum retention time, the maximum used space, and the location on disk.
Maximum Retention Time
Persistent output queues keep a timestamp for each event. Based on the maximum retention time defined in the configuration, events that are older than the maximum are deleted from the POQ files. The default retention time is 120 days.
Maximum Size
To keep disk usage in check, POQs enforce a size limit. The default maximum size of 500 MB can be adjusted as needed. When a queue’s maximum size is reached, old events are purged to make room for new data. This behavior is different from uberAgent’s in-memory buffer, which keep the earliest events. The persistent output queue, on the other hand, keeps the latest events.
Storage Location
By default, POQ files are stored in the %PROGRAMDATA%\vast limits\uberAgent\Output Queue directory on Windows and in /Library/Application Support/uberAgent/Output Queue on macOS. Of course, the storage location of POQ files can be changed to any directory that is writeable by the agent.
About uberAgent
The uberAgent product family offers innovative digital employee experience monitoring and endpoint security analytics for Windows and macOS.
uberAgent UXM highlights include detailed information about boot and logon duration, application unresponsiveness detection, network reliability drill-downs, process startup duration, application usage metering, browser performance, web app metrics, and Citrix insights. All these varied aspects of system performance and reliability are smartly brought together in the Experience Score dashboard.
uberAgent ESA excels with a sophisticated Threat Detection Engine, endpoint security & compliance rating, the uAQL query language, detection of risky activity, DNS query monitoring, hash calculation, registry monitoring, and Authenticode signature verification. uberAgent ESA comes with Sysmon and Sigma rule converters, a graphical rule editor, and uses a simple yet powerful query language instead of XML.
About vast limits
vast limits GmbH is the company behind uberAgent, the innovative digital employee experience monitoring and endpoint security analytics product. vast limits’ customer list includes organizations from industries like finance, healthcare, professional services, and education, ranging from medium-sized businesses to global enterprises. vast limits’ network of qualified solution partners ensures best-in-class service and support anywhere in the world.
