uberAgent 7.0 Preview: Intelligent Disk Buffering

While we’re finalizing version 7.0 of our user experience monitoring & endpoint security analytics products uberAgent UXM and uberAgent ESA, let’s take a look at another cool new feature: intelligent disk buffering.

Evolution of Outgoing Event Buffering

If you follow our blog closely, you’ll notice that we already talked about disk buffering when we announced Persistent Output Queue with uberAgent 6.2. So, what’s new?

In order to better explain the innovation that is Intelligent Disk Buffering, let me recap how uberAgent handled outgoing events in the past.

Memory Buffer

uberAgent always had a memory buffer for outgoing events. New events were stored in a 10 MB buffer in RAM until they could be sent to the backend. This was very efficient and worked well, but had certain disadvantages: when the buffer overflowed, new events were discarded. When the endpoint was restarted, all events in the buffer were lost.

In short, the memory buffer protected against short network outages but was not sufficient for devices such as laptops that may be offline for prolonged periods of time.

Persistent Output Queue

Introduced with uberAgent 6.2, Persistent Output Queue (POQ) adds a lightweight database layer to an endpoint’s outgoing event queue. When POQ is enabled for a backend, new events are not written to the memory buffer (see above), but to an SQLite database located on the endpoint’s disk. This guarantees that no events are lost independent of network connectivity. However, it also adds a small but constant amount of disk IO activity.

Intelligent Disk Buffering Combines the Best of Both Technologies

As we’ve seen above, the Persistent Output Queue is great in that it guarantees no events are lost. Isn’t it a little unfortunate, though, that it always writes new events to disk even if they could be sent off to the backend right away? In other words: why store new events on disk if network connectivity to the backend is available?

That’s exactly what we asked ourselves, too, and we’re thrilled to announce that Intelligent Disk Buffering combines the best of both buffering technologies mentioned above. Let’s see how it works.

Case A: Backend is Reachable

uberAgent 7.0 constantly monitors network connectivity to its backend. As long as the backend is reachable and processes new events correctly, our endpoint agent does not store events on disk but sends them to the backend directly, thus avoiding any disk IO on the endpoint.

Case B: Backend is Unreachable

If the backend becomes unreachable, either because network connectivity is lost or because the backend has issues accepting data, the agent writes new events to the Persistent Output Queue’s local database. uberAgent periodically checks if connectivity is back up. Once that is the case, it sends all events buffered to disk and reverts to the efficient direct sending (see case A).

Configuration Flexibility

As always, uberAgent is highly configurable. You can choose to disable Intelligent Disk Buffering, reverting the agent’s behavior to memory buffering only.

Choice & Control: Efficiency vs. Data Safety

If you leave Intelligent Disk Buffering at its default setting “enabled”, you can control how many events the agent keeps in RAM before it switches to disk buffering if the backend becomes unreachable. This allows you to fine-tune the agent’s behavior, either opting for maximum efficiency (as little disk IO as possible), maximum data safety (no events to be lost under any circumstances), or a balanced configuration, which is the default.