What Are Persistent Output Queues?

Persistent output queues (POQs) buffer the generated events on disk before uberAgent attempts to send them to the backend. Only when an event has been delivered successfully to Splunk (or one of the other backends supported by uberAgent), it gets removed from the POQ’s buffer.

uberAgent’s POQs ensure that no data is lost even in situations where the backend is unavailable for prolonged periods. The most important use case for POQ is for laptops.

On mobile devices, uberAgent was traditionally coupled with Splunk’s Universal Forwarder (UF) due to its persistent queue functionality. With uberAgent’s built-in POQs, it’s not necessary anymore to deploy the UF just for its disk buffering feature.

Disk Space Requirements

The maximum disk space to be used by the POQ can be limited in uberAgent’s configuration (see below). The following table is intended as a rough indicator of disk space requirements (PC with average usage, 10 hours uptime, default uberAgent configuration):

Configuration Settings

Persistent output queues are configured per receiver. Please note that receivers with a POQ must have a unique name.

Enabling or Disabling the POQ

PersistentOutputQueue

Enables or disables the persistent output queue for the current receiver.

• Setting name: PersistentOutputQueue
• Valid values: Off, SQLite
• Default: Off
• Required: no

Limiting POQ Disk Space Usage

PersistentOutputQueueMaxAgeDays

Configures the maximum retention time for POQ entries. Entries older than the specified number of days are removed.

• Setting name: PersistentOutputQueueMaxAgeDays
• Valid values: positive numbers (0 = unlimited)
• Default: 120
• Required: no

PersistentOutputQueueMaxSizeMb

Configures the maximum disk space in MB for the persistent output queue. The file size is checked after events are stored. It is therefore possible that the configured limit is slightly exceeded.

• Setting name: PersistentOutputQueueMaxSizeMb
• Valid values: positive numbers (0 = unlimited)
• Default: 500
• Required: no

POQ events are stored in the format expected by the receiver’s backend. Some backends expect the data in a JSON format while others accept raw data. Especially the JSON format bloats events considerably, which means that the required disk space can be much larger than the raw event data as indexed by Splunk.

POQ Location on Disk

PersistentOutputQueuePathWindows

Specifies the POQ storage location on Windows machines. Environment variables are supported.

• Setting name: PersistentOutputQueuePathWindows
• Valid values: any valid path
• Default: %PROGRAMDATA%\vast limits\uberAgent\Output Queue
• Required: no

When the default path is used, uberAgent sets the permissions on the directory so that only members of the local group Administrators and the local system have read/write access. When a custom path is specified, uberAgent does not set permissions on the POQ directory. We highly recommend securing the directory.

PersistentOutputQueuePathMacOS

The setting PersistentOutputQueuePathMacOS specifies the POQ storage location on Windows machines. Environment variables are not supported.

• Setting name: PersistentOutputQueuePathMacOS
• Valid values: any valid path
• Default: /Library/Application Support/uberAgent/Output Queue
• Required: no

When the default path is used, uberAgent sets the permissions on the directory so that only the root user has read/write access. When a custom path is specified, uberAgent does not set permissions on the POQ directory. We highly recommend securing the directory.

Miscellaneous Settings

PersistentOutputQueueErrorHandling

Configures how errors during POQ event processing are handled.

• Setting name: PersistentOutputQueueErrorHandling
• Valid values: IgnoreFile, StopReceiver, Shutdown
• Default: IgnoreFile
• Required: no

In case of an error (e.g., out of disk space), uberAgent may either stop the current receiver (StopReceiver), shut down the agent entirely (Shutdown), or ignore the current POQ file (IgnoreFile). In case all receivers have stopped, uberAgent shuts itself down.

Advanced Topics

Data Collection Without Network Connectivity

Receivers with a POQ can be used without a configured backed. In other words: the Servers setting can be left empty. This is useful to collect data on a machine without network connectivity, such as in a DMZ. The collected data can then be transferred to a machine with network connectivity simply by copying the POQ files.

Recommendations

DMZ machine:

• Configure the receiver’s settings Name, Type, and Protocol to match the settings on the machine you later plan to transfer the POQ files to.
• Once sufficient data has been collected, stop uberAgent and copy the POQ files off the machine.

Machine with network connectivity:

• Stop uberAgent.
• Copy the POQ files from the DMZ machine to the local POQ path.
• Start uberAgent. The agent sends the DMZ machine’s data off to its backend.

POQ File Data Format

The internal format of the persistent output queue is an SQLite database that is not protected and can be read by SQLite browsers like DB Browser for SQLite.