Introduction

What is Azure Data Explorer (ADX)?

Microsoft Azure Data Explorer (ADX) is a data analytics platform designed for querying and analyzing large volumes of data in near real time.

What is Azure Event Hubs?

Microsoft Azure Event Hubs is a real-time data ingestion service based on Apache Kafka. It can process millions of events per second which subsequently can be made available to hundreds of individual consumers (applications that read and process incoming data).

Why Combine Azure Data Explorer With Azure Event Hubs for uberAgent Data?

In enterprise scenarios, uberAgent’s endpoint agents may send events concurrently from hundreds of thousands of devices. This requires a backend service that is capable of ingesting millions of events per second. In the Azure cosmos, Event Hubs is exactly such a service. Once ingested, it buffers uberAgent’s events until they’ve been routed to Azure Data Explorer (ADX), where they can be efficiently queried from various additional services via the Kusto Query Language (KQL).

Advantages of Using Azure Data Explorer & Azure Event Hubs for uberAgent Data

By routing uberAgent’s data through Azure Event Hubs to Azure Data Explorer, enterprises can make DEX and endpoint security metrics available to individual teams within the organization in a publish/subscribe kind of model. Once access has been granted to an ADX database, a team can start working with the metrics it stores.

Azure Event Hubs makes it possible to distribute uberAgent’s metrics in a highly scalable manner, supporting hundreds of thousands of endpoints (data producers) and thousands of consumers. uberAgent natively supports Azure Data Explorer via Azure Event Hubs.

Prerequisites

In order for uberAgent to be able to send data to an Azure Data Explorer backend, the following Azure services are required:

• Microsoft Azure Event Hubs
• Microsoft Azure Data Explorer Cluster
• Microsoft Azure Enterprise Application

Please see the system requirements for details.

Setup

Navigate to the Azure portal and sign in with an organizational account associated with your Microsoft Azure subscription.

Creating an Azure Enterprise Application

In order to authenticate to Event Hubs, an Enterprise Application needs to be created. Follow the steps in the Microsoft documentation. Make sure to obtain a client ID and a client secret.

Creating the Azure Event Hubs Namespace & Instance

1. Create a new Event Hubs namespace Azure service by specifying a resource group, a unique namespace name, a location, and a pricing tier.
2. Assign the following permissions to the namespace for the enterprise application you created above:
   • Azure Event Hubs Data Sender
   • Schema Registry Contributor (Preview)
   • Schema Registry Reader (Preview)
3. Open up the newly created Event Hubs namespace and create an Event Hubs instance by specifying a unique name, the partition count (1 is sufficient), and the retention time.

Creating the Azure Data Explorer Cluster

1. Create a new Azure Data Explorer Cluster by specifying a resource group, a unique cluster name, and the workload.
2. Create a new database in the newly created Azure Data Explorer Cluster by specifying a database name, the retention period, and the cache period.
3. Open the newly created database and create a new data connection, using the Event Hubs instance created above as the source. Specify a data connection name, and the name of the Event Hubs namespace. Make sure to allow routing the data to other databases and leave the table name empty. Set the data format to CSV.

Creating the Azure Data Explorer Tables

The uberAgent endpoint agent does not create the database tables in Azure Data Explorer. The script uberAgentSchema.AzureDataExplorer.kql to create the tables is part of the uberAgent download package.

Configuring Azure Event Hubs as Data Receiver in uberAgent

Receiver Stanza

The following configuration settings are required in a Receiver stanza:

• Type: AzureEventHubs
• Protocol: HTTP
• AzureEventHubsConfigurationId: The ID of the AzureEventHubsConfiguration stanza (see below).

AzureEventHubsConfiguration Stanza

The following configuration settings are required in an AzureEventHubsConfiguration stanza:

• Id: Unique GUID this stanza is referenced by from a Receiver stanza.
• AuthenticationURL: The authentication URL (optional). The default value is: https://login.microsoftonline.com/###UA_TENANTID###/oauth2/token
• SendDataURL: The URL to send the data to (optional). The default value is: https://###UA_EVENTHUBSNAMESPACE###.servicebus.windows.net/###UA_HUBNAME###/messages
• AzureEventHubsNameSpace: The name of the Azure Event Hubs namespace. Placeholder variable: ###UA_EVENTHUBSNAMESPACE###
• AzureHubName: The name of the Azure Event Hubs instance. Placeholder variable: ###UA_HUBNAME###
• AzureTenantID: The Entra tenant ID. Placeholder variable: ###UA_TENANTID###
• AzureClientID: The Entra enterprise application client ID.
• AzureClientSecret: A client secret associated with the client ID.

The fields ###UA_TENANTID###, ###UA_EVENTHUBSNAMESPACE###, and ###UA_HUBNAME### are placeholders and are replaced by the agent with their corresponding value.

Example

[Receiver]
Name = vast limits Azure Event Hubs
Type = AzureEventHubs
Protocol = HTTP
AzureEventHubsConfigurationId = 82204bf8-fc99-4d59-8903-7206234dcbc8

[AzureEventHubsConfiguration]
Id = 82204bf8-fc99-4d59-8903-7206234dcbc8
AzureEventHubsNameSpace = vastlimits
AzureHubName = uberagent
AzureTenantID = YOUR_TENANT_TENANT_ID
AzureClientID = YOUR_ENTRA_ENTERPRISE_APPLICATION_CLIENT_ID
AzureClientSecret = YOUR_AZURE_CLIENT_SECRET

Implementation Details

Authentication Method

Authentication from the uberAgent endpoint agent to the Event Hubs instance happens by way of Azure AD JSON web tokens (JWT).

Routing Events to the Proper ADX Table

uberAgent sends many different types of events, distinguished by a so-called sourcetype. Each sourcetype has its own field definition and, therefore, needs to be stored in its own ADX table.

While uberAgent sends all types of events to a single Event Hubs ingestion service, from there the incoming events need to routed to the proper ADX tables depending on each event’s sourcetype. This routing is automatically accomplished by adding an HTTP header with the name of the ADX table, e.g.:

"Table" : "Process_ProcessStartup"

Timer scripts

Timers are mostly used to determine metrics (e.g. SessionDetail). They can also be used to start custom scripts (see here). The output of the scripts is sent to the backend where the timer name is part of the sourcetype. In most backends the sourcetype does not need to be created in advance, or it is created by uberAgent. In Azure Data Explorer the table needs to be created manually by the customer otherwise ingestion errors occur.
The table name has the following format: Script_<timer name>.

The following gives an example of the KQL script needed to run to ingest the result data of a custom script into the corresponding table in Azure Data Explorer.

[Timer]
Name = DetermineDepartment
Interval = 30000
Script = C:\Scripts\DetermineDepartment.cmd
ScriptContext = UserSessionAsUser

The output of the script DetermineDepartment.cmd is:

Department=Sales UserName=John

The KQL script to create the table and the ingestion mapping in Azure Data Explorer for this timer is:

.execute database script <|
.create table Script_DetermineDepartment (Timestamp: long, Host: string, Department: string, UserName: string);
.create table Script_DetermineDepartment ingestion csv mapping 'Script_DetermineDepartment_mapping' '[{"column":"Timestamp", "Properties":{"Ordinal":"0"}}, {"column":"Host", "Properties":{"Ordinal":"1"}}, {"column":"Department", "Properties":{"Ordinal":"2"}}, {"column":"UserName", "Properties":{"Ordinal":"3"}}]';

The fields Timestamp and Host are mandatory and are automatically added by uberAgent.
The field names in the output do not necessarily have to match the field names in the table.