Citrix ADC Monitoring & Alerting

uberAgent 5.2 introduced Citrix ADC monitoring (formerly NetScaler ADC). This functionality lets you check the appliance health, gateway data volume, and service availability of your ADCs. While it is great to have all this information in dashboards, it sometimes makes sense to get notified when things are not working the way they should. This article demonstrates how to use Splunk’s alerting capabilities to configure just that.

Splunk Alerting

If you are not familiar with Splunk alerting here is the explanation in a nutshell: you create a search; every time it produces a result you will get notified. Several notification methods are available (email, webhooks and so on). More on this topic is available in Splunk’s documentation.

Example Searches for Citrix ADC Alerts

We created some example searches showing what is possible with uberAgent’s data and Splunk’s alerting capabilities. Feel free to customize them to your needs or use them as a starting point for your own creations.

Max Bandwidth per Appliance

The purchased license determines the maximum amount of data the ADC will process – it is a software limitation. Hence it is important to keep an eye on the bandwidth usage to react in a timely manner and buy a bigger license to not slow down the users. Here is an example where you will be notified when the bandwidth usage exceeded 900 megabytes.

| pivot `uA_DM_CitrixADC_AppliancePerformance` CitrixADC_AppliancePerformance
   max(ApplianceMBReceived) as "max(ApplianceMBReceived)"
   max(ApplianceMBSent) as "max(ApplianceMBSent)"
| eval "Max. network data volume (MB)" = round('max(ApplianceMBReceived)' + 'max(ApplianceMBSent)',1)
| where 'Max. network data volume (MB)' > 900

Too Many HA Status Changes

A Citrix ADC high-availability (HA) pair consists of a primary and a secondary node. If the primary fails the secondary takes over. Too many status changes indicate that there is something wrong. Maybe your hypervisor does live migration of VMs? It should not for Citrix ADC VPX machines!

| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory
   dc(LastHAStatusChange) as "#Status changes"
| where '#Status changes' > 5

The current status is collected once per day by default consequently let that search run for at least a few days.

Get Notified When SSL Cards Are Down

In a hardware Citrix ADC appliance, SSL cards handle the encryption and decryption of SSL traffic. If an SSL card is down the encryption/decryption rate drops which possibly affects user experience.

| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory                     
   latest(SSLCards) as "#SSL cards"
   latest(SSLCardsUp) as "#SSL cards up"
| eval "#SSL cards down" = '#SSL cards' - '#SSL cards up'
| where '#SSL cards down' != 0

Get Notified When the Configuration Was Not Saved Within 24 Hours After a Change

Citrix ADC configuration changes are active immediately, even if they have not been specifically saved. However, unsaved changes are lost when the appliance is rebooted. With this search, you will be notified after 24 hours if someone forgot to save the configuration.

| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory
   latest(LastConfigSaveTime) as LastConfigSaveTime
   latest(LastConfigChangedTime) as LastConfigChangedTime
| eval LastConfigSaveTime_epoch = strptime(LastConfigSaveTime, "%Y-%m-%d %H:%S:%M")
| eval LastConfigChangedTime_epoch = strptime(LastConfigChangedTime, "%Y-%m-%d %H:%S:%M")
| eval Offset =  LastConfigChangedTime_epoch - LastConfigSaveTime_epoch
| where Offset > 86400

Get Notified When a Certificate Expires Within Four Weeks

The all-time favorite. A certificate is installed but the team managing it forgot to update it. Never let that happen again with this search!

| pivot `uA_DM_CitrixADC_Certificate` CitrixADC_Certificate
   latest(CertExpirationDate) as CertExpirationDate
      CertName as "Certificate name"
| eval CertExpirationDate_epoch = strptime(CertExpirationDate, "%Y-%m-%d %H:%S:%M")
| eval CurrentTime_epoch = now() | convert ctime(tnow)
| eval Offset =  CertExpirationDate_epoch - CurrentTime_epoch
| where Offset > 2419200
| sort CertExpirationDate

List All IPs Where Management Access Is Enabled but It Is Not an NSIP

It is a common best practice to enable management only on the NetScaler IP (NSIP). Anyway, managing it through the Subnet IP (SNIP) is much more easy, right? Just don’t! Keep safe with this search:

| pivot `uA_DM_CitrixADC_Ip` CitrixADC_Ip
   latest(IpNetmask) as Netmask
   latest(IpType) as Type
   latest(IpvServer) as "Bound to vServer"
   latest(IpMgmtAccess) as "Mgmt. access"
      IpAddress as "IP address"
   filter HostName is "ns1"
| eval "Mgmt. access" = if('Mgmt. access'=1,"Yes","No")
| where Type != "NSIP" and 'Mgmt. access' = "Yes"
| table
   "IP address"
   "Bound to vServer"
   "Mgmt. access"