Citrix ADC Monitoring & Alerting
uberAgent 5.2 introduced Citrix ADC monitoring (formerly NetScaler ADC). This functionality lets you check the appliance health, gateway data volume, and service availability of your ADCs. While it is great to have all this information in dashboards, it sometimes makes sense to get notified when things are not working the way they should. This article demonstrates how to use Splunk’s alerting capabilities to configure just that.
Splunk Alerting
If you are not familiar with Splunk alerting here is the explanation in a nutshell: you create a search; every time it produces a result you will get notified. Several notification methods are available (email, webhooks and so on). More on this topic is available in Splunk’s documentation.
Example Searches for Citrix ADC Alerts
We created some example searches showing what is possible with uberAgent’s data and Splunk’s alerting capabilities. Feel free to customize them to your needs or use them as a starting point for your own creations.
Max Bandwidth per Appliance
The purchased license determines the maximum amount of data the ADC will process – it is a software limitation. Hence it is important to keep an eye on the bandwidth usage to react in a timely manner and buy a bigger license to not slow down the users. Here is an example where you will be notified when the bandwidth usage exceeded 900 megabytes.
| pivot `uA_DM_CitrixADC_AppliancePerformance` CitrixADC_AppliancePerformance
max(ApplianceMBReceived) as "max(ApplianceMBReceived)"
max(ApplianceMBSent) as "max(ApplianceMBSent)"
splitrow
HostName
| eval "Max. network data volume (MB)" = round('max(ApplianceMBReceived)' + 'max(ApplianceMBSent)',1)
| where 'Max. network data volume (MB)' > 900
Too Many HA Status Changes
A Citrix ADC high-availability (HA) pair consists of a primary and a secondary node. If the primary fails the secondary takes over. Too many status changes indicate that there is something wrong. Maybe your hypervisor does live migration of VMs? It should not for Citrix ADC VPX machines!
| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory
dc(LastHAStatusChange) as "#Status changes"
splitrow
HostName
| where '#Status changes' > 5
The current status is collected once per day by default consequently let that search run for at least a few days.
Get Notified When SSL Cards Are Down
In a hardware Citrix ADC appliance, SSL cards handle the encryption and decryption of SSL traffic. If an SSL card is down the encryption/decryption rate drops which possibly affects user experience.
| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory
latest(SSLCards) as "#SSL cards"
latest(SSLCardsUp) as "#SSL cards up"
splitrow
HostName
| eval "#SSL cards down" = '#SSL cards' - '#SSL cards up'
| where '#SSL cards down' != 0
Get Notified When the Configuration Was Not Saved Within 24 Hours After a Change
Citrix ADC configuration changes are active immediately, even if they have not been specifically saved. However, unsaved changes are lost when the appliance is rebooted. With this search, you will be notified after 24 hours if someone forgot to save the configuration.
| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory
latest(LastConfigSaveTime) as LastConfigSaveTime
latest(LastConfigChangedTime) as LastConfigChangedTime
splitrow
HostName
| eval LastConfigSaveTime_epoch = strptime(LastConfigSaveTime, "%Y-%m-%d %H:%S:%M")
| eval LastConfigChangedTime_epoch = strptime(LastConfigChangedTime, "%Y-%m-%d %H:%S:%M")
| eval Offset = LastConfigChangedTime_epoch - LastConfigSaveTime_epoch
| where Offset > 86400
Get Notified When a Certificate Expires Within Four Weeks
The all-time favorite. A certificate is installed but the team managing it forgot to update it. Never let that happen again with this search!
| pivot `uA_DM_CitrixADC_Certificate` CitrixADC_Certificate
latest(CertExpirationDate) as CertExpirationDate
splitrow
CertName as "Certificate name"
| eval CertExpirationDate_epoch = strptime(CertExpirationDate, "%Y-%m-%d %H:%S:%M")
| eval CurrentTime_epoch = now() | convert ctime(tnow)
| eval Offset = CertExpirationDate_epoch - CurrentTime_epoch
| where Offset > 2419200
| sort CertExpirationDate
List All IPs Where Management Access Is Enabled but It Is Not an NSIP
It is a common best practice to enable management only on the NetScaler IP (NSIP). Anyway, managing it through the Subnet IP (SNIP) is much more easy, right? Just don’t! Keep safe with this search:
| pivot `uA_DM_CitrixADC_Ip` CitrixADC_Ip
latest(IpNetmask) as Netmask
latest(IpType) as Type
latest(IpvServer) as "Bound to vServer"
latest(IpMgmtAccess) as "Mgmt. access"
splitrow
IpAddress as "IP address"
filter HostName is "ns1"
| eval "Mgmt. access" = if('Mgmt. access'=1,"Yes","No")
| where Type != "NSIP" and 'Mgmt. access' = "Yes"
| table
"IP address"
Netmask
Type
"Bound to vServer"
"Mgmt. access"