Citrix ADC Monitoring & Alerting
uberAgent 5.2 introduced Citrix ADC monitoring (formerly NetScaler ADC). This functionality lets you check the appliance health, gateway data volume, and service availability of your ADCs. While it is great to have all this information in dashboards, it sometimes makes sense to get notified when things are not working the way they should. This article demonstrates how to use Splunk’s alerting capabilities to configure just that.
If you are not familiar with Splunk alerting here is the explanation in a nutshell: you create a search; every time it produces a result you will get notified. Several notification methods are available (email, webhooks and so on). More on this topic is available in Splunk’s documentation.
We created some example searches showing what is possible with uberAgent’s data and Splunk’s alerting capabilities. Feel free to customize them to your needs or use them as a starting point for your own creations.
The purchased license determines the maximum amount of data the ADC will process – it is a software limitation. Hence it is important to keep an eye on the bandwidth usage to react in a timely manner and buy a bigger license to not slow down the users. Here is an example where you will be notified when the bandwidth usage exceeded 900 megabytes.
| pivot `uA_DM_CitrixADC_AppliancePerformance` CitrixADC_AppliancePerformance max(ApplianceMBReceived) as "max(ApplianceMBReceived)" max(ApplianceMBSent) as "max(ApplianceMBSent)" splitrow HostName | eval "Max. network data volume (MB)" = round('max(ApplianceMBReceived)' + 'max(ApplianceMBSent)',1) | where 'Max. network data volume (MB)' > 900
A Citrix ADC high-availability (HA) pair consists of a primary and a secondary node. If the primary fails the secondary takes over. Too many status changes indicate that there is something wrong. Maybe your hypervisor does live migration of VMs? It should not for Citrix ADC VPX machines!
| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory dc(LastHAStatusChange) as "#Status changes" splitrow HostName | where '#Status changes' > 5
The current status is collected once per day by default consequently let that search run for at least a few days.
In a hardware Citrix ADC appliance, SSL cards handle the encryption and decryption of SSL traffic. If an SSL card is down the encryption/decryption rate drops which possibly affects user experience.
| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory latest(SSLCards) as "#SSL cards" latest(SSLCardsUp) as "#SSL cards up" splitrow HostName | eval "#SSL cards down" = '#SSL cards' - '#SSL cards up' | where '#SSL cards down' != 0
Citrix ADC configuration changes are active immediately, even if they have not been specifically saved. However, unsaved changes are lost when the appliance is rebooted. With this search, you will be notified after 24 hours if someone forgot to save the configuration.
| pivot `uA_DM_CitrixADC_ApplianceInventory` CitrixADC_ApplianceInventory latest(LastConfigSaveTime) as LastConfigSaveTime latest(LastConfigChangedTime) as LastConfigChangedTime splitrow HostName | eval LastConfigSaveTime_epoch = strptime(LastConfigSaveTime, "%Y-%m-%d %H:%S:%M") | eval LastConfigChangedTime_epoch = strptime(LastConfigChangedTime, "%Y-%m-%d %H:%S:%M") | eval Offset = LastConfigChangedTime_epoch - LastConfigSaveTime_epoch | where Offset > 86400
The all-time favorite. A certificate is installed but the team managing it forgot to update it. Never let that happen again with this search!
| pivot `uA_DM_CitrixADC_Certificate` CitrixADC_Certificate latest(CertExpirationDate) as CertExpirationDate splitrow CertName as "Certificate name" | eval CertExpirationDate_epoch = strptime(CertExpirationDate, "%Y-%m-%d %H:%S:%M") | eval CurrentTime_epoch = now() | convert ctime(tnow) | eval Offset = CertExpirationDate_epoch - CurrentTime_epoch | where Offset > 2419200 | sort CertExpirationDate
It is a common best practice to enable management only on the NetScaler IP (NSIP). Anyway, managing it through the Subnet IP (SNIP) is much more easy, right? Just don’t! Keep safe with this search:
| pivot `uA_DM_CitrixADC_Ip` CitrixADC_Ip latest(IpNetmask) as Netmask latest(IpType) as Type latest(IpvServer) as "Bound to vServer" latest(IpMgmtAccess) as "Mgmt. access" splitrow IpAddress as "IP address" filter HostName is "ns1" | eval "Mgmt. access" = if('Mgmt. access'=1,"Yes","No") | where Type != "NSIP" and 'Mgmt. access' = "Yes" | table "IP address" Netmask Type "Bound to vServer" "Mgmt. access"