Poison Queue Monitoring with Azure Functions
Some time ago I started researching about the options to monitor poison queues on Azure Storage in order to detect processing problems. I stumbled upon the great article by John Downs inspiring me to create a similar solution for a customer.
The basic idea is having a timer triggered Azure Function that “scans” for poison queues in the Azure Storage and checking the item count in the queues. This count is than written as a metric to Application Insights where this values can be monitored and even alerts created on.
How to set up
Setting up the whole solution is pretty easy and can be done within 30 minutes at all. In case you want to use an existing instance of Application Insights to track the metrics in, just pick up the instrumentation key and configure it later in your function settings (see below). Otherwise create a new App Insights instance and pick that instrumentation key.
Creating the Azure Function
You can either create a new Azure Function App in Azure or use an existing one (as in my case).
The code for the function may look like this (in case you use the pre-compiled functions template in Visual Studio)
In order to get the function running we need to set up some environment variables in Azure:
- The APPINSIGHTS_INSTRUMENTATIONKEY contains the instrumentation key of your App Insights instance you want to track the metrics on
- The MonitorQueueStorage settings has the connectionstring to Azure Storage which contains the queues to be monitored.
You can adjust the timer value for your needs in the function code:
[TimerTrigger(“0 */5 * * * *”)] (means every 5 minutes)
Depending on your needs to have a more or less fine grained monitoring result set the value higher or lower.
Creating alerts in Application Insights
There is a pretty simple way to get automatic notification when there are items in your poison queues. You can even start an external process like an Azure Function (via WebHook) or start an instance of a Logic App. All done by AppInsights Alerts.
- Navigate in the Azure Portal to the ApplicationInsights instance where the metrics are stored in.
- Navigate to “Alerts” on the menu.
- Click on “Add metric alert” on the right side.
- Enter the information about the
– Alert name and description.
– Subscription and Resource.
– Metric and condition.
– Threshold and period
– and the notification options.
Your alert is now active and fires when on of your poison queues raised the item threshold.