Poison Queue Monitoring with Azure Functions
Some time ago I started researching about the options to monitor poison queues on Azure Storage in order to detect processing problems. I stumbled upon the great article by John Downs inspiring me to create a similar solution for a customer.
The basic idea is having a timer triggered Azure Function that “scans” for poison queues in the Azure Storage and checking the item count in the queues. This count is than written as a metric to Application Insights where this values can be monitored and even alerts created on.
How to set up
Setting up the whole solution is pretty easy and can be done within 30 minutes at all. In case you want to use an existing instance of Application Insights to track the metrics in, just pick up the instrumentation key and configure it later in your function settings (see below). Otherwise create a new App Insights instance and pick that instrumentation key.
Creating the Azure Function
You can either create a new Azure Function App in Azure or use an existing one (as in my case).
The code for the function may look like this (in case you use the pre-compiled functions template in Visual Studio)
Source code available as gist.
In order to get the function running we need to set up some environment variables in Azure:
- The APPINSIGHTS_INSTRUMENTATIONKEY contains the instrumentation key of your App Insights instance you want to track the metrics on
- The MonitorQueueStorage settings has the connectionstring to Azure Storage which contains the queues to be monitored.
You can adjust the timer value for your needs in the function code:
[TimerTrigger(“0 */5 * * * *”)] (means every 5 minutes)
Depending on your needs to have a more or less fine grained monitoring result set the value higher or lower.
Creating alerts in Application Insights
There is a pretty simple way to get automatic notification when there are items in your poison queues. You can even start an external process like an Azure Function (via WebHook) or start an instance of a Logic App. All done by AppInsights Alerts.
- Navigate in the Azure Portal to the ApplicationInsights instance where the metrics are stored in.
- Navigate to “Alerts” on the menu.
- Click on “Add metric alert” on the right side.
- Enter the information about the
– Alert name and description.
– Subscription and Resource.
– Metric and condition.
– Threshold and period
– and the notification options.
Your alert is now active and fires when on of your poison queues raised the item threshold.
Thanks for a great article. I have implemented an Azure function and an Azure alert as you describe. However, the alert was only fired once even though there are items in the poison queue. I suspect it is related to the state of the monitor condition. On the first alert the monitor condition is “fired”. As I understand a new alert is only fired when the monitor condition is changed to ‘Resolved”. But I have been unable to change the monitor condition. I thought it would happen automatically when the metric was less than the threshold but that does not seems to be the case.
Do you know how to change the monitor condition state?
I do not use Classic Alerts any more. On the new alerts the resolve happens based on the settings for Period and Frequency in the section „ Evaluated based on“. Hope that helps.