How to Monitor the Length of Your Individual Azure Storage Queues

Azure storage queue monitoring: track the length of a single queue

In short: Azure storage queue lacks a single queue length indicator. However, you can use Azure SDK to query

and track the length of each queue. Use OpenTelemetry to send this data as a custom indicator. An example item can be used to automate this process through the Azure function to achieve reliable and scalable monitoring. approximate_message_count

If you use Azure to store queues and need (or just want) to monitor the length of each queue separately, then I have some bad news. ?

Azure only provides the total number of news indicators of the entire storage account through its built -in index function. Unfortunately, if you need to track the number of messages of a single queue, this makes these built -in indicators less useful.

The figure above shows the example of the built -in indicators. There are two queues at any given time, but we cannot recognize how much information among each queue. The screening function is disabled and there is no specific indicators for queue messages, as shown below.

Why is it important to monitor the length of a single queue?

Monitor the length of a single queue may be important, the reasons are as follows. For example, if you manage multiple queues, you may hope:

Tracking the toxic message queue

to avoid system interruption.

Monitor the pressure of the specific queue

to ensure that they handle the message efficiently.
• Management of scaling decision The growth of different loads by observing queues.
• Whether you are debugging or scaling, understanding the message count of each queue helps to maintain the health of the system.
Good news?
• Although Azure does not provide this function that is used in the box, there is a simple solution that this blog will guide you to complete this process.

How to get your indicators

As mentioned earlier, Azure did not provide a single storage queue length as a built -in indicator. Given that people have been asking this function in the past five years, for Microsoft, implementing it as a standard indicator may not be a simple task. Therefore, finding a solution may be your best choice. Naturally, this leads to such a question:

If the standard indicator does not provide this function, is there any other way to get it?

? >

Carefully check the Azure storage account SDK will find the queue attribute

, which allows you to access the required information -just through different methods. Knowing this, if you can use this data to track the length of the queue as an indicator, wouldn't it be good?

This is an idea: What if you do this? ?

You can query the length of each queue, create an index volume and regular update.

Let us gradually decompose it.

1. Get the queue length

Using Python SDK, you can easily retrieve the single length of the queue. See the following code fragment:

from azure.identity import DefaultAzureCredential
from azure.storage.queue import QueueClient

STORAGE_ACCOUNT_URL = "<storage-account-url>"
QUEUE_NAME = "<queue-name>"
STORAGE_ACCOUNT_KEY = "<key>"

credentials = STORAGE_ACCOUNT_KEY or DefaultAzureCredential()
client = QueueClient(
    STORAGE_ACCOUNT_URL,
    queue_name=QUEUE_NAME,
    credential=credentials,
)

try:
    properties = client.get_queue_properties()
    message_count = properties.approximate_message_count
    print(message_count)
except Exception as e:
    logger.exception(e)

Since SDK is built on the REST API, other SDKs also provide similar functions. The following is the reference of REST API and SDK in other languages:

• REST API-X-MS-Approximate-MESSAGES-COUNT: int-Value
• .NET -ApproximateMessageScount
• Java -GetApproximateMessageScount ()

2. Create quantity regulations and sending indicators

Next, you create a quantitative indicator to track the queue length.

Rules is an indicator type that measures a certain time point value, which makes it very suitable for tracking the changing queue length.

For this reason, we will use

OpenTelemetry , which is an open source observation framework, which is becoming more and more popular due to its multifunctionality in collecting indicators, tracking and logs. The following is an example of using OpenTelemetry to send queue length as a measure:

from opentelemetry.metrics import Meter, get_meter_provider

meter = get_meter_provider().get_meter(METER_NAME)

gauge = meter.create_gauge(
    name=gauge_name, description=gauge_description, unit="messages"
)

new_length = None

⋮ # 获取 approximate_message_count 并将其设置为 new_length 的代码

gauge.set(new_length)

OPENTELEMETRY is another advantage of its integrated integration with various observation tools (such as Prometheus, Azure Application Insights, Grafana, etc.).

3. Make it adapt to the production environment

Although the above method is very suitable for experiments, you may need a more powerful solution to adapt to the production environment. This is where elasticity and scalability play.

In the production environment, the continuous monitoring queue is not just the extraction indicator. You need to ensure that the system is reliable, can be expanded according to demand, and can deal with potential faults (such as network problems or a lot of data). For example, you do not want failure inquiries to stop your monitoring process.

If you are interested in understanding how to make it adapt to the production environment, I have created an example item: Azure-Storage-Queue-Monitor. This item packs all the contents we discuss to a Azure function running on the timer trigger. It processes elasticity, concurrency and scalability to ensure that you can reliably monitor the queue.

Conclusion

Now you have mastered the steps to track the length of a single queue and use it as a custom indicator. You can set this function for your environment. If you try it, please share your experience or improvement at any time -I am happy to hear your ideas and help you when you encounter any problems!

I wish you a happy monitoring! ?

The above is the detailed content of How to Monitor the Length of Your Individual Azure Storage Queues. For more information, please follow other related articles on the PHP Chinese website!