Implementing an Order Processing System: Part Monitoring and Alerting

1. Introduction and Goals

Welcome to the fourth installment of our series on implementing a sophisticated order processing system! In our previous posts, we laid the foundation for our project, explored advanced Temporal workflows, and delved into advanced database operations. Today, we’re focusing on an equally crucial aspect of any production-ready system: monitoring and alerting.

Recap of Previous Posts

1. In Part 1, we set up our project structure and implemented a basic CRUD API.
2. In Part 2, we expanded our use of Temporal, implementing complex workflows and exploring advanced concepts.
3. In Part 3, we focused on advanced database operations, including optimization, sharding, and ensuring consistency in distributed systems.

Importance of Monitoring and Alerting in Microservices Architecture

In a microservices architecture, especially one handling complex processes like order management, effective monitoring and alerting are crucial. They allow us to:

1. Understand the behavior and performance of our system in real-time
2. Quickly identify and diagnose issues before they impact users
3. Make data-driven decisions for scaling and optimization
4. Ensure the reliability and availability of our services

Overview of Prometheus and its Ecosystem

Prometheus is an open-source systems monitoring and alerting toolkit. It’s become a standard in the cloud-native world due to its powerful features and extensive ecosystem. Key components include:

1. Prometheus Server : Scrapes and stores time series data
2. Client Libraries : Allow easy instrumentation of application code
3. Alertmanager : Handles alerts from Prometheus server
4. Pushgateway : Allows ephemeral and batch jobs to expose metrics
5. Exporters : Allow third-party systems to expose metrics to Prometheus

We’ll also be using Grafana, a popular open-source platform for monitoring and observability, to create dashboards and visualize our Prometheus data.

Goals for this Part of the Series

By the end of this post, you’ll be able to:

1. Set up Prometheus to monitor our order processing system
2. Implement custom metrics in our Go services
3. Create informative dashboards using Grafana
4. Set up alerting rules to notify us of potential issues
5. Monitor database performance and Temporal workflows effectively

Let’s dive in!

2. Theoretical Background and Concepts

Before we start implementing, let’s review some key concepts that will be crucial for our monitoring and alerting setup.

Observability in Distributed Systems

Observability refers to the ability to understand the internal state of a system by examining its outputs. In distributed systems like our order processing system, observability typically encompasses three main pillars:

1. Metrics : Numerical representations of data measured over intervals of time
2. Logs : Detailed records of discrete events within the system
3. Traces : Representations of causal chains of events across components

In this post, we’ll focus primarily on metrics, though we’ll touch on how these can be integrated with logs and traces.

Prometheus Architecture

Prometheus follows a pull-based architecture:

1. Data Collection : Prometheus scrapes metrics from instrumented jobs via HTTP
2. Data Storage : Metrics are stored in a time-series database on the local storage
3. Querying : PromQL allows flexible querying of this data
4. Alerting : Prometheus can trigger alerts based on query results
5. Visualization : While Prometheus has a basic UI, it’s often paired with Grafana for richer visualizations

Metrics Types in Prometheus

Prometheus offers four core metric types:

1. Counter : 増加するだけの累積メトリクス (例: 処理されたリクエストの数)
2. ゲージ : 上下する可能性のあるメトリクス (現在のメモリ使用量など)
3. ヒストグラム : 観測値をサンプリングし、構成可能なバケット内でカウントします (リクエスト期間など)
4. 概要 : ヒストグラムに似ていますが、スライディング タイム ウィンドウにわたって構成可能な分位数を計算します

PromQL の概要

PromQL (Prometheus Query Language) は、Prometheus データをクエリするための強力な関数型言語です。時系列データをリアルタイムで選択して集計できます。主な機能は次のとおりです:

• インスタント ベクトル セレクター
• 範囲ベクトルセレクター
• オフセット修飾子
• 集計演算子
• 二項演算子

ダッシュボードとアラートを構築するときに、PromQL クエリの例を見ていきます。

グラファナの概要

Grafana は、マルチプラットフォームのオープンソース分析およびインタラクティブな視覚化 Web アプリケーションです。 Prometheus もその 1 つである、サポートされているデータ ソースに接続すると、Web にチャート、グラフ、アラートを提供します。主な機能は次のとおりです:

• 柔軟なダッシュボードの作成
• 幅広い視覚化オプション
• アラート機能
• ユーザーの認証と認可
• 拡張性を備えたプラグイン システム

これらの概念を説明したので、監視および警告システムの実装を開始しましょう。

3. 注文処理システム用の Prometheus のセットアップ

注文処理システムを監視するために Prometheus を設定することから始めましょう。

Prometheus のインストールと構成

まず、Prometheus を docker-compose.yml ファイルに追加しましょう。

services:
  # ... other services ...

  prometheus:
    image: prom/prometheus:v2.30.3
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090

volumes:
  # ... other volumes ...
  prometheus_data: {}

次に、./prometheus ディレクトリに prometheus.yml ファイルを作成します。

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'order_processing_api'
    static_configs:
      - targets: ['order_processing_api:8080']

  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres_exporter:9187']

この構成は、Prometheus に、Prometheus 自体、注文処理 API、および Postgres エクスポーター (後でセットアップします) からメトリクスを収集するように指示します。

Go サービス用の Prometheus Exporters の実装

Go サービスからのメトリクスを公開するには、Prometheus クライアント ライブラリを使用します。まず、go.mod に追加します:

go get github.com/prometheus/client_golang

次に、メインの Go ファイルを変更してメトリクスを公開しましょう:

package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint", "status"},
    )

    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "Duration of HTTP requests in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "endpoint"},
    )
)

func init() {
    prometheus.MustRegister(httpRequestsTotal)
    prometheus.MustRegister(httpRequestDuration)
}

func main() {
    r := gin.Default()

    // Middleware to record metrics
    r.Use(func(c *gin.Context) {
        timer := prometheus.NewTimer(httpRequestDuration.WithLabelValues(c.Request.Method, c.FullPath()))
        c.Next()
        timer.ObserveDuration()
        httpRequestsTotal.WithLabelValues(c.Request.Method, c.FullPath(), string(c.Writer.Status())).Inc()
    })

    // Expose metrics endpoint
    r.GET("/metrics", gin.WrapH(promhttp.Handler()))

    // ... rest of your routes ...

    r.Run(":8080")
}

このコードは 2 つのメトリクスを設定します:

1. http_requests_total: HTTP リクエストの合計数を追跡するカウンター
2. http_request_duration_seconds: HTTP リクエストの継続時間を追跡するヒストグラム

動的環境向けのサービス検出のセットアップ

より動的な環境のために、Prometheus はさまざまなサービス検出メカニズムをサポートしています。たとえば、Kubernetes 上で実行している場合は、Kubernetes SD 構成を使用できます。

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

この構成は、適切なアノテーションを持つポッドからメトリクスを自動的に検出して収集します。

Prometheus データの保持とストレージの構成

Prometheus は、ローカル ファイル システム上の時系列データベースにデータを保存します。 Prometheus 構成で保持時間とストレージ サイズを構成できます:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

storage:
  tsdb:
    retention.time: 15d
    retention.size: 50GB

# ... rest of the configuration ...

この構成では、保存期間が 15 日間、最大ストレージ サイズが 50 GB に設定されます。

次のセクションでは、注文処理システムのカスタム指標の定義と実装について詳しく説明します。

4. カスタムメトリクスの定義と実装

Prometheus がセットアップされ、基本的な HTTP メトリクスが実装されたので、注文処理システムに固有のカスタム メトリクスを定義して実装しましょう。

注文処理システムのメトリクス スキーマの設計

メトリクスを設計するときは、システムからどのような洞察を得たいかを考えることが重要です。弊社の注文処理システムでは、以下を追跡する必要がある場合があります:

1. 注文作成率
2. 注文処理時間
3. 注文状況の分布
4. 支払い処理の成功/失敗率
5. 在庫更新操作
6. 発送手配時間

これらのメトリクスを実装しましょう:

package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    OrdersCreated = promauto.NewCounter(prometheus.CounterOpts{
        Name: "orders_created_total",
        Help: "The total number of created orders",
    })

    OrderProcessingTime = promauto.NewHistogram(prometheus.HistogramOpts{
        Name: "order_processing_seconds",
        Help: "Time taken to process an order",
        Buckets: prometheus.LinearBuckets(0, 30, 10), // 0-300 seconds, 30-second buckets
    })

    OrderStatusGauge = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "orders_by_status",
        Help: "Number of orders by status",
    }, []string{"status"})

    PaymentProcessed = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "payments_processed_total",
        Help: "The total number of processed payments",
    }, []string{"status"})

    InventoryUpdates = promauto.NewCounter(prometheus.CounterOpts{
        Name: "inventory_updates_total",
        Help: "The total number of inventory updates",
    })

    ShippingArrangementTime = promauto.NewHistogram(prometheus.HistogramOpts{
        Name: "shipping_arrangement_seconds",
        Help: "Time taken to arrange shipping",
        Buckets: prometheus.LinearBuckets(0, 60, 5), // 0-300 seconds, 60-second buckets
    })
)

Go サービスでのアプリケーション固有のメトリクスの実装

メトリクスを定義したので、サービスに実装してみましょう:

package main

import (
    "time"

    "github.com/yourusername/order-processing-system/metrics"
)

func createOrder(order Order) error {
    startTime := time.Now()

    // Order creation logic...

    metrics.OrdersCreated.Inc()
    metrics.OrderProcessingTime.Observe(time.Since(startTime).Seconds())
    metrics.OrderStatusGauge.WithLabelValues("pending").Inc()

    return nil
}

func processPayment(payment Payment) error {
    // Payment processing logic...

    if paymentSuccessful {
        metrics.PaymentProcessed.WithLabelValues("success").Inc()
    } else {
        metrics.PaymentProcessed.WithLabelValues("failure").Inc()
    }

    return nil
}

func updateInventory(item Item) error {
    // Inventory update logic...

    metrics.InventoryUpdates.Inc()

    return nil
}

func arrangeShipping(order Order) error {
    startTime := time.Now()

    // Shipping arrangement logic...

    metrics.ShippingArrangementTime.Observe(time.Since(startTime).Seconds())

    return nil
}

メトリックの名前付けとラベル付けのベスト プラクティス

メトリクスに名前を付けてラベルを付けるときは、次のベスト プラクティスを考慮してください。

1. Use a consistent naming scheme (e.g., __)
2. Use clear, descriptive names
3. Include units in the metric name (e.g., _seconds, _bytes)
4. Use labels to differentiate instances of a metric, but be cautious of high cardinality
5. Keep the number of labels manageable

Instrumenting Key Components: API Endpoints, Database Operations, Temporal Workflows

For API endpoints, we’ve already implemented basic instrumentation. For database operations, we can add metrics like this:

func (s *Store) GetOrder(ctx context.Context, id int64) (Order, error) {
    startTime := time.Now()
    defer func() {
        metrics.DBOperationDuration.WithLabelValues("GetOrder").Observe(time.Since(startTime).Seconds())
    }()

    // Existing GetOrder logic...
}

For Temporal workflows, we can add metrics in our activity implementations:

func ProcessOrderActivity(ctx context.Context, order Order) error {
    startTime := time.Now()
    defer func() {
        metrics.WorkflowActivityDuration.WithLabelValues("ProcessOrder").Observe(time.Since(startTime).Seconds())
    }()

    // Existing ProcessOrder logic...
}

5. Creating Dashboards with Grafana

Now that we have our metrics set up, let’s visualize them using Grafana.

Installing and Configuring Grafana

First, let’s add Grafana to our docker-compose.yml:

services:
  # ... other services ...

  grafana:
    image: grafana/grafana:8.2.2
    ports:
      - 3000:3000
    volumes:
      - grafana_data:/var/lib/grafana

volumes:
  # ... other volumes ...
  grafana_data: {}

Connecting Grafana to Our Prometheus Data Source

1. Access Grafana at http://localhost:3000 (default credentials are admin/admin)
2. Go to Configuration > Data Sources
3. Click “Add data source” and select Prometheus
4. Set the URL to http://prometheus:9090 (this is the Docker service name)
5. Click “Save & Test”

Designing Effective Dashboards for Our Order Processing System

Let’s create a dashboard for our order processing system:

1. Click “Create” > “Dashboard”
2. Add a new panel

For our first panel, let’s create a graph of order creation rate:

1. In the query editor, enter: rate(orders_created_total[5m])
2. Set the panel title to “Order Creation Rate”
3. Under Settings, set the unit to “orders/second”

Let’s add another panel for order processing time:

1. Add a new panel
2. Query: histogram_quantile(0.95, rate(order_processing_seconds_bucket[5m]))
3. Title: “95th Percentile Order Processing Time”
4. Unit: “seconds”

For order status distribution:

1. Add a new panel
2. Query: orders_by_status
3. Visualization: Pie Chart
4. Title: “Order Status Distribution”

Continue adding panels for other metrics we’ve defined.

Implementing Variable Templating for Flexible Dashboards

Grafana allows us to create variables that can be used across the dashboard. Let’s create a variable for time range:

1. Go to Dashboard Settings > Variables
2. Click “Add variable”
3. Name: time_range
4. Type: Interval
5. Values: 5m,15m,30m,1h,6h,12h,24h,7d

Now we can use this in our queries like this: rate(orders_created_total[$time_range])

Best Practices for Dashboard Design and Organization

1. Group related panels together
2. Use consistent color schemes
3. Include a description for each panel
4. Use appropriate visualizations for each metric type
5. Consider creating separate dashboards for different aspects of the system (e.g., Orders, Inventory, Shipping)

In the next section, we’ll set up alerting rules to notify us of potential issues in our system.

6. Implementing Alerting Rules

Now that we have our metrics and dashboards set up, let’s implement alerting to proactively notify us of potential issues in our system.

Designing an Alerting Strategy for Our System

When designing alerts, consider the following principles:

1. Alert on symptoms, not causes
2. Ensure alerts are actionable
3. Avoid alert fatigue by only alerting on critical issues
4. Use different severity levels for different types of issues

For our order processing system, we might want to alert on:

1. High error rate in order processing
2. Slow order processing time
3. Unusual spike or drop in order creation rate
4. Low inventory levels
5. High rate of payment failures

Implementing Prometheus Alerting Rules

Let’s create an alerts.yml file in our Prometheus configuration directory:

groups:
- name: order_processing_alerts
  rules:
  - alert: HighOrderProcessingErrorRate
    expr: rate(order_processing_errors_total[5m]) / rate(orders_created_total[5m]) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: High order processing error rate
      description: "Error rate is over the last 5 minutes"

  - alert: SlowOrderProcessing
    expr: histogram_quantile(0.95, rate(order_processing_seconds_bucket[5m])) > 300
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: Slow order processing
      description: "95th percentile of order processing time is over the last 5 minutes"

  - alert: UnusualOrderRate
    expr: abs(rate(orders_created_total[1h]) - rate(orders_created_total[1h] offset 1d)) > (rate(orders_created_total[1h] offset 1d) * 0.3)
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: Unusual order creation rate
      description: "Order creation rate has changed by more than 30% compared to the same time yesterday"

  - alert: LowInventory
    expr: inventory_level  0.1
    for: 15m
    labels:
      severity: critical
    annotations:
      summary: High payment failure rate
      description: "Payment failure rate is over the last 15 minutes"

Update your prometheus.yml to include this alerts file:

rule_files:
  - "alerts.yml"

Setting Up Alertmanager for Alert Routing and Grouping

Now, let’s set up Alertmanager to handle our alerts. Add Alertmanager to your docker-compose.yml:

services:
  # ... other services ...

  alertmanager:
    image: prom/alertmanager:v0.23.0
    ports:
      - 9093:9093
    volumes:
      - ./alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'

Create an alertmanager.yml in the ./alertmanager directory:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-notifications'

receivers:
- name: 'email-notifications'
  email_configs:
  - to: 'team@example.com'
    from: 'alertmanager@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'alertmanager@example.com'
    auth_identity: 'alertmanager@example.com'
    auth_password: 'password'

Update your prometheus.yml to point to Alertmanager:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

Configuring Notification Channels

In the Alertmanager configuration above, we’ve set up email notifications. You can also configure other channels like Slack, PagerDuty, or custom webhooks.

Implementing Alert Severity Levels and Escalation Policies

In our alerts, we’ve used severity labels. We can use these in Alertmanager to implement different routing or notification strategies based on severity:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-notifications'
  routes:
  - match:
      severity: critical
    receiver: 'pagerduty-critical'
  - match:
      severity: warning
    receiver: 'slack-warnings'

receivers:
- name: 'email-notifications'
  email_configs:
  - to: 'team@example.com'
- name: 'pagerduty-critical'
  pagerduty_configs:
  - service_key: '<your-pagerduty-service-key>'
- name: 'slack-warnings'
  slack_configs:
  - api_url: '<your-slack-webhook-url>'
    channel: '#alerts'

</your-slack-webhook-url></your-pagerduty-service-key>

7. Monitoring Database Performance

Monitoring database performance is crucial for maintaining a responsive and reliable system. Let’s set up monitoring for our PostgreSQL database.

Implementing the Postgres Exporter for Prometheus

First, add the Postgres exporter to your docker-compose.yml:

services:
  # ... other services ...

  postgres_exporter:
    image: wrouesnel/postgres_exporter:latest
    environment:
      DATA_SOURCE_NAME: "postgresql://user:password@postgres:5432/dbname?sslmode=disable"
    ports:
      - 9187:9187

Make sure to replace user, password, and dbname with your actual PostgreSQL credentials.

Key Metrics to Monitor for Postgres Performance

Some important PostgreSQL metrics to monitor include:

1. Number of active connections
2. Database size
3. Query execution time
4. Cache hit ratio
5. Replication lag (if using replication)
6. Transaction rate
7. Tuple operations (inserts, updates, deletes)

Creating a Database Performance Dashboard in Grafana

Let’s create a new dashboard for database performance:

1. Create a new dashboard in Grafana
2. Add a panel for active connections:
   • Query: pg_stat_activity_count{datname="your_database_name"}
   • Title: “Active Connections”
3. Add a panel for database size:
   • Query: pg_database_size_bytes{datname="your_database_name"}
   • Title: “Database Size”
   • Unit: bytes(IEC)
4. Add a panel for query execution time:
   • Query: rate(pg_stat_database_xact_commit{datname="your_database_name"}[5m]) + rate(pg_stat_database_xact_rollback{datname="your_database_name"}[5m])
   • Title: “Transactions per Second”
5. Add a panel for cache hit ratio:
   • Query: pg_stat_database_blks_hit{datname="your_database_name"} / (pg_stat_database_blks_hit{datname="your_database_name"} + pg_stat_database_blks_read{datname="your_database_name"})
   • Title: “Cache Hit Ratio”

Setting Up Alerts for Database Issues

Let’s add some database-specific alerts to our alerts.yml:

  - alert: HighDatabaseConnections
    expr: pg_stat_activity_count > 100
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: High number of database connections
      description: "There are active database connections"

  - alert: LowCacheHitRatio
    expr: pg_stat_database_blks_hit / (pg_stat_database_blks_hit + pg_stat_database_blks_read) 



<h2>
  
  
  8. Monitoring Temporal Workflows
</h2>

<p>Monitoring Temporal workflows is essential for ensuring the reliability and performance of our order processing system.</p>

<h3>
  
  
  Implementing Temporal Metrics in Our Go Services
</h3>

<p>Temporal provides a metrics client that we can use to expose metrics to Prometheus. Let’s update our Temporal worker to include metrics:<br>
</p>

<pre class="brush:php;toolbar:false">import (
    "go.temporal.io/sdk/client"
    "go.temporal.io/sdk/worker"
    "go.temporal.io/sdk/contrib/prometheus"
)

func main() {
    // ... other setup ...

    // Create Prometheus metrics handler
    metricsHandler := prometheus.NewPrometheusMetricsHandler()

    // Create Temporal client with metrics
    c, err := client.NewClient(client.Options{
        MetricsHandler: metricsHandler,
    })
    if err != nil {
        log.Fatalln("Unable to create Temporal client", err)
    }
    defer c.Close()

    // Create worker with metrics
    w := worker.New(c, "order-processing-task-queue", worker.Options{
        MetricsHandler: metricsHandler,
    })

    // ... register workflows and activities ...

    // Run the worker
    err = w.Run(worker.InterruptCh())
    if err != nil {
        log.Fatalln("Unable to start worker", err)
    }
}

Key Metrics to Monitor for Temporal Workflows

Important Temporal metrics to monitor include:

1. Workflow start rate
2. Workflow completion rate
3. Workflow execution time
4. Activity success/failure rate
5. Activity execution time
6. Task queue latency

Creating a Temporal Workflow Dashboard in Grafana

Let’s create a dashboard for Temporal workflows:

1. Create a new dashboard in Grafana
2. Add a panel for workflow start rate:
   • Query: rate(temporal_workflow_start_total[5m])
   • Title: “Workflow Start Rate”
3. Add a panel for workflow completion rate:
   • Query: rate(temporal_workflow_completed_total[5m])
   • Title: “Workflow Completion Rate”
4. Add a panel for workflow execution time:
   • Query: histogram_quantile(0.95, rate(temporal_workflow_execution_time_bucket[5m]))
   • Title: “95th Percentile Workflow Execution Time”
   • Unit: seconds
5. Add a panel for activity success rate:
   • Query: rate(temporal_activity_success_total[5m]) / (rate(temporal_activity_success_total[5m]) + rate(temporal_activity_fail_total[5m]))
   • Title: “Activity Success Rate”

Setting Up Alerts for Workflow Issues

Let’s add some Temporal-specific alerts to our alerts.yml:

  - alert: HighWorkflowFailureRate
    expr: rate(temporal_workflow_failed_total[15m]) / rate(temporal_workflow_completed_total[15m]) > 0.05
    for: 15m
    labels:
      severity: critical
    annotations:
      summary: High workflow failure rate
      description: "Workflow failure rate is over the last 15 minutes"

  - alert: LongRunningWorkflow
    expr: histogram_quantile(0.95, rate(temporal_workflow_execution_time_bucket[1h])) > 3600
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: Long-running workflows detected
      description: "95th percentile of workflow execution time is over 1 hour"

These alerts will help you detect issues with your Temporal workflows, such as high failure rates or unexpectedly long-running workflows.

In the next sections, we’ll cover some advanced Prometheus techniques and discuss testing and validation of our monitoring setup.

9. Advanced Prometheus Techniques

As our monitoring system grows more complex, we can leverage some advanced Prometheus techniques to improve its efficiency and capabilities.

Using Recording Rules for Complex Queries and Aggregations

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. This can significantly speed up the evaluation of dashboards and alerts.

Let’s add some recording rules to our Prometheus configuration. Create a rules.yml file:

groups:
- name: example_recording_rules
  interval: 5m
  rules:
  - record: job:order_processing_rate:5m
    expr: rate(orders_created_total[5m])

  - record: job:order_processing_error_rate:5m
    expr: rate(order_processing_errors_total[5m]) / rate(orders_created_total[5m])

  - record: job:payment_success_rate:5m
    expr: rate(payments_processed_total{status="success"}[5m]) / rate(payments_processed_total[5m])

Add this file to your Prometheus configuration:

rule_files:
  - "alerts.yml"
  - "rules.yml"

Now you can use these precomputed metrics in your dashboards and alerts, which can be especially helpful for complex queries that you use frequently.

Implementing Push Gateway for Batch Jobs and Short-Lived Processes

The Pushgateway allows you to push metrics from jobs that can’t be scraped, such as batch jobs or serverless functions. Let’s add a Pushgateway to our docker-compose.yml:

services:
  # ... other services ...

  pushgateway:
    image: prom/pushgateway
    ports:
      - 9091:9091

Now, you can push metrics to the Pushgateway from your batch jobs or short-lived processes. Here’s an example using the Go client:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/push"
)

func runBatchJob() {
    // Define a counter for the batch job
    batchJobCounter := prometheus.NewCounter(prometheus.CounterOpts{
        Name: "batch_job_processed_total",
        Help: "Total number of items processed by the batch job",
    })

    // Run your batch job and update the counter
    // ...

    // Push the metric to the Pushgateway
    pusher := push.New("http://pushgateway:9091", "batch_job")
    pusher.Collector(batchJobCounter)
    if err := pusher.Push(); err != nil {
        log.Printf("Could not push to Pushgateway: %v", err)
    }
}

Don’t forget to add the Pushgateway as a target in your Prometheus configuration:

scrape_configs:
  # ... other configs ...

  - job_name: 'pushgateway'
    static_configs:
      - targets: ['pushgateway:9091']

Federated Prometheus Setups for Large-Scale Systems

For large-scale systems, you might need to set up Prometheus federation, where one Prometheus server scrapes data from other Prometheus servers. This allows you to aggregate metrics from multiple Prometheus instances.

Here’s an example configuration for a federated Prometheus setup:

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="order_processing_api"}'
        - '{job="postgres_exporter"}'
    static_configs:
      - targets:
        - 'prometheus-1:9090'
        - 'prometheus-2:9090'

This configuration allows a higher-level Prometheus server to scrape specific metrics from other Prometheus servers.

Using Exemplars for Tracing Integration

Exemplars allow you to link metrics to trace data, providing a way to drill down from a high-level metric to a specific trace. This is particularly useful when integrating Prometheus with distributed tracing systems like Jaeger or Zipkin.

To use exemplars, you need to enable them in your Prometheus configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  exemplar_storage:
    enable: true

Then, when instrumenting your code, you can add exemplars to your metrics:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    orderProcessingDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "order_processing_duration_seconds",
            Help: "Duration of order processing in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"status"},
    )
)

func processOrder(order Order) {
    start := time.Now()
    // Process the order...
    duration := time.Since(start)

    orderProcessingDuration.WithLabelValues(order.Status).Observe(duration.Seconds(),
        prometheus.Labels{
            "traceID": getCurrentTraceID(),
        },
    )
}

This allows you to link from a spike in order processing duration directly to the trace of a slow order, greatly aiding in debugging and performance analysis.

10. Testing and Validation

Ensuring the reliability of your monitoring system is crucial. Let’s explore some strategies for testing and validating our Prometheus setup.

Unit Testing Metric Instrumentation

When unit testing your Go code, you can use the prometheus/testutil package to verify that your metrics are being updated correctly:

import (
    "testing"

    "github.com/prometheus/client_golang/prometheus/testutil"
)

func TestOrderProcessing(t *testing.T) {
    // Process an order
    processOrder(Order{ID: 1, Status: "completed"})

    // Check if the metric was updated
    expected := `
        # HELP order_processing_duration_seconds Duration of order processing in seconds
        # TYPE order_processing_duration_seconds histogram
        order_processing_duration_seconds_bucket{status="completed",le="0.005"} 1
        order_processing_duration_seconds_bucket{status="completed",le="0.01"} 1
        # ... other buckets ...
        order_processing_duration_seconds_sum{status="completed"} 0.001
        order_processing_duration_seconds_count{status="completed"} 1
    `
    if err := testutil.CollectAndCompare(orderProcessingDuration, strings.NewReader(expected)); err != nil {
        t.Errorf("unexpected collecting result:\n%s", err)
    }
}

Integration Testing for Prometheus Scraping

To test that Prometheus is correctly scraping your metrics, you can set up an integration test that starts your application, waits for Prometheus to scrape it, and then queries Prometheus to verify the metrics:

func TestPrometheusIntegration(t *testing.T) {
    // Start your application
    go startApp()

    // Wait for Prometheus to scrape (adjust the sleep time as needed)
    time.Sleep(30 * time.Second)

    // Query Prometheus
    client, err := api.NewClient(api.Config{
        Address: "http://localhost:9090",
    })
    if err != nil {
        t.Fatalf("Error creating client: %v", err)
    }

    v1api := v1.NewAPI(client)
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    result, warnings, err := v1api.Query(ctx, "order_processing_duration_seconds_count", time.Now())
    if err != nil {
        t.Fatalf("Error querying Prometheus: %v", err)
    }
    if len(warnings) > 0 {
        t.Logf("Warnings: %v", warnings)
    }

    // Check the result
    if result.(model.Vector).Len() == 0 {
        t.Errorf("Expected non-empty result")
    }
}

Load Testing and Observing Metrics Under Stress

It’s important to verify that your monitoring system performs well under load. You can use tools like hey or vegeta to generate load on your system while observing your metrics:

hey -n 10000 -c 100 http://localhost:8080/orders

While the load test is running, observe your Grafana dashboards and check that your metrics are updating as expected and that Prometheus is able to keep up with the increased load.

Validating Alerting Rules and Notification Channels

To test your alerting rules, you can temporarily adjust the thresholds to trigger alerts, or use Prometheus’s API to manually fire alerts:

curl -H "Content-Type: application/json" -d '{
  "alerts": [
    {
      "labels": {
        "alertname": "HighOrderProcessingErrorRate",
        "severity": "critical"
      },
      "annotations": {
        "summary": "High order processing error rate"
      }
    }
  ]
}' http://localhost:9093/api/v1/alerts

This will send a test alert to your Alertmanager, allowing you to verify that your notification channels are working correctly.

11. Challenges and Considerations

As you implement and scale your monitoring system, keep these challenges and considerations in mind:

Managing Cardinality in High-Dimensional Data

High cardinality can lead to performance issues in Prometheus. Be cautious when adding labels to metrics, especially labels with many possible values (like user IDs or IP addresses). Instead, consider using histogram metrics or reducing the cardinality by grouping similar values.

Scaling Prometheus for Large-Scale Systems

For large-scale systems, consider:

• Using the Pushgateway for batch jobs
• Implementing federation for large-scale setups
• Using remote storage solutions for long-term storage of metrics

Ensuring Monitoring System Reliability and Availability

Your monitoring system is critical infrastructure. Consider:

• Prometheus と Alertmanager の高可用性の実装
• 監視システムの監視 (メタ監視)
• Prometheus データを定期的にバックアップする

メトリクスとアラートのセキュリティに関する考慮事項

次のことを確認してください:

• Prometheus と Grafana へのアクセスは適切に保護されています
• 機密情報はメトリクスやアラートでは公開されません
• TLS は監視スタック内のすべての通信に使用されます

一時的な問題とアラートの羽ばたきへの対処

アラートノイズを軽減するには:

• アラート ルールで適切な時間枠を使用します
• Alertmanager でアラートのグループ化を実装する
• 関連するアラートに対してアラート抑制の使用を検討してください

12. 次のステップとパート 5 のプレビュー

この投稿では、Prometheus と Grafana を使用した注文処理システムの包括的な監視とアラートについて説明しました。私たちはカスタム指標を設定し、有益なダッシュボードを作成し、アラートを実装し、高度な技術と考慮事項を検討しました。

シリーズの次のパートでは、分散トレースとロギングに焦点を当てます。以下について説明します:

1. OpenTelemetry を使用した分散トレーシングの実装
2. ELK スタックを使用した集中ログのセットアップ
3. 効果的なデバッグのためのログ、トレース、メトリクスの関連付け
4. ログの集計と分析の実装
5. マイクロサービス アーキテクチャにログインするためのベスト プラクティス

今後も注文処理システムの強化を継続し、分散システムの動作とパフォーマンスについてより深い洞察を得ることに注力していきますので、ご期待ください!

---

助けが必要ですか?

困難な問題に直面していますか、それとも新しいアイデアやプロジェクトに関して外部の視点が必要ですか?お手伝いできます！大規模な投資を行う前にテクノロジーの概念実証を構築したい場合でも、難しい問題についてのガイダンスが必要な場合でも、私がお手伝いいたします。

提供されるサービス:

• 問題解決: 革新的なソリューションで複雑な問題に取り組みます。
• コンサルティング: プロジェクトに関する専門家のアドバイスと新鮮な視点を提供します。
• 概念実証: アイデアをテストおよび検証するための予備モデルを開発します。

私と協力することに興味がある場合は、hungaikevin@gmail.com まで電子メールでご連絡ください。

課題をチャンスに変えましょう!

The above is the detailed content of Implementing an Order Processing System: Part Monitoring and Alerting. For more information, please follow other related articles on the PHP Chinese website!