How to use Docker for container log analysis and exception monitoring

Docker is a popular containerization technology that packages an application and its dependencies into a container to run as a single portable application unit. This technology allows developers to easily deploy and manage applications in different environments. In practical applications, log analysis and exception monitoring of Docker containers are very necessary. This article will introduce how to use Docker for container log analysis and exception monitoring, including the following aspects:

1. Docker container log
2. Use the Docker log command to view the log
3. Use Logstash for log collection and analysis
4. Use Elasticsearch for data indexing and storage
5. Use Kibana for data visualization display

First we need to know about Docker containers log.

1. Docker container logs

Docker container logs record the operation information in the container, including: application output information, error information, access logs, system logs, etc. This information is very important for application operation and maintenance, tracking, exception handling, etc., so we need to collect and analyze the logs of Docker containers.

2. Use the Docker log command to view the log

Docker provides the log command, which can be used to view the log information output by the container. Using the log command, we can easily view the real-time output information of the running container and output this information to the console or save it to a file. The following is an example of using the log command to view container logs:

// 查看容器ID为xxx的日志
docker logs xxx

// 查看容器ID为xxx的日志，输出到控制台并实时更新
docker logs -f xxx 

// 查看容器ID为xxx的最近10条日志
docker logs --tail 10 xxx 

By using the log command, developers can easily view the real-time output information of the container and quickly determine the problem, but this method is suitable for a single For containers on the host, when the size of the container increases, it becomes difficult to manually view the logs, so log collection tools need to be used to automatically collect and analyze the logs.

3. Use Logstash for log collection and analysis

Logstash is an open source tool for collecting, filtering, converting and sending logs. Data is collected through input plug-ins and processed and converted by filters. data, and then the output plugin sends the processed data to a destination such as Elasticsearch, Kafka, Amazon S3, etc. In the log collection of Docker containers, we can use Logstash as a tool to collect and analyze logs. The following is an example of using Logstash for log collection and analysis:

1. Install Logstash

Download Logstash from the official website and unzip the file to use. The command to start Logstash is as follows:

cd logstash-7.15.1/bin
./logstash -f logstash.conf

2. Configure Logstash

To use Logstash as the log collection tool for the container, we need to configure the input plug-in and output plug-in in Logstash. The following is an example of the configuration file logstash.conf:

input {
  docker {
    endpoint => "unix:///var/run/docker.sock"
    container_id => "ALL"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
  }
  stdout {
    codec => "json_lines"
  }
}

The above configuration file means that we need to collect log information from all docker containers, filter and parse the data through the grok filter, and finally output the processed data into Elasticsearch.

4. Use Elasticsearch for data indexing and storage

Elasticsearch is a distributed open source search engine that can be used to search various types of documents. In the log collection of Docker containers, we will use Elasticsearch as the index and storage of data. The following is an example of using Elasticsearch for data indexing and storage:

1. Install Elasticsearch

Download Elasticsearch from the official website and unzip the file to use. The command to start Elasticsearch is as follows:

cd elasticsearch-7.15.1/bin
./elasticsearch

2. Configure Elasticsearch

Configure the name and node name of the ES cluster by modifying the elasticsearch.yml file. The following is a simple elasticsearch.yml configuration file example:

cluster.name: docker-cluster
node.name: es-node1
network.host: 0.0.0.0

The above configuration means that we create a cluster named docker-cluster, where the node name is es-node1, and the ES service is bound to all available on the network interface.

3. Create an index

In Elasticsearch, we need to first create an index for the data and specify the fields in the data. The sample code is as follows:

PUT /logstash-test
{
  "mappings": {
    "properties": {
      "host": {
        "type": "keyword"
      },
      "message": {
        "type": "text"
      },
      "path": {
        "type": "text"
      },
      "verb": {
        "type": "keyword"
      }
    }
  }
}

The above code creates an index named "logstash-test" in Elasticsearch and defines the fields and field types included in the index.

5. Use Kibana for data visualization display

Kibana is an open source data visualization tool that can be used to display data obtained from Elasticsearch. During the log collection process of Docker containers, we will use Kibana for data visualization display. The following is an example of using Kibana for data visualization display:

1. Install Kibana

Download Kibana from the official website and unzip the file to use. The command to start Kibana is as follows:

cd kibana-7.15.1/bin
./kibana

2. Index template settings

In Kibana, we need to set up the index template. The index template contains data field definitions and query analysis information. The sample code is as follows:

PUT _index_template/logstash-template
{
  "index_patterns": ["logstash-*"],
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "@version": { "type": "keyword" },
        "message": { "type": "text" },
        "path": { "type": "text" }
      }
    }
  }
}

The above code means that an index template named "logstash-template" is created and applied to indexes whose names start with "logstash-*".

3. Data visualization

На панели плагинов Kibana вы можете устанавливать визуальные шаблоны и управлять ими. С помощью панели мы можем легко создавать различные типы визуальных диаграмм, такие как линейные диаграммы, гистограммы, круговые диаграммы и т. д.

Подводя итог, в этой статье рассказывается, как использовать Docker для анализа журналов контейнеров и мониторинга исключений, а также приводятся конкретные примеры кода. Сам Docker предоставляет команду log для просмотра журналов контейнера, но просмотр журналов вручную становится сложнее по мере увеличения масштаба контейнера. Используя такие инструменты, как Logstash, Elasticsearch и Kibana, мы можем автоматически собирать и анализировать журналы контейнера и отображать рабочее состояние контейнера, что очень полезно для работы и обслуживания приложений, а также для обработки сбоев.

The above is the detailed content of How to use Docker for container log analysis and exception monitoring. For more information, please follow other related articles on the PHP Chinese website!