PHP implements open source ELK log analysis platform

With the continuous development of the Internet and information technology, log analysis has become an indispensable part of enterprise business, network security and system optimization processes. In the past, log analysis methods mainly relied on manual reading, filtering and analysis, which was difficult for large amounts of data. The emergence of log analysis platforms can process log data efficiently and accurately, further increasing the value of the data. This article will introduce an open source ELK log analysis platform implemented using PHP.

1. Introduction to ELK

ELK is the abbreviation of three open source software: Elasticsearch, Logstash, and Kibana. Elasticsearch is a Lucene-based search engine that can process large amounts of data and quickly query; Logstash is an open source log collection and processing tool that can collect, parse, filter and transform a variety of logs; Kibana is a data processing tool Visual and interactive tools that can quickly generate various charts and dashboards.

ELK has the characteristics of simple use, efficient performance, strong scalability, etc., and supports multiple data sources. It can help enterprises quickly build a powerful log analysis platform to monitor the running status of systems and applications.

2. PHP implements ELK

In the process of using ELK, we generally use Logstash to collect, parse and transform logs, and then store the data in Elasticsearch. After using Kibana Make a visual display. As a popular server-side scripting language, PHP can also collect and store logs by using Logstash and Elasticsearch libraries.

1. Install Logstash

The installation of Logstash is very simple. We can select the corresponding version through the download page of the official website, and then unzip it to the specified directory. For example, we can install it in a Linux system through the following command:

curl -L -O https://download.elastic.co/logstash/logstash/logstash-5.5.2.tar.gz
tar -zxvf logstash-5.5.2.tar.gz
cd logstash-5.5.2/bin/
./logstash -e 'input { stdin { } } output { stdout {} }'

After executing the above command, we can test whether Logstash is successfully installed through standard input. Of course, in order to better introduce the use of PHP to collect logs, we also need to install related libraries.

2. Install the Elasticsearch library

We use Composer to manage the dependencies of the PHP library. After installing Logstash, we can use the following command to install the Elasticsearch dependent library:

composer require elasticsearch/elasticsearch

3. Configure Logstash

Before using Logstash for log collection, we also need to configure the relevant parameters of Logstash. First, we need to define the input to Logstash. In the input configuration, we can use some very useful plug-ins, such as:

• file: used to read logs in files
• udp, tcp: used to read UDP and TCP protocol logs
• syslog: used to read system Syslog logs
• beats: can directly receive Beats protocol logs

Here, we use file plug-in to read log files on the server, parse and process them. For example:

input {
  file {
    path => "/var/log/apache2/access.log"
    type => "apache_access"
  }
}

Next, we need to configure Logstash filtering. Filtering can analyze and process logs, such as extracting specific fields, parsing IP addresses or URL addresses, etc. The following is a simple filter:

filter {
  if [type] == "apache_access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }

    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  }
}

Then, we can define the Logstash output. Output can output the processed data to Elasticsearch or other data storage media, such as databases, files, etc. The following is an output configuration:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "changeme"
  }
}

4. Use PHP to collect logs

After completing the above configuration, we can use PHP to collect logs. The following is a simple PHP script that can run in Linux or other UNIX-like environments:

<?php

require 'vendor/autoload.php';

use ElasticsearchClientBuilder;

$client = ClientBuilder::create()->build();

$log_path = '/var/log/apache2/access.log';
$log_index = 'apache_access';

if(!file_exists($log_path)) {
  echo "Log file '{$log_path}' not exists.";
  exit;
}

$file_size = filesize($log_path);
if($file_size == 0) {
  exit;
}

$lines = file($log_path);
if(empty($lines)) {
  exit;
}

foreach($lines as $line) {
  $log = [];
  $log['@timestamp'] = date('c');
  $log['message'] = $line;
  $log['type'] = $log_index;

  $params = [
    'body' => $log,
    'index' => 'logs',
    'type' => $log_index
  ];

  $response = $client->index($params);
}

In the above code, we first use the Elasticsearch client library to create a client instance. Then, we define a $log_path variable to specify the path of the log file to be read. Next, we use the file_exists() function to determine whether the file exists, the filesize() function to obtain the file size, and the file() function to read the file contents.

In the foreach loop, we traverse each line in the file and store the format of each line of log into the $log array. Here, we also store the log type and current timestamp into the $log array. Finally, we send the $log array to Elasticsearch using Elasticsearch's index() method.

3. Summary

Through the above introduction, we can see the workflow of the ELK log analysis platform. By using Logstash to collect, parse and transform logs, then storing the data in Elasticsearch, and using Kibana to visualize and interactively display the data, we can help us analyze log data quickly and efficiently. At the same time, PHP, as a popular server-side scripting language, can also use Logstash and Elasticsearch libraries for log collection and storage.

The above is the detailed content of PHP implements open source ELK log analysis platform. For more information, please follow other related articles on the PHP Chinese website!