Log collection system implemented in PHP

This article mainly introduces the log collection system implemented in PHP, which has certain reference value. Now I share it with everyone. Friends in need can refer to it

Recent business involves logs of remote servers Collecting requirements, out of the idea of ​​limiting the expansion of the technology stack, implemented it using PHP.

There are some small points that need attention during the implementation process, which are recorded as follows:

1. Active acquisition. Because There are many servers. If you use an architecture such as Flume, you need to install software on each server, which incurs operation and maintenance costs. Therefore, we use the method of active acquisition by the collector. There is no need to install software on the producer (server) .

2.SSH connection. Each server is configured with SSH connection permissions. You can use PHP's ssh2 extension to remotely connect and access server content.

3. Server log structure Unified. The log files on each server are placed according to the same directory rules to simplify the program logic.

4. CLI operation. Collection is a continuously running program, using CLI mode. Please note that the INI file problem.

5. SSH connection exception. Sometimes, due to network problems, SSH connection or verification fails, just delay and try again.

6 .Log truncation and compression. Usually, our operation and maintenance will truncate and compress logs at a fixed time every day, so there are two types of files that need to be read: compressed and uncompressed logs, which need to be processed separately.

7. Timestamps in logs. Timestamps in seconds are not enough to distinguish requests. We increase $msec to measure in milliseconds. Within the same millisecond, the same IP source and the same UA can be considered A request.

8. Read the directory. Use readdir to read the remote directory in SSH format, readdir("ssh2.sft://..."); to filter out unnecessary After the file, sort it by the file creation time and process it one by one.

9. Read the compressed file. If you use file_get_contents, the interface will become unresponsive for a long time. I used fopen and fread to read it step by step. Read 8K at a time (No matter how big it is, it is useless). After each reading a certain number of times, a progress display is output.

10. Compressed file caching. After reading successfully, save it to the cache directory for backup and next use. If the program errors or reruns, check the cache directory first. If there is a cache file, there is no need to read it from the network.

11. Decompress. Just use gzdecode. This will cause PHP The memory needs to increase dramatically. Adjust PHP.INI to expand the memory limit.

12. Record the completion of compression log processing. After processing a compressed file, record it in the database. After the PHP program is run, it will No need to repeat processing.

13. Uncompressed log processing. Uncompressed logs indicate that this log is still growing. No caching is required. Use database records and current file pointers (use ftell, fseek ). Record the file creation date.

14. Uncompressed log judgment. When the file date is different from the recorded date, or the file is smaller than the file size in the record, it means that the file has been updated and needs to be Reset the file pointer.

Otherwise, you can directly locate (fseek) to continue from the last processed position.

15. Log line decomposition. Use regular rules, that is Yes, it can be distinguished according to spaces and delimiters. You can also use the logParser third-party class library to process. In order to save memory overhead, you can use the Iterator coroutine mode to return line by line.

16. Log Judgment. Read the last log timestamp (milliseconds) and IP, UA of each server in advance.

17. Log saving. I used MYSQL to save the log. Each line of log is executed once MYSQL will waste a lot of running time. You can accumulate 4,000 rows and insert them all at once.

18. Error handling. In addition to the SSH connection failure, half a line of logs will be read, causing the decomposition to fail. At this time, it will also throw An exception occurs. Catch it by the main program and run it again.

Related recommendations:

php output log

PHP writes log function

php implements a log function

The above is the detailed content of Log collection system implemented in PHP. For more information, please follow other related articles on the PHP Chinese website!