Redis stores logs and popular articles

A variety of data structures can be implemented using the Redis list data type, which can be regarded as an index array in PHP. It can implement a variety of data structures such as stacks, queues, and message queues. Today, I would like to introduce to you how to use redis to save system logs and popular article lists.

Storing logs

As we all know, nginx logs will not be automatically cut by default. They will always be stored in a file and appended to. We need to do the log cutting operation ourselves. In addition to nginx, logs are used in many places. When something goes wrong, logs are one of the main ways we look for clues.

We now plan to write the system logs to redis. Daily logs will be recorded in a list to prevent a single log file from being too large.

The basic idea is to write daily log information into a separate list, and then do a scheduled task. The function of the scheduled task is to take out the log list from 1 month ago and persist it. to a text file, and then delete the log list from 1 month ago in redis to prevent redis from occupying too much memory.

You can use the compression function to compress log information and reduce memory usage. In addition, maintain a list to store the key names of the log list to facilitate retrieving the key names of the log list. The pseudocode for storing logs is as follows:

$log = ... // 日志信息
// 日志列表键名
$key = 'log:'.strtotime(date('Y-m-d'));

// 维护一个键名列表
if (!$redis->exists($key)) {
 $listlogkey = 'log:key';
 $redis->rpush($listlogkey, $key);
}

// 日志信息存放到redis中
$redis->rpush($key, $log);

The scheduled task code is as follows:

$lastMonth = strtotime("-30 day");

while ($logkey =  $redis->lpop('log:key')) {
    $logTime = explode(':', $logkey)[1];
    
    if ($logTime < $lastMonth) {
        // 从日志列表里去日志信息，一次取50条
        for ($start = 0, $end = 49;true;$start +=50, $end+=50) {
            $logs = $redis->lrange($logkey, $start, $end);
            if (!$logs) break;
            // 将日志信息解压缩，然后追加写入文本文件中
             ……
             
            // 删除该日志列表
            $redis->del($logkey);
        }   
    } else {
        // 一个月之内的，重新push到左侧
        $redis->lpush(&#39;log:key&#39;, $logkey);
        exit;
    }
}

There are a few points to note here. If the persistent log fails, or it is a log within the past month, you need to Re-push the log list key name from the left. In addition, when fetching logs from the log list, do not fetch them all at once, as this may easily lead to redis blocking. Each time, take a certain number (such as 50) and take them out in a loop.

Storing popular news IDs

I won’t post the code here, but mainly talk about the ideas. I used to create a system with a menu function, including today's hottest, hottest week, and hottest in January. At that time, our website had quite a lot of traffic, but within a few days, the website went down. The reason is that mysql's slow query problem. Because this piece of SQL includes grouping, COUNT(), conditional judgment, etc.

Let me tell you about our solution: write a mysql stored procedure and call the stored procedure regularly. The function of this stored procedure is to filter out the most popular articles today, this week, and January, take 100 article IDs respectively, and store their article IDs in the redis queue. For the most popular articles, we only display the top 100. In this way, our system will have no slow queries.

The above is the detailed content of Redis stores logs and popular articles. For more information, please follow other related articles on the PHP Chinese website!