search
HomeOperation and MaintenanceNginxHow does Python obtain the Nginx log corresponding to the task request in real time?

Requirement Description

During the project requirement testing process, you need to send some use case requests to the Nginx server, and then check the corresponding Nginx log to determine whether there is characteristic content to determine whether the task is successfully executed. In order to improve efficiency, this process needs to be automated.

Practical environment

Python 3.6.5

Code design and implementation

#!/usr/bin/env python
# -*- coding:utf-8 -*-


"""
@CreateTime: 2021/06/26 9:05
@Author : shouke
"""


import time
import threading
import subprocess
from collections import deque


def collect_nginx_log():
    global nginx_log_queue
    global is_tasks_compete
    global task_status


    args = "tail -0f /usr/local/openresty/nginx/logs/access.log"
    while task_status != "req_log_got":
        with subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines = True) as proc:
            log_for_req = ""
            outs, errs = "", ""

            try:
                outs, errs = proc.communicate(timeout=2)
            except subprocess.TimeoutExpired:
                print("获取nginx日志超时,正在重试")
                proc.kill()
                try:
                    outs, errs = proc.communicate(timeout=5)
                except subprocess.TimeoutExpired:
                    print("获取nginx日志超时,再次超时,停止重试")
                    break
            finally:
                for line in outs.split("
"):
                    flag = ""client_ip":"10.118.0.77"" # 特征
                    if flag in line: # 查找包含特征内容的日志
                        log_for_req += line

                if task_status == "req_finished":
                    nginx_log_queue.append(log_for_req)
                    task_status = "req_log_got"



def run_tasks(task_list):
    """
    运行任务
    :param task_list 任务列表
    """

    global nginx_log_queue
    global is_tasks_compete
    global task_status


    for task in task_list:
        thread = threading.Thread(target=collect_nginx_log,
                                    name="collect_nginx_log")
        thread.start()
        time.sleep(1) # 执行任务前,让收集日志线程先做好准备

        print("正在执行任务:%s" % task.get("name"))

        # 执行Nginx任务请求
        # ...

        task_status = "req_finished"
        time_to_wait = 0.1
        while task_status != "req_log_got": # 请求触发的nginx日志收集未完成
            time.sleep(time_to_wait)
            time_to_wait += 0.01
        else:# 获取到用例请求触发的nginx日志
            if nginx_log_queue:
                nginx_log = nginx_log_queue.popleft()
                task_status = "req_ready"
                # 解析日志
                # do something here
                # ...
            else:
                print("存储请求日志的队列为空")
                # do something here
                # ...


if __name__ == "__main__":
    nginx_log_queue = deque()
    is_tasks_compete = False # 所有任务是否执行完成

    task_status = "req_ready" # req_ready,req_finished,req_log_got  # 存放执行次任务任务的一些状态
    print("###########################任务开始###########################")

    tast_list = [{"name":"test_task", "other":"..."}]
    run_tasks(tast_list)

    is_tasks_compete = True

    current_active_thread_num = len(threading.enumerate())
    while current_active_thread_num != 1:
        time.sleep(2)
        current_active_thread_num = len(threading.enumerate())
    print("###########################任务完成###########################")

Note:

1. Why is the above code not In one step, directly tail -0f /usr/local/openresty/nginx/logs/access.log | grep "Characteristic content"? This is because the Nginx log cannot be obtained by doing this

2. In practice, it was found that when proc.communicate(timeout=2) was executed for the first time to obtain the log, it was always impossible to obtain it. It will time out and require a second acquisition, and the timeout is set too small (in practice, I tried setting it to 1 second), which will also cause the Nginx log to be unable to be obtained during the second execution.

The above is the detailed content of How does Python obtain the Nginx log corresponding to the task request in real time?. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:亿速云. If there is any infringement, please contact admin@php.cn delete
NGINX and Apache: Understanding the Key DifferencesNGINX and Apache: Understanding the Key DifferencesApr 26, 2025 am 12:01 AM

NGINX and Apache each have their own advantages and disadvantages, and the choice should be based on specific needs. 1.NGINX is suitable for high concurrency scenarios because of its asynchronous non-blocking architecture. 2. Apache is suitable for low-concurrency scenarios that require complex configurations, because of its modular design.

NGINX Unit: Key Features and CapabilitiesNGINX Unit: Key Features and CapabilitiesApr 25, 2025 am 12:17 AM

NGINXUnit is an open source application server that supports multiple programming languages ​​and provides functions such as dynamic configuration, zero downtime updates and built-in load balancing. 1. Dynamic configuration: You can modify the configuration without restarting. 2. Multilingual support: compatible with Python, Go, Java, PHP, etc. 3. Zero downtime update: Supports application updates that do not interrupt services. 4. Built-in load balancing: Requests can be distributed to multiple application instances.

NGINX Unit vs. Other Application ServersNGINX Unit vs. Other Application ServersApr 24, 2025 am 12:14 AM

NGINXUnit is better than ApacheTomcat, Gunicorn and Node.js built-in HTTP servers, suitable for multilingual projects and dynamic configuration requirements. 1) Supports multiple programming languages, 2) Provides dynamic configuration reloading, 3) Built-in load balancing function, suitable for projects that require high scalability and reliability.

NGINX Unit: The Architecture and How It WorksNGINX Unit: The Architecture and How It WorksApr 23, 2025 am 12:18 AM

NGINXUnit improves application performance and manageability with its modular architecture and dynamic reconfiguration capabilities. 1) Modular design includes master processes, routers and application processes, supporting efficient management and expansion. 2) Dynamic reconfiguration allows seamless update of configuration at runtime, suitable for CI/CD environments. 3) Multilingual support is implemented through dynamic loading of language runtime, improving development flexibility. 4) High performance is achieved through event-driven models and asynchronous I/O, and remains efficient even under high concurrency. 5) Security is improved by isolating application processes and reducing the mutual influence between applications.

Using NGINX Unit: Deploying and Managing ApplicationsUsing NGINX Unit: Deploying and Managing ApplicationsApr 22, 2025 am 12:06 AM

NGINXUnit can be used to deploy and manage applications in multiple languages. 1) Install NGINXUnit. 2) Configure it to run different types of applications such as Python and PHP. 3) Use its dynamic configuration function for application management. Through these steps, you can efficiently deploy and manage applications and improve project efficiency.

NGINX vs. Apache: A Comparative Analysis of Web ServersNGINX vs. Apache: A Comparative Analysis of Web ServersApr 21, 2025 am 12:08 AM

NGINX is more suitable for handling high concurrent connections, while Apache is more suitable for scenarios where complex configurations and module extensions are required. 1.NGINX is known for its high performance and low resource consumption, and is suitable for high concurrency. 2.Apache is known for its stability and rich module extensions, which are suitable for complex configuration needs.

NGINX Unit's Advantages: Flexibility and PerformanceNGINX Unit's Advantages: Flexibility and PerformanceApr 20, 2025 am 12:07 AM

NGINXUnit improves application flexibility and performance with its dynamic configuration and high-performance architecture. 1. Dynamic configuration allows the application configuration to be adjusted without restarting the server. 2. High performance is reflected in event-driven and non-blocking architectures and multi-process models, and can efficiently handle concurrent connections and utilize multi-core CPUs.

NGINX vs. Apache: Performance, Scalability, and EfficiencyNGINX vs. Apache: Performance, Scalability, and EfficiencyApr 19, 2025 am 12:05 AM

NGINX and Apache are both powerful web servers, each with unique advantages and disadvantages in terms of performance, scalability and efficiency. 1) NGINX performs well when handling static content and reverse proxying, suitable for high concurrency scenarios. 2) Apache performs better when processing dynamic content and is suitable for projects that require rich module support. The selection of a server should be decided based on project requirements and scenarios.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.