


How does Python obtain the Nginx log corresponding to the task request in real time?
Requirement Description
During the project requirement testing process, you need to send some use case requests to the Nginx server, and then check the corresponding Nginx log to determine whether there is characteristic content to determine whether the task is successfully executed. In order to improve efficiency, this process needs to be automated.
Practical environment
Python 3.6.5
Code design and implementation
#!/usr/bin/env python # -*- coding:utf-8 -*- """ @CreateTime: 2021/06/26 9:05 @Author : shouke """ import time import threading import subprocess from collections import deque def collect_nginx_log(): global nginx_log_queue global is_tasks_compete global task_status args = "tail -0f /usr/local/openresty/nginx/logs/access.log" while task_status != "req_log_got": with subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines = True) as proc: log_for_req = "" outs, errs = "", "" try: outs, errs = proc.communicate(timeout=2) except subprocess.TimeoutExpired: print("获取nginx日志超时,正在重试") proc.kill() try: outs, errs = proc.communicate(timeout=5) except subprocess.TimeoutExpired: print("获取nginx日志超时,再次超时,停止重试") break finally: for line in outs.split(" "): flag = ""client_ip":"10.118.0.77"" # 特征 if flag in line: # 查找包含特征内容的日志 log_for_req += line if task_status == "req_finished": nginx_log_queue.append(log_for_req) task_status = "req_log_got" def run_tasks(task_list): """ 运行任务 :param task_list 任务列表 """ global nginx_log_queue global is_tasks_compete global task_status for task in task_list: thread = threading.Thread(target=collect_nginx_log, name="collect_nginx_log") thread.start() time.sleep(1) # 执行任务前,让收集日志线程先做好准备 print("正在执行任务:%s" % task.get("name")) # 执行Nginx任务请求 # ... task_status = "req_finished" time_to_wait = 0.1 while task_status != "req_log_got": # 请求触发的nginx日志收集未完成 time.sleep(time_to_wait) time_to_wait += 0.01 else:# 获取到用例请求触发的nginx日志 if nginx_log_queue: nginx_log = nginx_log_queue.popleft() task_status = "req_ready" # 解析日志 # do something here # ... else: print("存储请求日志的队列为空") # do something here # ... if __name__ == "__main__": nginx_log_queue = deque() is_tasks_compete = False # 所有任务是否执行完成 task_status = "req_ready" # req_ready,req_finished,req_log_got # 存放执行次任务任务的一些状态 print("###########################任务开始###########################") tast_list = [{"name":"test_task", "other":"..."}] run_tasks(tast_list) is_tasks_compete = True current_active_thread_num = len(threading.enumerate()) while current_active_thread_num != 1: time.sleep(2) current_active_thread_num = len(threading.enumerate()) print("###########################任务完成###########################")
Note:
1. Why is the above code not In one step, directly tail -0f /usr/local/openresty/nginx/logs/access.log | grep "Characteristic content"
? This is because the Nginx log cannot be obtained by doing this
2. In practice, it was found that when proc.communicate(timeout=2)
was executed for the first time to obtain the log, it was always impossible to obtain it. It will time out and require a second acquisition, and the timeout
is set too small (in practice, I tried setting it to 1 second
), which will also cause the Nginx log to be unable to be obtained during the second execution.
The above is the detailed content of How does Python obtain the Nginx log corresponding to the task request in real time?. For more information, please follow other related articles on the PHP Chinese website!

NGINX and Apache each have their own advantages and disadvantages, and the choice should be based on specific needs. 1.NGINX is suitable for high concurrency scenarios because of its asynchronous non-blocking architecture. 2. Apache is suitable for low-concurrency scenarios that require complex configurations, because of its modular design.

NGINXUnit is an open source application server that supports multiple programming languages and provides functions such as dynamic configuration, zero downtime updates and built-in load balancing. 1. Dynamic configuration: You can modify the configuration without restarting. 2. Multilingual support: compatible with Python, Go, Java, PHP, etc. 3. Zero downtime update: Supports application updates that do not interrupt services. 4. Built-in load balancing: Requests can be distributed to multiple application instances.

NGINXUnit is better than ApacheTomcat, Gunicorn and Node.js built-in HTTP servers, suitable for multilingual projects and dynamic configuration requirements. 1) Supports multiple programming languages, 2) Provides dynamic configuration reloading, 3) Built-in load balancing function, suitable for projects that require high scalability and reliability.

NGINXUnit improves application performance and manageability with its modular architecture and dynamic reconfiguration capabilities. 1) Modular design includes master processes, routers and application processes, supporting efficient management and expansion. 2) Dynamic reconfiguration allows seamless update of configuration at runtime, suitable for CI/CD environments. 3) Multilingual support is implemented through dynamic loading of language runtime, improving development flexibility. 4) High performance is achieved through event-driven models and asynchronous I/O, and remains efficient even under high concurrency. 5) Security is improved by isolating application processes and reducing the mutual influence between applications.

NGINXUnit can be used to deploy and manage applications in multiple languages. 1) Install NGINXUnit. 2) Configure it to run different types of applications such as Python and PHP. 3) Use its dynamic configuration function for application management. Through these steps, you can efficiently deploy and manage applications and improve project efficiency.

NGINX is more suitable for handling high concurrent connections, while Apache is more suitable for scenarios where complex configurations and module extensions are required. 1.NGINX is known for its high performance and low resource consumption, and is suitable for high concurrency. 2.Apache is known for its stability and rich module extensions, which are suitable for complex configuration needs.

NGINXUnit improves application flexibility and performance with its dynamic configuration and high-performance architecture. 1. Dynamic configuration allows the application configuration to be adjusted without restarting the server. 2. High performance is reflected in event-driven and non-blocking architectures and multi-process models, and can efficiently handle concurrent connections and utilize multi-core CPUs.

NGINX and Apache are both powerful web servers, each with unique advantages and disadvantages in terms of performance, scalability and efficiency. 1) NGINX performs well when handling static content and reverse proxying, suitable for high concurrency scenarios. 2) Apache performs better when processing dynamic content and is suitable for projects that require rich module support. The selection of a server should be decided based on project requirements and scenarios.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

SublimeText3 Linux new version
SublimeText3 Linux latest version

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.
