search
HomeOperation and MaintenanceNginxHow to integrate Kafka with Nginx

How to integrate Kafka with Nginx

May 21, 2023 pm 12:07 PM
nginxkafka

Background

nginx-kafka-module is a plug-in for nginx. It can integrate kafka into nginx to facilitate the collection of front-end page buried point data in web projects. If the front-end page has set buried points, that is Some of the user's access and request data can be sent directly to the message middleware kafka through http requests. The backend can consume the messages in kafka through the program to perform real-time calculations. For example, sparkstream is used to consume data in Kafka in real time to analyze user pv, uv, some user behaviors and the funnel model conversion rate of the page, so as to better optimize the system or conduct real-time dynamic analysis of visiting users.

Specific integration steps

1. Install git

yum install -y git

2. Switch to the /usr/local/src directory, and then clone the kafka c client source code Go to local

cd /usr/local/src
git clone https://github.com/edenhill/librdkafka

3. Enter librdkafka, and then compile it

cd librdkafka
yum install -y gcc gcc-c++ pcre-devel zlib-devel
./configure
make && make install

4. Install nginx integration kafka plug-in, enter /usr/local/src, clone nginx integrates the source code of kafka

cd /usr/local/src
git clone https://github.com/brg-liuwei/ngx_kafka_module

5. Enter the nginx source package directory (compile nginx, and then compile the plug-in at the same time)

cd /usr/local/src/nginx-1.12.2
./configure --add-module=/usr/local/src/ngx_kafka_module/
make && make install

6. Modify nginx Configuration file: Set a location and kafaka topic

#添加配置(2处)
kafka;
kafka_broker_list f1:9092 f2:9092 f3:9092;

location = /kafka/access {
kafka_topic access888;
}

as shown below:

How to integrate Kafka with Nginx

How to integrate Kafka with Nginx

7. Start zk and kafka clusters (create topic)

zkserver.sh start
kafka-server-start.sh -daemon config/server.properties

8. Start nginx and report an error. The file kafka.so.1 cannot be found

error while loading shared libraries: librdkafka.so.1: cannot open shared object file: no such file or directory

##9. Loading so library


#开机加载/usr/local/lib下面的库
echo "/usr/local/lib" >> /etc/ld.so.conf
#手动加载
ldconfig

10. Test, write data to nginx, and then observe whether the kafka consumer can consume the data


curl http://localhost/kafka/access -d "message send to kafka topic"
curl http://localhost/kafka/access -d "小伟666"测试

You can also simulate the page buried request interface to send information:

How to integrate Kafka with Nginx

The background kafka consumption information is as shown in the figure:

How to integrate Kafka with Nginx

The above is the detailed content of How to integrate Kafka with Nginx. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:亿速云. If there is any infringement, please contact admin@php.cn delete
NGINX vs. Apache: A Look at Their ArchitecturesNGINX vs. Apache: A Look at Their ArchitecturesApr 28, 2025 am 12:13 AM

The main architecture difference between NGINX and Apache is that NGINX adopts event-driven, asynchronous non-blocking model, while Apache uses process or thread model. 1) NGINX efficiently handles high-concurrent connections through event loops and I/O multiplexing mechanisms, suitable for static content and reverse proxy. 2) Apache adopts a multi-process or multi-threaded model, which is highly stable but has high resource consumption, and is suitable for scenarios where rich module expansion is required.

NGINX vs. Apache: Examining the Pros and ConsNGINX vs. Apache: Examining the Pros and ConsApr 27, 2025 am 12:05 AM

NGINX is suitable for handling high concurrent and static content, while Apache is suitable for complex configurations and dynamic content. 1. NGINX efficiently handles concurrent connections, suitable for high-traffic scenarios, but requires additional configuration when processing dynamic content. 2. Apache provides rich modules and flexible configurations, which are suitable for complex needs, but have poor high concurrency performance.

NGINX and Apache: Understanding the Key DifferencesNGINX and Apache: Understanding the Key DifferencesApr 26, 2025 am 12:01 AM

NGINX and Apache each have their own advantages and disadvantages, and the choice should be based on specific needs. 1.NGINX is suitable for high concurrency scenarios because of its asynchronous non-blocking architecture. 2. Apache is suitable for low-concurrency scenarios that require complex configurations, because of its modular design.

NGINX Unit: Key Features and CapabilitiesNGINX Unit: Key Features and CapabilitiesApr 25, 2025 am 12:17 AM

NGINXUnit is an open source application server that supports multiple programming languages ​​and provides functions such as dynamic configuration, zero downtime updates and built-in load balancing. 1. Dynamic configuration: You can modify the configuration without restarting. 2. Multilingual support: compatible with Python, Go, Java, PHP, etc. 3. Zero downtime update: Supports application updates that do not interrupt services. 4. Built-in load balancing: Requests can be distributed to multiple application instances.

NGINX Unit vs. Other Application ServersNGINX Unit vs. Other Application ServersApr 24, 2025 am 12:14 AM

NGINXUnit is better than ApacheTomcat, Gunicorn and Node.js built-in HTTP servers, suitable for multilingual projects and dynamic configuration requirements. 1) Supports multiple programming languages, 2) Provides dynamic configuration reloading, 3) Built-in load balancing function, suitable for projects that require high scalability and reliability.

NGINX Unit: The Architecture and How It WorksNGINX Unit: The Architecture and How It WorksApr 23, 2025 am 12:18 AM

NGINXUnit improves application performance and manageability with its modular architecture and dynamic reconfiguration capabilities. 1) Modular design includes master processes, routers and application processes, supporting efficient management and expansion. 2) Dynamic reconfiguration allows seamless update of configuration at runtime, suitable for CI/CD environments. 3) Multilingual support is implemented through dynamic loading of language runtime, improving development flexibility. 4) High performance is achieved through event-driven models and asynchronous I/O, and remains efficient even under high concurrency. 5) Security is improved by isolating application processes and reducing the mutual influence between applications.

Using NGINX Unit: Deploying and Managing ApplicationsUsing NGINX Unit: Deploying and Managing ApplicationsApr 22, 2025 am 12:06 AM

NGINXUnit can be used to deploy and manage applications in multiple languages. 1) Install NGINXUnit. 2) Configure it to run different types of applications such as Python and PHP. 3) Use its dynamic configuration function for application management. Through these steps, you can efficiently deploy and manage applications and improve project efficiency.

NGINX vs. Apache: A Comparative Analysis of Web ServersNGINX vs. Apache: A Comparative Analysis of Web ServersApr 21, 2025 am 12:08 AM

NGINX is more suitable for handling high concurrent connections, while Apache is more suitable for scenarios where complex configurations and module extensions are required. 1.NGINX is known for its high performance and low resource consumption, and is suitable for high concurrency. 2.Apache is known for its stability and rich module extensions, which are suitable for complex configuration needs.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.