How nodejs is distributed

With the continuous development of Internet applications, it is often difficult for a single server to meet the needs of high concurrency and large traffic. In order to solve this problem, distributed systems came into being. Node.js is a very popular server-side JavaScript running environment. It uses an event-driven, non-blocking I/O model and can handle high-concurrency and high-throughput requests. However, the processing power of a single Node.js process is still limited. Therefore, this article will introduce how to implement a distributed system using Node.js.

Distribution refers to decomposing a task into multiple subtasks, assigning these subtasks to different working nodes for execution, and completing the entire task collaboratively through network communication. There are two main ways to implement distributed systems in Node.js: one is to use multi-process mode, and the other is to use message queues.

1. Use multi-process mode

Node.js provides an API for creating child processes through the built-in child_process module. We can easily create multiple child processes to process the same task concurrently. In multi-process mode, each sub-process is independent, and data is exchanged between them through IPC (inter-process communication).

1. Master-Worker mode

The Master-Worker mode is one of the most classic multi-process modes. In this mode, there is a Master process and multiple Worker processes. The Master process is responsible for managing all Worker processes, including starting, stopping, restarting, etc., while the Worker process is responsible for processing specific requests or tasks.

In Node.js, the Master-Worker mode can be implemented through the cluster module. The cluster module is an advanced module encapsulated based on the child_process module. It can easily implement the Master-Worker mode, as shown below:

const cluster = require('cluster');
const http = require('http');
const numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  console.log(`Master ${process.pid} is running`);

  // 当主进程被终止时，关闭所有工作进程
  process.on('SIGINT', () => {
    console.log('Received SIGINT. Shutting down workers...');
    for (const id in cluster.workers) {
      cluster.workers[id].kill();
    }
  });

  // 根据CPU数量创建工作进程
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  // 当有工作进程被断开连接（崩溃）时，自动重新启动
  cluster.on('exit', (worker, code, signal) => {
    console.log(`Worker ${worker.process.pid} died`);
    cluster.fork();
  });
} else {
  console.log(`Worker ${process.pid} started`);

  // Workers可以处理具体的任务，例如下面是创建HTTP服务器的代码
  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('Hello from worker!');
  }).listen(3000);
}

The above code demonstrates how to use the cluster module to create a Master process and multiple Worker processes. In actual use, we can put specific tasks and business logic such as HTTP servers into the Worker process for execution.

2. Process pool mode

The process pool mode is a more efficient multi-process mode. In this mode, we can reuse already created processes to achieve performance optimization. Generally, the number of processes in the process pool should be dynamically adjusted according to the number of system CPUs to ensure that requests can be satisfied under high load.

Node.js does not have a built-in process pool module, but we can implement it through third-party modules. For example, the generic-pool module can be used to easily implement a Worker process pool, as shown below:

const http = require('http');
const pool = require('generic-pool');
const numCPUs = require('os').cpus().length;

const workerFactory = {
  create: function() {
    return new Promise(resolve => {
      const worker = child_process.fork('./worker.js');
      worker.once('message', msg => {
        if (msg.ready) {
          resolve(worker);
        }
      });
    });
  },
  destroy: function(worker) {
    return new Promise(resolve => {
      worker.once('exit', () => {
        resolve();
      });
      worker.send('exit');
    });
  }
};

const workerPool = pool.createPool(workerFactory, { max: numCPUs });

// 创建HTTP服务器
http.createServer(async (req, res) => {
  const worker = await workerPool.acquire();
  worker.send({ type: 'request', path: req.url });
  worker.once('message', msg => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify(msg));
    workerPool.release(worker);
  });
}).listen(3000);

The above code demonstrates how to use the generic-pool module to create a Worker process pool and call the process in the HTTP server Workers in the pool handle specific requests.

2. Using message queue

Message queue is a distributed communication mechanism based on asynchronous (non-blocking) communication mode. In message queue mode, we can send messages to the queue, and the receiver gets the message from the queue and processes it. Therefore, message queues can solve problems such as task distribution and data transmission in distributed systems, and improve the reliability and scalability of the system.

There are many message queue implementations in Node.js, such as RabbitMQ, Redis, Kafka, etc. Here we take RabbitMQ as an example to introduce.

1. Producer-Consumer Pattern

The producer-consumer pattern is a classic message queue pattern. In this mode, the producer is responsible for sending messages to the queue, and the consumer is responsible for getting messages from the queue and processing them.

In Node.js, you can use the amqp.node module to connect to RabbitMQ, and use concepts such as queues and switches to implement the producer-consumer model. Here is a simple example:

const amqp = require('amqp');
const connection = amqp.createConnection({ host: 'localhost' });

// 连接RabbitMQ服务器
connection.on('ready', function() {
  console.log('Connected to RabbitMQ');

  // 创建消息队列
  connection.queue('hello-queue', { durable: true }, function(queue) {
    console.log('Created queue: ' + queue.name);

    // 创建消息生产者
    setInterval(function() {
      const message = 'Hello ' + new Date();
      console.log('Sending message: ' + message);
      connection.publish(queue.name, message, { persistent: true });
    }, 1000);

    // 创建消息消费者
    queue.subscribe(function(message) {
      console.log('Received message: ' + message.data.toString());
    });
  });
});

The above code demonstrates how to use the amqp.node module to connect to the RabbitMQ server and create a producer and a consumer. The producer sends a message to the queue every 1 second, and the consumer gets the message from the queue and processes it.

2. Publish-Subscribe Pattern

Publish-Subscribe pattern is another common message queue pattern. In this mode, there is a message publisher and multiple message subscribers. The publisher sends messages to a topic, and subscribers can obtain messages from the topic according to their own subscription rules.

In Node.js, we can also use the amqp.node module to implement the publish-subscribe mode. The following is a simple example:

const amqp = require('amqp');
const connection = amqp.createConnection({ host: 'localhost' });

// 连接RabbitMQ服务器
connection.on('ready', function() {
  console.log('Connected to RabbitMQ');

  // 创建消息主题
  const exchange = connection.exchange('logs', { type: 'fanout' }, function() {
    console.log('Created exchange: ' + exchange.name);

    // 创建消息订阅者
    connection.queue('', { exclusive: true }, function(queue) {
      console.log('Created queue: ' + queue.name);
      queue.bind(exchange, '');

      queue.subscribe(function(message) {
        console.log('Received message: ' + message.data.toString());
      });
    });

    // 创建消息发布者
    setInterval(function() {
      const message = 'Hello ' + new Date();
      console.log('Sending message: ' + message);
      exchange.publish('', message);
    }, 1000);
  });
});

The above code demonstrates how to use the amqp.node module to create a message topic, a message subscriber and a message publisher. The publisher sends a message to the topic every 1 second, and the subscriber gets the message from the topic and processes it.

Summary

This article introduces how to use Node.js to implement a distributed system. In practical applications, we can choose different distributed communication mechanisms according to specific business needs, such as using multi-process mode or message queue mode. No matter which method you choose, you need to pay attention to issues such as reliability, scalability, and security of distributed systems.

The above is the detailed content of How nodejs is distributed. For more information, please follow other related articles on the PHP Chinese website!