Introduction to server-side character encoding, decoding and garbled processing methods in Nodejs

This article mainly introduces the advanced server-side character encoding and decoding and garbled processing of Nodejs. It has certain reference value. Interested friends can refer to it

Written in front

In web server development, character encoding and decoding have to be dealt with almost every day. Once the encoding and decoding is not handled properly, troublesome garbled characters will occur.

Many students who are engaged in node server development often find themselves at a loss when encountering problems due to insufficient knowledge of character encoding codes and spend a lot of time troubleshooting and solving problems.

The text first briefly introduces the basic knowledge of character encoding and decoding, then gives an example of how to encode and decode in node, and finally is a server-side code example. Code examples related to this article can be found here.

About character encoding and decoding

In the process of network communication, binary bits are transmitted, regardless of whether the content sent is text or pictures, the language used Is it Chinese or English.

For example, the client sends "Hello" to the server.

Client --- Hello ---> Server

This contains two key steps, corresponding to encoding and decoding.

1. Client: Encode the string "Hello" into the binary bits required by the computer network.

2. Server: Decode the received binary bits into the string "Hello".

To summarize:

1. Encoding: Convert the data to be transmitted into the corresponding binary bits.

2. Decoding: Convert binary bits into original data.

Some important technical details are not mentioned above, the answers are in the next section.

• How does the client know the number of bits corresponding to the character "Hello"?

• After the server receives the binary bits, how does it know what the corresponding string is?

About character set and character encoding

The character and binary conversion issues are mentioned above. Since the two can be converted to each other, that is to say, there are clear conversion rules, and the characters e09be6022d700e04aeaa85a5f42fdcb2 can be converted into binary.

The conversion rules mentioned here are actually the character sets & character encodings we often hear.

Character set is a collection of a series of characters (text, punctuation marks, etc.). There are many character sets, common ones include ASCII, Unicode, GBK, etc. The main difference between different character sets is the number of characters they contain.

After understanding the concept of character set, let’s introduce character encoding.

The character set tells us which characters are supported, but how to encode specific characters is determined by the character encoding. For example, the Unicode character set supports character encodings such as UTF8 (commonly used), UTF16, and UTF32.

To summarize:

• Character set: A collection of characters. Different character sets contain different numbers of characters.

• Character encoding: The actual encoding of characters in the character set.

• A character set may have multiple character encoding methods.

Character encoding can be regarded as a mapping table. The client and server use this mapping table to implement character and binary encoding and decoding conversion.

For example, the character "you" occupies three bytes 0xe4 0xbd 0xa0 in UTF8 encoding, and occupies two bytes 0xc4 0xe3 in GBK encoding.

Character encoding and decoding examples

The basic knowledge required for character encoding and decoding has been mentioned above. Let's look at a simple example below, where we use the icon-lite library to help us implement encoding and decoding operations.

As you can see, we use gbk when encoding characters. When decoding, if you also use gbk, you can get the original characters. When we use utf8 when decoding, garbled characters appear.

var iconv = require('iconv-lite');

var oriText = '你';

var encodedBuff = iconv.encode(oriText, 'gbk');
console.log(encodedBuff);
// <Buffer c4 e3>

var decodedText = iconv.decode(encodedBuff, &#39;gbk&#39;);
console.log(decodedText);
// 你

var wrongText = iconv.decode(encodedBuff, &#39;utf8&#39;);
console.log(wrongText);
// ��

Practical example: Server-side encoding and decoding

Usually the scenarios where we need to handle encoding and decoding include file reading and writing, and network requests deal with. Here is an example of a network request, introducing how to encode and decode on the server side.

Suppose we are running the following http service, listening for requests from clients. The client uses gbk encoding when transmitting data, while the server uses utf8 encoding by default.

If the default utf8 is used to decode the request at this time, garbled characters will appear, so special processing is required.

The server code is as follows (to simplify the code, the judgment of request method and request encoding is skipped here)

var http = require('http');
var iconv = require('iconv-lite');

// 假设客户端采用post方法，编码为gbk
var server = http.createServer(function (req, res) {
  var chunks = [];
  
  req.on('data', function (chunk) {
    chunks.push(chunk)
  });

  req.on('end', function () {
    chunks = Buffer.concat(chunks);

    // 对二进制进行解码
    var body = iconv.decode(chunks, 'gbk');
    console.log(body);

    res.end('HELLO FROM SERVER');
  });

});

server.listen(3000);

The corresponding client code is as follows:

var http = require('http');
var iconv = require('iconv-lite');

var charset = 'gbk';

// 对字符"你"进行编码
var reqBuff = iconv.encode('你', charset);

var options = {
  hostname: '127.0.0.1',
  port: '3000',
  path: '/',
  method: 'POST',
  headers: {
    'Content-Type': 'text/plain',
    'Content-Encoding': 'identity',
    'Charset': charset // 设置请求字符集编码
  }
};

var client = http.request(options, function(res) {
  res.pipe(process.stdout);
});

client.end(reqBuff);

The above is the detailed content of Introduction to server-side character encoding, decoding and garbled processing methods in Nodejs. For more information, please follow other related articles on the PHP Chinese website!