Detailed explanation of Node.js file encoding format conversion steps

This time I will bring you Node.js Detailed explanation of the file encoding format conversion steps, what are the precautions for Node.js file encoding format conversion, the following is a practical case, one Get up and take a look.

There are many Lua files in the project that are not in UTF-8 format. When viewed using EditPlus, they are displayed as ASCII. There are also some with BOM, which is easier to deal with. I have written about it before, and there are certain rules.

ASCII encoding is a pain in the ass. By searching online resources and repeatedly testing and comparing, I finally came up with the following more reliable method (some EditPlus display encoding is utf-8 but the node.js library returns something else. Coding>_

To determine whether the modification is correct, you only need to submit it through SVN after the modification, browse the submission list, double-click any file to be submitted, if the dialog box shown in the figure below is displayed, It means that the modification is successful. Others will see that the Chinese text has become garbled.

var fs = require('fs');
var chardet = require('chardet');
var jschardet = require("jschardet");
var encoding = require("encoding");
var path = "lua目录";
function readDirectory(dirPath) {
  if (fs.existsSync(dirPath)) {
    var files = fs.readdirSync(dirPath);
    files.forEach(function (file) {
      var filePath = dirPath + "/" + file;
      var stats = fs.statSync(filePath);
      if (stats.isDirectory()) {
        // console.log('/n读取目录：\n', filePath, "\n");
        readDirectory(filePath);
      } else if (stats.isFile() && /\.lua$/.test(filePath)) {
        var buff = fs.readFileSync(filePath);
        if (buff.length && buff[0].toString(16).toLowerCase() == "ef" && buff[1].toString(16).toLowerCase() == "bb" && buff[2].toString(16).toLowerCase() == "bf") {
          //EF BB BF 239 187 191
          console.log('\n发现BOM文件：', filePath, "\n");
          buff = buff.slice(3);
          fs.writeFile(filePath, buff.toString(), "utf8");
        }
        // { encoding: 'UTF-8', confidence: 0.99 }
        // var charset = chardet.detectFileSync(filePath);
        var info = jschardet.detect(buff);
        if (info.encoding == "GB2312" || info.encoding == "ascii") {
          var resultBuffer = encoding.convert(buff, "UTF-8", info.encoding);
          fs.writeFile(filePath, resultBuffer, "utf8");
        }
        else if (info.encoding != "UTF-8" && chardet.detectFileSync(filePath) != "UTF-8")
        {
          if (buff.toString().indexOf("\r\n") > -1)
          {
            var resultBuffer = encoding.convert(buff, "UTF-8", "GBK");
            fs.writeFile(filePath, resultBuffer, "utf8");
          }
        }
      }
    });
  } else {
    console.log('Not Found Path : ', dirPath);
  }
}
readDirectory(path);

Pay attention to the above judgment. When the first one is clearly GB2312 or ascii, directly change the corresponding Convert encoding to utf-8. And if the format returned is, first determine whether there is a line break character under the PC, and if so, treat it all as GBK for processing.

The whole idea is actually relatively simple. The difficulty lies in determining the file encoding format. This is really difficult>_

Note: The files modified by the above method are consistent with the list of files that need to be submitted on Mac. At least it solves the problem I'm currently experiencing. If there is something special, you can modify the above code.

Third-party libraries used:

encoding https://github.com/andris9/encoding
jschardet https://github.com/aadsm /jschardet
node-chardet https://github.com/runk/node-chardet

I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to other related articles on the PHP Chinese website!

Recommended reading:

Vue implementation of PopupWindow component usage steps analysis

vue jquery lodash top suspension fixed function implementation when sliding Detailed explanation

The above is the detailed content of Detailed explanation of Node.js file encoding format conversion steps. For more information, please follow other related articles on the PHP Chinese website!