Node.js implements compression and decompression

Compression format

zip and gzip are the two most common compression formats we see. Of course, gzip is rarely touched under Windows. Tar is an archive format that does not compress by default. It needs to be combined with gzip to compress the final tar file into a tar.gz file in gzip format, which is usually abbreviated to tgz.

Why is rar not mentioned? Because it is a patent-protected algorithm, you can get the decompression tool for free, while the compression tool has to be paid. Therefore, in our general application scenarios, rar compressed files are rarely provided.

This article will introduce respectively how to realize the compression and decompression of gzip, tar, tgz and zip under Node.js.

Uncompressed file library

The uncompressed file library used in this article comes from urllib. You need to clone it first and go to the specified directory.

git clone https://github.com/node-modules/urllib.git nodejs-compressing-demo

gzip

In the Linux world, the responsibilities of each tool will be very pure and very single. For example, gzip will only compress files. As for files, It has nothing to do with how the folder is packaged and compressed. That is tar's responsibility.

gzip command line compresses a file

For example, if we want to gzip the nodejs-compressing-demo/lib/urllib.js file, we will get a urllib.js .gz file, the source file will be deleted.

$ ls -l nodejs-compressing-demo/lib/urllib.js
-rw-r--r-- 1 a a 31318 Feb 12 11:27 nodejs-compressing-demo/lib/urllib.js

$ gzip nodejs-compressing-demo/lib/urllib.js

$ ls -l nodejs-compressing-demo/lib/urllib.js.gz
-rw-r--r-- 1 a a 8909 Feb 12 11:27 nodejs-compressing-demo/lib/urllib.js.gz

# 还原压缩文件
$ gunzip nodejs-compressing-demo/lib/urllib.js.gz

The file size is reduced from 31318 bytes to 8909 bytes, more than 3.5 times the compression effect.

You can also compress and save the file as any file through the pipe method, combined with the cat command:

$ ls -l nodejs-compressing-demo/README.md
-rw-r--r-- 1 a a 13747 Feb 12 11:27 nodejs-compressing-demo/README.md

$ cat nodejs-compressing-demo/README.md | gzip > README.md.gz

$ ls -l README.md.gz
-rw-r--r-- 1 a a 4903 Feb 12 11:50 README.md.gz

Node.js implements gzip

Of course, we will not really implement a gzip algorithm and tool from scratch. In the world of Node.js, these basic libraries have already been prepared for you, we just need to use them out of the box.

This article will use the compressing module to implement all compression and decompression codes.

Why choose compressing? Because it has sufficient code quality and unit testing guarantees, is in an active maintenance state, has a very friendly API, and also supports streaming interfaces.

Promise interface

const compressing = require('compressing');

// 选择 gzip 格式，然后调用 compressFile 方法
compressing.gzip.compressFile('nodejs-compressing-demo/lib/urllib.js', 'nodejs-compressing-demo/lib/urllib.js.gz')
 .then(() => {
  console.log('success');
 })
 .catch(err => {
  console.error(err);
 });

// 解压缩是反响过程，接口都统一为 uncompress
compressing.gzip.uncompress('nodejs-compressing-demo/lib/urllib.js.gz', 'nodejs-compressing-demo/lib/urllib.js2')
 .then(() => {
  console.log('success');
 })
 .catch(err => {
  console.error(err);
 });

Combined with the async/await programming model, the code is written as an ordinary asynchronous io operation.

const compressing = require('compressing');

async function main() {
 try {
  await compressing.gzip.compressFile('nodejs-compressing-demo/lib/urllib.js',
   'nodejs-compressing-demo/lib/urllib.js.gz');
  console.log('success');
 } catch (err) {
  console.error(err);
 }

 // 解压缩
 try {
  await compressing.gzip.uncompress('nodejs-compressing-demo/lib/urllib.js.gz',
   'nodejs-compressing-demo/lib/urllib.js2');
  console.log('success');
 } catch (err) {
  console.error(err);
 }
}

main();

Stream interface

It should be noted that when programming in Stream mode, you need to handle the error event of each stream. And all streams must be destroyed manually.

fs.createReadStream('nodejs-compressing-demo/lib/urllib.js')
 .on('error', handleError)
 .pipe(new compressing.gzip.FileStream()) // It's a transform stream
 .on('error', handleError)
 .pipe(fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js.gz2'))
 .on('error', handleError);

// 解压缩，就是 pipe 的方向倒转过来
fs.createReadStream('nodejs-compressing-demo/lib/urllib.js.gz2')
 .on('error', handleError)
 .pipe(new compressing.gzip.UncompressStream()) // It's a transform stream
 .on('error', handleError)
 .pipe(fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js3'))
 .on('error', handleError);

According to the official Backpressuring in Streams recommendation, we should use the pump module to cooperate with Stream mode programming, and let pump complete the cleaning work of these Streams.

const pump = require('pump');

const source = fs.createReadStream('nodejs-compressing-demo/lib/urllib.js');
const target = fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js.gz2');

pump(source, new compressing.gzip.FileStream(), target, err => {
 if (err) {
  console.error(err);
 } else {
  console.log('success');
 }
});

// 解压缩
pump(fs.createReadStream('nodejs-compressing-demo/lib/urllib.js.gz2'),
  new compressing.gzip.FileStream(),
  fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js3'),
  err => {
 if (err) {
  console.error(err);
 } else {
  console.log('success');
 }
});

Advantages of the Stream interface

The Stream interface seems much more complicated than the Promise interface, why is there such an application? What about the scene?

In fact, in the field of HTTP services, the Stream model will have greater advantages, because the HTTP request itself is a Request Stream. If you want to return an uploaded file with gzip compression, you do not need to save the uploaded file using the Stream interface. to the local disk, but consume this file stream directly.

Using the sample code for egg file upload, we can achieve gzip compression and return with a slight modification.

const pump = require('pump');

class UploadFormController extends Controller {
 // ... other codes

 async upload() {
  const stream = await this.ctx.getFileStream();
  // 直接将压缩流赋值给 ctx.body，实现边压缩边返回的流式响应
  this.ctx.body = pump(stream, new compressing.gzip.FileStream());
 }
}

tar | gzip > tgz

gzip Chapter You can know in advance that tar is responsible for packaging the folder: package :of.

For example, if you want to package the entire nodejs-compressing-dem o folder into a file and send it to others, you can use the tar command.

$ tar -c -f nodejs-compressing-demo.tar nodejs-compressing-demo/

$ ls -l nodejs-compressing-demo.tar
-rw-r--r-- 1 a a 206336 Feb 12 14:01 nodejs-compressing-demo.tar

As you can see, the file packaged by tar is generally larger because it is uncompressed and the size is close to the total size of the actual folder. So we will all compress at the same time as packaging.

$ tar -c -z -f nodejs-compressing-demo.tgz nodejs-compressing-demo/

$ ls -l nodejs-compressing-demo.tgz
-rw-r--r-- 1 a a 39808 Feb 12 14:07 nodejs-compressing-demo.tgz

The difference in size between tar and tgz is more than 5 times, which can greatly reduce network transmission bandwidth.

Node.js implements tgz

Promise interface

First use compressing.tar.compressDir(sourceDir, targetFile) to Pack a folder into a tar file, and then use the gzip compression method above to compress the tar file into a tgz file.

const compressing = require('compressing');

compressing.tar.compressDir('nodejs-compressing-demo', 'nodejs-compressing-demo.tar')
 .then(() => {
  return compressing.gzip.compressFile('nodejs-compressing-demo.tar',
   'nodejs-compressing-demo.tgz');
 });
 .then(() => {
  console.log('success');
 })
 .catch(err => {
  console.error(err);
 });

// 解压缩
compressing.gzip.uncompress('nodejs-compressing-demo.tgz', 'nodejs-compressing-demo.tar')
 .then(() => {
  return compressing.tar.uncompress('nodejs-compressing-demo.tar',
   'nodejs-compressing-demo2');
 });
 .then(() => {
  console.log('success');
 })
 .catch(err => {
  console.error(err);
 });

Combined with the async/await programming model, the code will be easier to read:

const compressing = require('compressing');

async function main() {
 try {
  await compressing.tar.compressDir('nodejs-compressing-demo',
   'nodejs-compressing-demo.tar');
  await compressing.gzip.compressFile('nodejs-compressing-demo.tar',
   'nodejs-compressing-demo.tgz');
  console.log('success');
 } catch (err) {
  console.error(err);
 }
 
 // 解压缩
 try {
  await compressing.gzip.uncompress('nodejs-compressing-demo.tgz',
   'nodejs-compressing-demo.tar');
  await compressing.tar.uncompress('nodejs-compressing-demo.tar',
   'nodejs-compressing-demo2');
  console.log('success');
 } catch (err) {
  console.error(err);
 }
}

main();

Stream interface

Through the compressing.tar.Stream class, you can dynamically add any files and folders to a tar stream object, which is very flexible.

const tarStream = new compressing.tar.Stream();
// dir
tarStream.addEntry('dir/path/to/compress');
// file
tarStream.addEntry('file/path/to/compress');
// buffer
tarStream.addEntry(buffer);
// stream
tarStream.addEntry(stream);

const destStream = fs.createWriteStream('path/to/destination.tgz');
pump(tarStream, new compressing.gzip.FileStream(), destStream, err => {
 if (err) {
  console.error(err);
 } else {
  console.log('success');
 }
});

zip

zip can actually be regarded as a "commercial" combination of tar + gzip, which allows users to You need to distinguish whether it is a compressed file or a compressed folder. Anyway, just use my zip.

Example of using the zip command line tool to compress a folder:

$ zip -r nodejs-compressing-demo.zip nodejs-compressing-demo/
 adding: nodejs-compressing-demo/ (stored 0%)
 adding: nodejs-compressing-demo/test/ (stored 0%)
 ...
 adding: nodejs-compressing-demo/.travis.yml (deflated 36%)

$ ls -l nodejs-compressing-demo.*
-rw-r--r-- 1 a a 206336 Feb 12 14:06 nodejs-compressing-demo.tar
-rw-r--r-- 1 a a  39808 Feb 12 14:07 nodejs-compressing-demo.tgz
-rw-r--r-- 1 a a  55484 Feb 12 14:34 nodejs-compressing-demo.zip

By comparing the sizes of tgz and zip files, you can see that under the default compression parameters , gzip will perform better than zip.

Node.js implements zip

实现代码跟 tar 类似，只不过默认是压缩的，不需要再添加 gzip 的过程。

const compressing = require('compressing');

compressing.zip.compressDir('nodejs-compressing-demo', 'nodejs-compressing-demo.zip')
 .then(() => {
  console.log('success');
 })
 .catch(err => {
  console.error(err);
 });

// 解压缩
compressing.zip.uncompress('nodejs-compressing-demo.zip', 'nodejs-compressing-demo3')
 .then(() => {
  console.log('success');
 })
 .catch(err => {
  console.error(err);
 });

总结

基于 Node.js 实现的压缩和解压缩是否比想象中简单？感谢 npm 这个巨人，让我们编程也能拥有命令行工具那样简单的体验。

无论是 Promise 接口，还是 Stream 接口，都有它最合适的场景，你会选择了吗？

到此，你拥有的压缩和解压缩能力，你能够做什么样的服务和功能呢？

相关推荐：

Grunt针对静态文件的压缩,版本控制打包详解

微信小程序图片压缩功能的实现方法

php解压zip压缩包到指定目录实现方法

The above is the detailed content of Node.js implements compression and decompression. For more information, please follow other related articles on the PHP Chinese website!