Introduction to the method of Node batch downloading files to the local (with code)

This article brings you an introduction to the method of Node batch downloading files to the local (with code). It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

Process multiple files separately

If the number of files is controllable and there is no requirement for the downloaded file format, you can use the simplest method to directly traverse the files and process them separately. Each download link creates a single-file download or iframe download link.

zip package batch download

Although it is possible to traverse all files and then download individual files in batches, this experience is not very good after all. The most common approach is Download and package batch files into zip.

So the first implementation idea is: in the proxy service, first traverse all files to request file data, then compress it into a zip package, and then return the zip package to the client.

This is OK if the download volume data is relatively small, but if the batch file is very large and large, the user has to wait for the background to request all the data and package them into a compressed package before the front end can give feedback. This can take a long time and the user experience can be poor.

During the preliminary research of colleagues, it was said that there can be a streaming compression and downloading capability here. The general idea is to chunk back the package and add streaming compression.

......

let fileCounter = 0;
const zippedFilename = encodeURIComponent(downloadData.name);
const list = downloadData.list || [];
const header = {
  'Content-Type': 'application/x-zip',
  'Pragma': 'public',
  'Expires': '0',
  'Cache-Control': 'private, must-revalidate, post-check=0, pre-check=0',
  'Content-disposition': 'attachment; filename="' + zippedFilename + '"',
  'Transfer-Encoding': 'chunked',
  'Content-Transfer-Encoding': 'binary'
};
res.writeHead(200, header);
archive.store = true;
archive.pipe(res);
list.map(item => {
  fileCounter++;
  let inStream = request.get(item.downLoadUrl);
  let name = item.fileName;
  let length = 0;
  inStream.on('response', function(awsData) {
    archive.append(inStream, {
      name: name
    });
  }).on('data', function(data) {
    length += data.length;
  }).on('error', function(e) {
    console.error(name + '-error', e);
  }).on('end', function(endData) {
    fileCounter--;
    if (fileCounter < 1) {
      archive.finalize();
    }
  });
});
archive.on('error', function(err) {
  throw err;
});
archive.on('finish', function(err) {
  return res.end();
});

......

Of course, there are still some details that need to be dealt with: such as the problem of Chinese file names, whether the total size of the downloaded file needs to be limited, whether the file does not exist, etc.

The above is the detailed content of Introduction to the method of Node batch downloading files to the local (with code). For more information, please follow other related articles on the PHP Chinese website!