node.js - nodejs 多线程下载图片如何做到多张图片只请求一次？

Question

如题。比如

http://www.xx.com/abc.jpg，很多个这样的地址。
因为是异步进行的。你每次请求之前没法判断这张图片是否已经下载获取存在过了。所以发出了很多个下载同一地址的请求。

Answer 1

There are two different situations: all the URLs are known before the request and all the URLs are not known before the request
The first situation: first remove the duplicates from the requested image URL array, and then initiate the request
Second Two cases: put the requested URL in an object, and then check whether it has been requested. If it has been requested, no request will be initiated

In addition, the mechanism of NodeJS itself is not multi-threaded, but a single-process event loop, so simple flag bits can be used to implement the sentinel mechanism

Answer 2

Either you have requested it and then don’t request again. However, there is no guarantee that the requested download will be successful. If the download fails, request again

Answer 3

Is the name of the file saved after downloading the same? If they are the same, check whether the file exists and then download it. This can avoid repeated downloads.

Answer 4

md5 and url