python - pyspider爬取了接近1G的数据，无法导出csv？

Question

已经增加虚拟机内存至2g，依然无法导出。

Answer 1

Generally, it is exported in files. If the data of one unit is about 1M, it will be divided into 1024 files.

In addition, it is impractical to synthesize a 1G CSV, because a 1G CSV is almost impossible to open. If you really want to synthesize 1G, you can synthesize more than a thousand exported files into a CSV. File synthesis is easy to achieve in Python.

Answer 2

Monthly CSV processing of 1G data is not reasonable. Try sqlite3.

Answer 3

8G is not enough. It is recommended to choose a database as a storage solution. A 1G CSV has no meaning.

Answer 4

Downloading is very fast using ftp intranet. I uploaded 8g two days ago. It feels like it downloaded in just a few minutes