Home > Article > Backend Development > Export millions of data to csv with multiple processes in one minute and upload to oss
Export millions of data to csv with multiple processes in one minute and upload to oss
- 藏色散人forward
- 2023-04-20 15:30:511574browse
This article brings you relevant knowledge about PHP. It mainly introduces how to export millions of data to csv and upload it to oss in more than one minute. Friends who are interested can take a look. ,I hope everyone has to help.
Requirements
- The product needs to export the walking logs of certain users of a certain team to a csv file, and it must be in one file, and the speed must be fast and not too time-consuming.
Problem
- Logs often mean large amounts of data.
- To import it into a file, even if the data volume is large, it can still be solved. It is nothing more than paging query and writing to csv.
- The speed is fast and the time is short. It can also be solved by exporting multiple files through multiple processes. There is nothing difficult, but the requirement is one file!
Solution idea
- First query the total amount of data to be exported
- Put it into the batch queue to start multiple consumer queries and write into multiple csv.
- Record the specific csv path and file name written by each queue when posting to the queue.
- Use the characteristics of the batch queue to merge the multiple csv recorded in the previous step into one csv in order when the execution is completed.
- After the merging is completed, delete the multiple csv before merging. Keep the merged csv and upload it to oss
Result
After testing more than 500,000 data, each queue in the batch queue is assigned 50,000 pieces of data to process, and a process is completed It takes less than a minute. How fast this method is depends on how many consumers and processes the server can start. When I tested, I used the automatic strategy, and the maximum was 8 consumers executing at the same time.
Recommended study: "laravel video tutorial"
The above is the detailed content of Export millions of data to csv with multiple processes in one minute and upload to oss. For more information, please follow other related articles on the PHP Chinese website!