python - 100G超大文件合并排序

Question

目前想实现两个100G文件合并，日志文件，都有日期，日期会有所交集，如一份1日到10日，另一份5日到15日，所以合并后也需要对日期进行排序。

目标是，查询某一时间区段的信息，目前的打算是：

我可以知道每个文件大小和日起始时间，所以若文件过大，就分割它，然后获取分割小文件的的起始时间，时间区段只要稍加限制，同时由于log的连续性和记录频率不太高，就能确保就只有几个小文件包含了用户需要的信息，然后只返回那几小份的文件的信息，然后全部直接扔进内存排序都不怕，效率也可以。

但我想想，是不是也有办法，直接真的可以把2个100G合成一个200G的？
或有什么更好的办法，目前在写模板的基本方法，想封装好一点以后不管了。但不知道这个要怎么玩。求教，谢谢！

Answer 1

Provide an idea: I think your idea of ​​segmentation is actually quite good, but it doesn’t actually need to be segmented. All you have to do is maintain an index file. That is to say, by reading the file once, for every 1000 entries (for example), the starting file offset and end offset of the 1000 entries are associated with the start time and end time of these 1000 entries. This way you get an index file.

时间1~时间2，文件1，offset1~offset2
时间3~时间4，文件2，offset3~offset4
...

When you want to query in the future, check the index file first, and then you can know which file and which range the required data is in. Because each of your files is sorted by time, there is no need to sort when indexing.

Answer 2

It’s so big, I still need to query why it is regularly transferred to the database.