java - How to decompress the data in some individual data blocks of a gzip file when streaming it?
Scene description:
现有许多行日志文本，按天压缩成一个个TB级的gzip文件。
使用流对每个压缩文件的数据段进行传输然后解压，对解压出的文本分词并索引
以后查到这个词时，定位到这个词所在的文件和段，再用流传输并解压
（实际上是想利用已有的压缩文件构造一个类似ES的搜索引擎）
The problem now is that because what is received is not a complete compressed file but a block of binary data, the received data cannot be decompressed due to incomplete information
Now I want to implement such a function: first decompress the received stream data and restore it to complete data (the original log data is separated by newlines, and you can get the text before compression of each stream data and the offset of the corresponding file) Good), and then consider that processes such as transmission and storage may cause data errors, so for each data stream, decompress as much data as possible in the event of errors.
Part of the relevant code is as follows: (modified from https://stackoverflow.com/que...)
import zlib
import traceback
CHUNKSIZE=30
d = zlib.decompressobj(16 + zlib.MAX_WBITS)
f = open('test.py.gz','rb')
buffer = f.read(CHUNKSIZE)
i = 0
while buffer :
i += 1
try:
#skip two chunk
if i < 3 or i > 4:
outstr = d.decompress(buffer)
print('*'*10 + outstr + '#'*10)
except Exception, e:
print(traceback.print_exc())
finally:
buffer = f.read(CHUNKSIZE)
outstr = d.flush()
print(outstr)
f.close()When i>=3, an error is reported every time in the loop
My conclusion is that if the stream is not continuous (skipping to receive part of the data), then the subsequent data cannot be decompressed.
Question 1: What if we can correctly decompress each part of the data received? (Because it may involve the algorithm and data structure of gzip compression, I am looking at the relevant code. If the problem can be solved by appending a certain chuck in the transmission header or some chuck before and after the data that needs to be decompressed, it can be considered)
Question 2:
If you cannot correctly decompress every part of the data received, how can you decompress as much data as possible?
