support compress & decompress by stream(block by block) #384
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
for decompressor
Added several lower-level APIs with little as possible source code changes, to support decompress block by block;
And updated the gzip cmdline with new APIs, the memory usage when decompress will be greatly reduced, usually <1MB!
Also, because it needs to support any .gz format (not just libdeflate), the decompress buffer will automatically adjust the size to fit the input maximum block size.
Because decompressor reused a small memory, the CPU cache hit ratio will be greatly improved, resulting in an increase in decompress speed;
In my multiple test cases, it was able to get 20%--50% faster.
for compressor
Added several lower-level APIs with minor source code changes, to support compress block by block;
And updated the gzip cmdline by new APIs, the memory usage when compress will be greatly reduced;
The memory requires is related to the block size set and compress level, not related to input file size; when the block size is 2 MB and compress level is 12, the memory usage ~ 13 MB.
These new APIs can be used for multi-threaded parallel compression, and there is almost no loss of compress ratio.
( I used libdeflate for my multiple actual projects and make a lot of feature changes; Hopefully commit some useful changes back to libdeflate. )