Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce memory usage #1

Open
urbanslug opened this issue Mar 21, 2022 · 0 comments
Open

Reduce memory usage #1

urbanslug opened this issue Mar 21, 2022 · 0 comments
Labels
performance Memory and time usage improvements

Comments

@urbanslug
Copy link
Owner

Expects sorted VCF

  1. Replace Vec<T> with Box<T> for the value in data
    • would this avoid duplicates?
  2. Read the whole VCF into a buffer and use non-owned slices to refer to the lines.
    • That'd avoid the heap overhead of allocations.
  3. mmap the file and avoid duplicating the page cache in your process's memory
  4. Serialize the VCF in fixed size chunks to disk
    • less memory but more time
@urbanslug urbanslug added the performance Memory and time usage improvements label Mar 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Memory and time usage improvements
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@urbanslug