Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support in-memory data for Transfer Manager download_chunks_concurrently and upload_chunks_concurrently #1341

Open
MattIrv opened this issue Aug 22, 2024 · 1 comment
Assignees
Labels
api: storage Issues related to the googleapis/python-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@MattIrv
Copy link

MattIrv commented Aug 22, 2024

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

  • Today Transfer Manager's download_chunks_concurrently and upload_chunks_concurrently require you to provide a filename. Unfortunately I have bytes that are already in-memory and may not fit on disk (or it may be slow to persist it to disk) and can't use Transfer Manager as a result.

A clear and concise description of what you want to happen.

  • Provide functions that support the same interface as download_chunks_concurrently and upload_chunks_concurrently but take a file-like object instead of a filename
  • Or, for write, provide a similar function that returns a stream to the caller, and then writes to the stream are uploaded via multipart upload as needed (not sure if there's any realistic similar thing here for reads)
  • It's ok if this functionality can only be threaded and not use multi-processing since in practice we still see a large speed increase from threading
@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/python-storage API. label Aug 22, 2024
@MattIrv
Copy link
Author

MattIrv commented Aug 22, 2024

@andrewsg fyi

@cojenco cojenco added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

2 participants