Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transient data cleanup #183

Open
3 of 4 tasks
Zeitsperre opened this issue Aug 9, 2024 · 1 comment
Open
3 of 4 tasks

Transient data cleanup #183

Zeitsperre opened this issue Aug 9, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request notebooks Run tests against notebooks

Comments

@Zeitsperre
Copy link
Collaborator

Zeitsperre commented Aug 9, 2024

Addressing a Problem?

Building the documentation creates many artifacts (many of which are persistent and found in docs/notebooks/_data/**). Many types of files are written there during execution of the notebooks, with no real order/organizational scheme.

Additionally, running pytest relies on class-housed testing setup code that would be better off created as pytest fixtures. As it stands currently, some functions will create/fetch remote data to be used in multiple tests; When running with pytest-xdist this causes segmentation faults due to synchronous read/write operations on these files.

Potential Solution

  • Using session-scoped pytest fixtures could help immensely with ensuring that testing data is fetched once and yielded for tests as needed.
  • The notebooks need to make use of the tempdir library and the write operations should be verified so that notebook execution order is not important.
  • We should be making use of pooch registries

We can likely work on this after merging #175 which houses an initial implementation.

Additional context

No response

Contribution

  • I would be willing/able to open a Pull Request to contribute this feature.
@Zeitsperre
Copy link
Collaborator Author

#212 Tackles the first and third points listed here.

It will make more sense to tackle the final point concerning notebook data artifacts once this has been merged to main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request notebooks Run tests against notebooks
Projects
None yet
Development

No branches or pull requests

1 participant