Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add performances table for deeprank2 #493

Merged
merged 17 commits into from
Sep 22, 2023
Merged

docs: add performances table for deeprank2 #493

merged 17 commits into from
Sep 22, 2023

Conversation

gcroci2
Copy link
Collaborator

@gcroci2 gcroci2 commented Sep 6, 2023

I am using the same PDB raw data used in the tutorials (see https://zenodo.org/record/8187806).

  • Experiments done on Apple M1 Pro, 1 CPU used
  • PPIs
    • atomic resolution, distance cutoff 5.5, grid resolution 1 (n points 35x30x30), 33 features (all except for conservation features module)
    • mean on 100 PDB files, values are per data point (one ppi)
    • graph: 2.9879 s (std 0.2260), 0.5387 MB (std 0.0706)
    • graph + grid: 11.3493 s (std 1.3049), 16.0929 MB (std 0.4414)
  • Variants
    • atomic resolution, distance cutoff 5.5, radius 10, grid resolution 1 (n points 35x30x30), 26 features (all except for conservation)
    • mean on 96 PDB files, values are per data point (one srv)
    • graph: 2.1989 s (std 0.0790), 0.0543 MB (std 0.0024)
    • graph + grid: 2.8514 s (std 0.1035), 17.5153 MB (std 0.5932)
  • There are two scripts I used for calculating the computational performances for the table, and I think it would be useful to keep them. Not sure where thought. In tools/ maybe, or in a perf/ folder in tutorials? Or just in a branch we can reference. Not sure about this, @DaniBodor waiting for your thoughts :)
  • I think it would be fair to compare with previous packages, but after thinking about how to do this I realized that it would take much more time than I would like to devote to this task now. I opened a new issue about it (Add performances comparisons with previous packages #500) and I placed it in the new SS kanban board. For the future framework paper (not JOSS), it would be nice to have a proper computational performances comparison.

@gcroci2 gcroci2 self-assigned this Sep 6, 2023
@gcroci2 gcroci2 linked an issue Sep 6, 2023 that may be closed by this pull request
@gcroci2 gcroci2 mentioned this pull request Sep 19, 2023
@DaniBodor
Copy link
Collaborator

I will take a closer look at this, but for now already a few comments:

I feel that this should be much lower in the README, maybe just before the Package Development header. It's nice information to have there, but most users will not care that much, so it shouldn't be something they need to scroll past before getting to the information they are looking for.

There are two scripts I used for calculating the computational performances for the table, and I think it would be useful to keep them. Not sure where thought. In tools/ maybe, or in a perf/ folder in tutorials? Or just in a branch we can reference. Not sure about this, @DaniBodor waiting for your thoughts :)

Yes, I would create a subfolder performance/, probably in tools.

I think it would be fair to compare with previous packages, but after thinking about how to do this I realized that it would take much more time than I would like to devote to this task now. I opened a new issue about it (

Definitely out of the scope of this PR/the JOSS paper

DSSP expects PDB files only and we have another format for variants

I think it's actually the same format, just a different extension. If you just replace the extension with .pdb, I think it should work. Not sure whether it's worth the effort though.

@gcroci2
Copy link
Collaborator Author

gcroci2 commented Sep 22, 2023

I feel that this should be much lower in the README, maybe just before the Package Development header. It's nice information to have there, but most users will not care that much, so it shouldn't be something they need to scroll past before getting to the information they are looking for.

Agreed, I moved it down

DSSP expects PDB files only and we have another format for variants

I think it's actually the same format, just a different extension. If you just replace the extension with .pdb, I think it should work. Not sure whether it's worth the effort though.

Indeed it worked :) I updated the results

Regarding the location of the perf scripts, I created perf/ in tests/ and I placed them there. The reason why I don't want to put them into the package is that they are not tested (and doesn't make sense to test them in my opinion), and they will increase lines of code of the package while decreasing the overall coverage. Also they're not something that we wish is installed together with the package. I'm still not sure of the best place for them, but I'd say not in deeprank2/ code base folder.

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
tests/perf/ppi_perf.py Outdated Show resolved Hide resolved
tests/perf/srv_perf.py Show resolved Hide resolved
tests/perf/srv_perf.py Show resolved Hide resolved
Co-authored-by: Dani Bodor <d.bodor@esciencecenter.nl>
README.md Outdated Show resolved Hide resolved
Co-authored-by: Dani Bodor <d.bodor@esciencecenter.nl>
@gcroci2 gcroci2 merged commit 9b50219 into main Sep 22, 2023
@gcroci2 gcroci2 deleted the perf_table_gcroci2 branch September 22, 2023 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add computational performance table
2 participants