Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document recommended benchmarking practice #36

Open
heshamelmatary opened this issue Sep 25, 2023 · 2 comments
Open

Document recommended benchmarking practice #36

heshamelmatary opened this issue Sep 25, 2023 · 2 comments

Comments

@heshamelmatary
Copy link
Member

Having a set of guidelines while benchmarking CheriBSD will be good. Currently I am aware that we should:

  • Build NODEBUG kernels (which disables INVARIANTS and WITNESS).
  • [malloc] Build with MALLOC_PRODUCTION
@jrtc27
Copy link
Member

jrtc27 commented Sep 25, 2023

We ship kernels built with NODEBUG, and ship libc built with MALLOC_PRODUCTION. Instructions for users should not have any mention of rebuilding CheriBSD, and if we’re missing something they need we should ship it so they don’t have to.

@rwatson
Copy link
Member

rwatson commented Sep 25, 2023

However, it would be good to state that ..

  • State that benchmarking on a prototype microarchitecture and with a research compilers and OSes is challenging, as experimental design can be complex, and the data can be difficult to interpret
  • 23.10 is the first release suitable for generalised benchmarking as:
    • the benchmark ABI exists, should be used, and cross reference to the performance report
    • third-party packages in prior releases were not compiled with optimisation, and included debug symbols
  • If there are experiment design questions, or help is needed in interpreting results, the CHERI-CPU Slack is a good place to ask

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants