You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The pass/fail output from a validation run is too simple, really, to be very useful. The validation runs should:
Catch definitely wrong behaviour.
Allow comparisons of fidelity between builds and simulators.
Proposal:
Make the pass criterion for the tests quite lax, so that FAIL really means that something went wrong.
Produce a 'standard' report output from the validation tests. This would follow a well defined schema and contain a list of derived values of interest, their pass thresholds (and less than, greater than, etc.), and metadata for sim and nsuite version.
Manually running the validation tests in a new 'verbose' mode would output a human- and script- readable version of this data.
Reports from multiple tests in the same validation run should be able to be combined and presented in a form that is easy for programmatic consumption, e.g. for presentation in Jenkins or other tools.
The text was updated successfully, but these errors were encountered:
The pass/fail output from a validation run is too simple, really, to be very useful. The validation runs should:
Proposal:
The text was updated successfully, but these errors were encountered: