Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to modify the regression threshold #11

Open
Tinovdk opened this issue Jul 1, 2022 · 1 comment
Open

Unable to modify the regression threshold #11

Tinovdk opened this issue Jul 1, 2022 · 1 comment

Comments

@Tinovdk
Copy link

Tinovdk commented Jul 1, 2022

I'm setting up an automated performance testing pipeline using Unity and Jenkins but I'm running into the issue where even the slightest regression is marked as a failure. Is there a way to adjust the regression threshold? I noticed in the 1.0 manual this was once possible but has since been removed for some reason.

image

cds-mirroring-service-user pushed a commit that referenced this issue Apr 25, 2023
fix standard deviation being added to threshold automatically
@magnoahlia
Copy link

I know this is a fairly old post but you can safely add the Threshold property back to SampleGroup to allow setting the threshold.

  1. Copy the Performance Testing Extension package contents from the package cache into your project's Packages folder, and change the PTE entry in manifest.json to embedded. (Docs)
  2. Edit Runtime/SampleGroup.cs and add the Threshold property:
/// <summary>
/// The threshold, as a percentage of the aggregated sample group value, to use for regression detection.
/// </summary>
/// <remarks>
/// This value is not used directly in the Performance Testing Extension, but recorded for later use in a reporting tool (such as the Unity Performance Benchmark Reporter) to determine whether or not a performance regression has occurred when used with a baseline result set.
/// NOTE: Removed sometime around 2.0 of the Unity package without a note, added back in by in our custom version of the package.
/// </remarks>
public double Threshold = 0.15;
  1. (Optional) Add threshold to the SampleGroup constructor:
public SampleGroup(string name, SampleUnit unit = SampleUnit.Millisecond, double threshold = 0.15, bool increaseIsBetter = false)

The Threshold property will now be serialized in the test results JSON and picked up by the Performance Benchmark Reporter!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants