Allow categorizing integration tests as long running and running them nightly

[cmacknz]

The change in #2955 allows running a test to confirm that every released version can upgrade to the current version. This is valuable as a sanity check, but will take too long to run on every PR. We need a way to categorize these tests as long running so that we can easily filter them out.

This is the first example of a common problem, where we do want to test something, but we don't want to test it by default because the probability of failure is low for the change. Another good example is regularly testing on multiple versions of Ubuntu. We need to test both 22.04 and 18.04, but if the change works on 22.04 there is a very strong chance it also works on 18.04 unless it is making an OS dependant change.

We need to define a way to tag tests as long running or part of an exhaustive test suite, so that we can filter them out of the standard test run and move them to a nightly test job to ensure we still run but aren't burdened with an hours long default test suite.

Go provides the -short flag for this purpose but it would preferable to explicitly opt in to the long running exhaustive test suite instead of having to remember to opt in to the reasonable one.

It might make sense to allow tagging the tests with their heaviness or size in the define directives in the code since we already use that to build a test filter. We could an add an Exhaustive: true field for example.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pchila]

Just an aspect to be taken into account: excluding some tests based on their run time using define will not work for tests that dynamically define subtests (like the upgrade integration tests) since the define clause must be the first statement in a test so it would force people to define 2 separate test (a short one that will always run and a longer one that will test the complete set of testcases).

I would prefer sticking to the standard go way of signaling a short run and then have each test decide what that means in that specific instance instead of having the integration test framework bluntly decide to run or not to run a full test.

That being said, regarding #2955, we already have a way of shortening the run and instead of applying the short run to PRs it can be set everywhere except for specific scheduled jobs.

[cmacknz]

Thanks, I think I agree we can start with just the -short option and have it be provided by default through all the mage targets. This is by far the simplest approach to start with and we can move to something more complex later.

[jlind23]

@pierrehilbert @cmacknz
Now that we have the ability to group tests, shouldn't we close this issue?

[cmacknz]

The group is separate, it controls which tests can run on the same VM and therefore controls test parallelism and the total number of VMs we create.

I don't think there is a strong need for this right now, so closing. We can reopen later if needed.