Run CATS

[georgethebeatle]

Background

Currently we have a major testing gap around the cf cli and its interaction with the Korifi API. We are only running a handful of smoke tests. In addition we do not have a clear picture of the cf commands that are supported by Korifi.

Action to take

Run the CATS test suite nightly against a Korifi deployment (e.g. acceptance)

Dev notes

• Fork the CATS test repo (KATS?)
• Implement wrappers arount ginkgo's It, When, Describe, e.g. QIt (QIt stands for question it, or quantum It) that runs the test and expects it to fail. If it does not fail, it should prompt us that this test could be altered to It as we have apparently make it work.

With this setup, we could:

• First, replace all Its with QIts and run tests
• See which tests are reported as supported in the meanwhile and change their QIts back to Its
• As we carry on implementing new endpoints, future failures should prompt us that we need to replace more QIts with Its. Ideally, in the end we should no longer use QIts.
• The results would also prompt us if a newer cf cli version no longer works e.g. because it uses the API with unsupported parameters

We should also ensure that tests are always run with latest cf cli.

We should evaluate how flaky KATS are. If it turns out to be quite flaky, in order to reduce noise, we could use the --flake-attempts ginkgo flag in order to rerun failed test a couple of times.