[BrowserTest] Fix flaky gallery test

[christian-byrne]

Make tests added in #2125 more robust by:

• Using alt text (filename) to locate images instead of indices
• Adding 64ms delay after keyboard navigation
• Removing ws fixture

┆Issue is synchronized with this Notion page by Unito

[huchenlei]

I am going to disable these tests first before they are fixed. #2158

[christian-byrne]

It's difficult to debug because I cannot reproduce locally. The flakiness only occurs on the gallery tests.

[webfiltered]

Options to increase flakiness:

1. Run playwright with --workers=10. Depending on your specs this may actually need to be lower.
2. Run tests inside a resource-constrained VM, with 2 vCPUs. Our GH action runners only run 2 threads.
3. Recompile a large C project whilst running tests. 😃

[christian-byrne]

Options to increase flakiness:

1. Run playwright with --workers=10. Depending on your specs this may actually need to be lower.
2. Run tests inside a resource-constrained VM, with 2 vCPUs. Our GH action runners only run 2 threads.
3. Recompile a large C project whilst running tests. 😃

Those are good ideas. I tried options 1 and 3 but it's difficult to find the exact right amount of throttling. Tests begin to fail as I increase network and processor throttling but many other tests fail as well.

I will try option 2 tomorrow with this.

[christian-byrne]

Using act works well:

act pull_request \
  --secret GITHUB_TOKEN=$GH_TOKEN \
  --secret ACTIONS_RUNTIME_TOKEN=$GH_TOKEN \
  --env PLAYWRIGHT_TEST_URL=http://127.0.0.1:8188 \
  --artifact-server-path "$PWD/playwright-report" \
  -j playwright-tests-chromium

Can run (cores - 1) workers and still repro test env errors reliably.

I am able to fix the flakiness by increasing the global timeout in the playwright config beyond 15s. I will try to find a better solution if it's possible.