Add ability to stress test flakes in CI #4127
Annotations
10 errors, 5 warnings, and 1 notice
Run Playwright tests:
playwright/e2e/settings/account-user-settings-tab.spec.ts#L27
1) [Chrome] › settings/account-user-settings-tab.spec.ts:26:9 › Account user settings tab › should be rendered properly @screenshot
Error: expect(locator).toHaveScreenshot(expected)
1146 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/settings/account-user-settings-tab.spec.ts/account-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-574cd-should-be-rendered-properly-Chrome-repeat4/account-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-574cd-should-be-rendered-properly-Chrome-repeat4/account-diff.png
Call log:
- expect.toHaveScreenshot(account.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 1146 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 1146 pixels (ratio 0.01 of all image pixels) are different.
25 |
26 | test("should be rendered properly", { tag: "@screenshot" }, async ({ uut, user }) => {
> 27 | await expect(uut).toMatchScreenshot("account.png");
| ^
28 |
29 | // Assert that the top heading is rendered
30 | await expect(uut.getByRole("heading", { name: "Account", exact: true })).toBeVisible();
at /home/runner/work/element-web/element-web/playwright/e2e/settings/account-user-settings-tab.spec.ts:27:27
|
Run Playwright tests:
playwright/e2e/settings/account-user-settings-tab.spec.ts#L27
1) [Chrome] › settings/account-user-settings-tab.spec.ts:26:9 › Account user settings tab › should be rendered properly @screenshot
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(locator).toHaveScreenshot(expected)
1146 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/settings/account-user-settings-tab.spec.ts/account-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-574cd-should-be-rendered-properly-Chrome-retry1-repeat4/account-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-574cd-should-be-rendered-properly-Chrome-retry1-repeat4/account-diff.png
Call log:
- expect.toHaveScreenshot(account.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 1146 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 1146 pixels (ratio 0.01 of all image pixels) are different.
25 |
26 | test("should be rendered properly", { tag: "@screenshot" }, async ({ uut, user }) => {
> 27 | await expect(uut).toMatchScreenshot("account.png");
| ^
28 |
29 | // Assert that the top heading is rendered
30 | await expect(uut.getByRole("heading", { name: "Account", exact: true })).toBeVisible();
at /home/runner/work/element-web/element-web/playwright/e2e/settings/account-user-settings-tab.spec.ts:27:27
|
Run Playwright tests:
playwright/e2e/settings/account-user-settings-tab.spec.ts#L27
1) [Chrome] › settings/account-user-settings-tab.spec.ts:26:9 › Account user settings tab › should be rendered properly @screenshot
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(locator).toHaveScreenshot(expected)
1146 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/settings/account-user-settings-tab.spec.ts/account-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-574cd-should-be-rendered-properly-Chrome-retry2-repeat4/account-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-574cd-should-be-rendered-properly-Chrome-retry2-repeat4/account-diff.png
Call log:
- expect.toHaveScreenshot(account.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 1146 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 1146 pixels (ratio 0.01 of all image pixels) are different.
25 |
26 | test("should be rendered properly", { tag: "@screenshot" }, async ({ uut, user }) => {
> 27 | await expect(uut).toMatchScreenshot("account.png");
| ^
28 |
29 | // Assert that the top heading is rendered
30 | await expect(uut.getByRole("heading", { name: "Account", exact: true })).toBeVisible();
at /home/runner/work/element-web/element-web/playwright/e2e/settings/account-user-settings-tab.spec.ts:27:27
|
Run Playwright tests:
playwright/e2e/settings/account-user-settings-tab.spec.ts#L76
2) [Chrome] › settings/account-user-settings-tab.spec.ts:74:9 › Account user settings tab › should respond to small screen sizes @screenshot
Error: expect(locator).toHaveScreenshot(expected)
896 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/settings/account-user-settings-tab.spec.ts/account-smallscreen-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-b4333-spond-to-small-screen-sizes-Chrome-repeat4/account-smallscreen-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-b4333-spond-to-small-screen-sizes-Chrome-repeat4/account-smallscreen-diff.png
Call log:
- expect.toHaveScreenshot(account-smallscreen.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 896 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 896 pixels (ratio 0.01 of all image pixels) are different.
74 | test("should respond to small screen sizes", { tag: "@screenshot" }, async ({ page, uut }) => {
75 | await page.setViewportSize({ width: 700, height: 600 });
> 76 | await expect(uut).toMatchScreenshot("account-smallscreen.png");
| ^
77 | });
78 |
79 | test("should show tooltips on narrow screen", async ({ page, uut }) => {
at /home/runner/work/element-web/element-web/playwright/e2e/settings/account-user-settings-tab.spec.ts:76:27
|
Run Playwright tests:
playwright/e2e/settings/account-user-settings-tab.spec.ts#L76
2) [Chrome] › settings/account-user-settings-tab.spec.ts:74:9 › Account user settings tab › should respond to small screen sizes @screenshot
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(locator).toHaveScreenshot(expected)
896 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/settings/account-user-settings-tab.spec.ts/account-smallscreen-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-b4333-spond-to-small-screen-sizes-Chrome-retry1-repeat4/account-smallscreen-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-b4333-spond-to-small-screen-sizes-Chrome-retry1-repeat4/account-smallscreen-diff.png
Call log:
- expect.toHaveScreenshot(account-smallscreen.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 896 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 896 pixels (ratio 0.01 of all image pixels) are different.
74 | test("should respond to small screen sizes", { tag: "@screenshot" }, async ({ page, uut }) => {
75 | await page.setViewportSize({ width: 700, height: 600 });
> 76 | await expect(uut).toMatchScreenshot("account-smallscreen.png");
| ^
77 | });
78 |
79 | test("should show tooltips on narrow screen", async ({ page, uut }) => {
at /home/runner/work/element-web/element-web/playwright/e2e/settings/account-user-settings-tab.spec.ts:76:27
|
Run Playwright tests:
playwright/e2e/settings/account-user-settings-tab.spec.ts#L76
2) [Chrome] › settings/account-user-settings-tab.spec.ts:74:9 › Account user settings tab › should respond to small screen sizes @screenshot
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(locator).toHaveScreenshot(expected)
896 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/settings/account-user-settings-tab.spec.ts/account-smallscreen-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-b4333-spond-to-small-screen-sizes-Chrome-retry2-repeat4/account-smallscreen-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/settings-account-user-sett-b4333-spond-to-small-screen-sizes-Chrome-retry2-repeat4/account-smallscreen-diff.png
Call log:
- expect.toHaveScreenshot(account-smallscreen.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 896 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('.mx_Dialog').filter({ has: locator('.mx_UserSettingsDialog') })
- locator resolved to <div class="mx_Dialog">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 896 pixels (ratio 0.01 of all image pixels) are different.
74 | test("should respond to small screen sizes", { tag: "@screenshot" }, async ({ page, uut }) => {
75 | await page.setViewportSize({ width: 700, height: 600 });
> 76 | await expect(uut).toMatchScreenshot("account-smallscreen.png");
| ^
77 | });
78 |
79 | test("should show tooltips on narrow screen", async ({ page, uut }) => {
at /home/runner/work/element-web/element-web/playwright/e2e/settings/account-user-settings-tab.spec.ts:76:27
|
Run Playwright tests:
playwright/e2e/share-dialog/share-dialog.spec.ts#L43
3) [Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot
Error: expect(locator).toHaveScreenshot(expected)
333 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/share-dialog/share-dialog.spec.ts/share-dialog-user-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/share-dialog-share-dialog--ba9c4--should-share-a-room-member-Chrome-repeat4/share-dialog-user-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/share-dialog-share-dialog--ba9c4--should-share-a-room-member-Chrome-repeat4/share-dialog-user-diff.png
Call log:
- expect.toHaveScreenshot(share-dialog-user.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for getByRole('dialog', { name: 'Share User' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 333 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for getByRole('dialog', { name: 'Share User' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 333 pixels (ratio 0.01 of all image pixels) are different.
41 | const dialog = page.getByRole("dialog", { name: "Share User" });
42 | await expect(dialog.getByText(`https://matrix.to/#/${user.userId}`)).toBeVisible();
> 43 | await expect(dialog).toMatchScreenshot("share-dialog-user.png", {
| ^
44 | // QRCode changes at every run
45 | mask: [page.locator(".mx_QRCode")],
46 | });
at /home/runner/work/element-web/element-web/playwright/e2e/share-dialog/share-dialog.spec.ts:43:30
|
Run Playwright tests:
playwright/e2e/share-dialog/share-dialog.spec.ts#L43
3) [Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(locator).toHaveScreenshot(expected)
333 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/share-dialog/share-dialog.spec.ts/share-dialog-user-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/share-dialog-share-dialog--ba9c4--should-share-a-room-member-Chrome-retry1-repeat4/share-dialog-user-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/share-dialog-share-dialog--ba9c4--should-share-a-room-member-Chrome-retry1-repeat4/share-dialog-user-diff.png
Call log:
- expect.toHaveScreenshot(share-dialog-user.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for getByRole('dialog', { name: 'Share User' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 333 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for getByRole('dialog', { name: 'Share User' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 333 pixels (ratio 0.01 of all image pixels) are different.
41 | const dialog = page.getByRole("dialog", { name: "Share User" });
42 | await expect(dialog.getByText(`https://matrix.to/#/${user.userId}`)).toBeVisible();
> 43 | await expect(dialog).toMatchScreenshot("share-dialog-user.png", {
| ^
44 | // QRCode changes at every run
45 | mask: [page.locator(".mx_QRCode")],
46 | });
at /home/runner/work/element-web/element-web/playwright/e2e/share-dialog/share-dialog.spec.ts:43:30
|
Run Playwright tests:
playwright/e2e/share-dialog/share-dialog.spec.ts#L43
3) [Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(locator).toHaveScreenshot(expected)
333 pixels (ratio 0.01 of all image pixels) are different.
Expected: /home/runner/work/element-web/element-web/playwright/snapshots/share-dialog/share-dialog.spec.ts/share-dialog-user-linux.png
Received: /home/runner/work/element-web/element-web/playwright/test-results/share-dialog-share-dialog--ba9c4--should-share-a-room-member-Chrome-retry2-repeat4/share-dialog-user-actual.png
Diff: /home/runner/work/element-web/element-web/playwright/test-results/share-dialog-share-dialog--ba9c4--should-share-a-room-member-Chrome-retry2-repeat4/share-dialog-user-diff.png
Call log:
- expect.toHaveScreenshot(share-dialog-user.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for getByRole('dialog', { name: 'Share User' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 333 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for getByRole('dialog', { name: 'Share User' })
- locator resolved to <div role="dialog" class="mx_ShareDialog" data-focus-lock-disabled="false" aria-describedby="mx_Dialog_content" aria-labelledby="mx_BaseDialog_title">…</div>
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 333 pixels (ratio 0.01 of all image pixels) are different.
41 | const dialog = page.getByRole("dialog", { name: "Share User" });
42 | await expect(dialog.getByText(`https://matrix.to/#/${user.userId}`)).toBeVisible();
> 43 | await expect(dialog).toMatchScreenshot("share-dialog-user.png", {
| ^
44 | // QRCode changes at every run
45 | mask: [page.locator(".mx_QRCode")],
46 | });
at /home/runner/work/element-web/element-web/playwright/e2e/share-dialog/share-dialog.spec.ts:43:30
|
Run Playwright tests:
playwright/e2e/timeline/timeline.spec.ts#L1
4) [Chrome] › timeline/timeline.spec.ts:277:13 › Timeline › configure room › should add inline start margin to an event line on IRC layout @screenshot
Test timeout of 30000ms exceeded.
|
Run Playwright tests:
[Chrome] › spotlight/spotlight.spec.ts#L1
[Chrome] › spotlight/spotlight.spec.ts took 2.3m
|
Run Playwright tests:
[Chrome] › read-receipts/redactions-in-threads.spec.ts#L1
[Chrome] › read-receipts/redactions-in-threads.spec.ts took 1.8m
|
Run Playwright tests:
[Chrome] › read-receipts/redactions-main-timeline.spec.ts#L1
[Chrome] › read-receipts/redactions-main-timeline.spec.ts took 1.3m
|
Run Playwright tests:
[Chrome] › read-receipts/new-messages-in-threads.spec.ts#L1
[Chrome] › read-receipts/new-messages-in-threads.spec.ts took 1.3m
|
Run Playwright tests:
[Chrome] › sliding-sync/sliding-sync.spec.ts#L1
[Chrome] › sliding-sync/sliding-sync.spec.ts took 58.3s
|
Run Playwright tests
4 failed
[Chrome] › settings/account-user-settings-tab.spec.ts:26:9 › Account user settings tab › should be rendered properly @screenshot
[Chrome] › settings/account-user-settings-tab.spec.ts:74:9 › Account user settings tab › should respond to small screen sizes @screenshot
[Chrome] › share-dialog/share-dialog.spec.ts:32:9 › Share dialog › should share a room member @screenshot
[Chrome] › timeline/timeline.spec.ts:277:13 › Timeline › configure room › should add inline start margin to an event line on IRC layout @screenshot
3 flaky
[Chrome] › read-receipts/new-messages-main-timeline.spec.ts:50:17 › Read receipts › new messages › in the main timeline › Reading an older message leaves the room unread @mergequeue
[Chrome] › room/room.spec.ts:56:9 › Room Directory › should memorize the timeline position when switch Room A -> Room B -> Room A
[Chrome] › timeline/timeline.spec.ts:343:13 › Timeline › message displaying › should align generic event list summary with messages and emote on IRC layout @screenshot
33 skipped
335 passed (43.9m)
|
Loading