E2E: Try to stabilize Atomic likes (again)

[WunderBart]

Related: #75947

Proposed Changes

When adding a comment on an Atomic site, it takes a bit until that comment is settled. Sometimes, when we send the like request immediately after creating a comment, we get an unknown_comment error response. To address that, let's make a couple like request retries before failing the test.

Testing Instructions

TEST_ON_ATOMIC=true yarn workspace wp-e2e-tests test likes__comment.ts

[matticbot]

Calypso Live (direct link)

https://calypso.live?image=registry.a8c.com/calypso/app:build-65239

Jetpack Cloud live (direct link)

https://calypso.live?image=registry.a8c.com/calypso/app:build-65239&env=jetpack

[WunderBart]

Just using some modern syntax here, not a crucial change.

[worldomonation]

I had an experimental branch where I made a similar change, so I can vouch it works.

[WunderBart]

Changed it to it because the title gives a bit more info immediately when looking at the error in CI.

[matticbot]

This PR does not affect the size of JS and CSS bundles shipped to the user's browser.

Generated by performance advisor bot at iscalypsofastyet.com.

[WunderBart]

@worldomonation I think there's something wrong with the comment like request. The request retries ensure that the response is what we expect it to be - a liked comment with the given ID. Even with that out of the way, that very comment doesn't always render liked❗ Maybe we should look a bit further into this? Maybe it's some caching issue?

/cc @fullofcaffeine

---

Edit: I was right 😄

This is what the like response looks like after a few retries:

First try:

{
  error: 'unknown_comment',
  message: 'Unknown comment'
}

A few retries later. Notice how the comment is seemingly liked (i_like: true), but the like_count is still 0.

{
  success: true,
  i_like: true,
  like_count: 0,
}

A couple more retries later, finally, a correct response. The comment is liked by us, and the count is now correctly at 1. 😌

{
  success: true,
  i_like: true,
  like_count: 1,
}

This is quite a doozy to me. You can clearly see the payload inconsistency now. Possibly worth reporting to the AT team?

---

Another edit:

So the interesting part right now is that the response can get stuck at like_count: 0 even after 30 retries (with 1s delay between). This is something that we might not be able to handle and which will keep making this spec flaky. Definitely, something to tackle with the AT team at this point?

[worldomonation]

comment:non-blocking - I have a WIP branch locally to try to stabilize this test as well, and I tried to extract the retry mechanism to ElementHelper and ran into difficulties generalizing the parameter definition and forgot to ask @dpasque whether generics would help with the scenario.

What I had was the following:

// In the test itself
		if ( envVariables.TEST_ON_ATOMIC ) {
			ElementHelper.retryRequest(
				restAPIClient.getComment(
					testAccount.credentials.testSites?.primary.id as number,
					commentToBeLiked.ID
				)
			);
		}

// in ElementHelper
export async function retryRequest< T, P >(
	func: () => Promise< void >,
	parameters: [ P ],
	{ timeout = 5000 }: { timeout?: number } = {}
): Promise< void > {
	for ( let retries = 3; retries > 0; retries -= 1 ) {
		try {
			return await func( ...parameters );
		} catch ( err ) {
			// Throw the error if final retry failed.
			if ( retries === 1 ) {
				throw err;
			} else {
				await new Promise( ( resolve ) => setTimeout( resolve, timeout ) );
			}
		}
	}
	return;
}

[WunderBart]

If we only need the retry logic in one place, I don't think we don't need to worry about turning it into a helper. Is there any other place that it'd currently be useful?

[WunderBart]

I'd even say that this should generally not be encouraged, so I'd actually avoid creating a helper for it. We need this here because of a faulty app logic which, ultimately, should be addressed somewhere else.

[worldomonation]

So the interesting part right now is that the response can get stuck at like_count: 0 even after 30 retries (with 1s delay between). This is something that we might not be able to handle and which will keep making this spec flaky. Definitely, something to tackle with the AT team at this point?

This is definitely strange.

Previously I've filed #75952 for the first situation you've covered, where the comment is posted but the API returns a 404 when attempting to access the comment.

I did waffle a bit on whether to expand the scope of #75952 to cover the later two situations you've mentioned, but have elected to file a new issue to cover the situation for like_count and i_like being out of sync.

Filed #76444.

[worldomonation]

I've left a message in the AT channel, but not sure if it's the right place. p1682922721277819-slack-C7YPW6K40