HttpClient may leak UnobservedTaskException in CheckUsabilityOnScavenge

[salvois]

Description

We have a .NET 9 console application (namely, a Windows Service running on Microsoft.Extensions.Hosting.BackgroundService, hosted by Windows Server 2019 on x86-64) making multiple HTTP requests (GETs or POSTs) using a singleton instance of HttpClient created during service startup using new HttpClient { Timeout = TimeSpan.FromMinutes(10) };

All otherwise unhandled application exceptions are handled by a top level catch that does any logging and deadlettering required for our processing. We also register a handler to TaskScheduler.UnobservedTaskException to log any exception that may leak, usually from our code forgetting an await.

In several cases, under heavy load, our handler for UnobservedTaskException's received exceptions from what appear to be the internals of HttpClient itself, such as in the following stack trace, apparently related to #61256:

System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Http.HttpConnection.<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|39_0()

We also experienced #104324 when the application ran on .NET 8, and upgraded to .NET 9 hoping to solve the issue, but we are still receiving the exception about CheckUsabilityOnScavenge nonetheless.

Reproduction Steps

Unfortunately we have not been able to reproduce the problem deterministically. It seems related to our system being under heavy load.

Expected behavior

We expect no unobserved exceptions, even less exceptions from internal implementation.

Actual behavior

An UnobservedTaskException is caught by the handler registered to TaskScheduler.UnobservedTaskException, which is very scary to us because in our mind "unobserved task excetion == possible data loss" 😄 (due to the deadletter mechanism being bypassed).
If I understand the implementation correctly (which I don't fully, I must confess), that should not be the case in our scenario, though, because the exception seems to be thrown when the HTTP connection is about to be dropped anyway, and our application should proceed normally. Please advise whether this is the case.

Regression?

We don't have memory of this issue nor #104324 prior to switching to .NET 8 (the application used .NET 6, .NET 5, .NET Core 2 and .NET Framework 4.6.2 before).

Known Workarounds

No response

Configuration

No response

Other information

Based on a quick review of the source of CheckUsabilityOnScavenge in HttpConnection, I think a solution similar to PR #104335 may do the job. I can try to create a pull request based on it.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[salvois]

Should it matter, I noticed our application also starts a SignalR client created as follows, to a web app hosted by IIS with Windows authentication enabled:

_hubConnection = new HubConnectionBuilder()
    .WithUrl("https://example.com/app/notificationHub", options => options.UseDefaultCredentials = true)
    .WithAutomaticReconnect(new RandomRetryPolicy())
    .Build();

public class RandomRetryPolicy : IRetryPolicy
{
    public TimeSpan? NextRetryDelay(RetryContext retryContext) => TimeSpan.FromSeconds(Random.Shared.Next(1, 5));
}

Maybe our unobserved exception comes from an HttpClient created by the SignalR client library (if it works like this). We have a log about a re-enstablished SignalR connection on the same machine 40 seconds after the offending UnobservedTaskException.

[MihaZupan]

That fact it's SignalR shouldn't matter, but with Windows authentication enabled could.

Auth is a thing that happens after the PrepareForReuse call. We'll call into here

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs

Lines 76 to 78 in 6045e29

	private static async Task<HttpResponseMessage> SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, bool async, ICredentials credentials, TokenImpersonationLevel impersonationLevel, bool isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
	{
	HttpResponseMessage response = await InnerSendAsync(request, async, isProxyAuth, connectionPool, connection, cancellationToken).ConfigureAwait(false);

, but doing the actual send is the first thing we do, so that should be fine as well.

That method also does some manual connection management (this bit), but that also looks fine at first glance.