[wip] [serverless] Add S3 span pointers

[nhulston]

What does this PR do?

Adds span pointers in S3 for putObject, copyObject, and completeMultipartUpload requests.

Span pointers are similar to Span Links, but for cases when it is impossible to pass the Trace ID and Span ID between the spans that need to be linked.

The changes in this tracer handle the 'upstream' case, for when a Lambda/EC2 instance/anything else using the tracer makes an S3 request. The changes in this PR in the serverless agent handles the 'downstream' case, for when a Lambda is triggered by an S3 update.

Both sides calculate a unique hash for the S3 update. When the calculated hashes for the upstream and downstream sides match, the Datadog backend will automatically link the two traces together.

When clicking on the linked span, a new tab opens linking to the downstream Lambda function that was triggered by this S3 object update. When the link in the downstream Lambda is clicked, it links back to the upstream trace.

Motivation

This is a new feature the Serverless team is implementing in every Lambda runtime.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[datadog-datadog-prod-us1]

Datadog Report

Branch report: nicholas.hulston/s3-span-pointers
Commit report: 543f2ce
Test service: dd-trace-go

❌ 7 Failed (0 Known Flaky), 5173 Passed, 71 Skipped, 2m 35.76s Total Time

❌ Failed Tests (7)

This report shows up to 5 failed tests.

• TestWithErrorCheck - gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/aws-sdk-go-v2/aws - Details

  Expand for error

   Failed
   
   === RUN   TestWithErrorCheck
   --- FAIL: TestWithErrorCheck (0.06s)
  

• TestWithErrorCheck/with_defaults - gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/aws-sdk-go-v2/aws - Details

  Expand for error

   Failed
   
   === RUN   TestWithErrorCheck/with_defaults
   Finishing span
       aws_test.go:1211: 
           	Error Trace:	/home/runner/work/dd-trace-go/dd-trace-go/contrib/aws/aws-sdk-go-v2/aws/aws_test.go:1211
           	Error:      	Not equal: 
           	            	expected: true
           	            	actual  : false
           	Test:       	TestWithErrorCheck/with_defaults
   ...
  

• TestWithErrorCheck/with_errCheck_true - gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/aws-sdk-go-v2/aws - Details

  Expand for error

   Failed
   
   === RUN   TestWithErrorCheck/with_errCheck_true
   Finishing span
       aws_test.go:1211: 
           	Error Trace:	/home/runner/work/dd-trace-go/dd-trace-go/contrib/aws/aws-sdk-go-v2/aws/aws_test.go:1211
           	Error:      	Not equal: 
           	            	expected: true
           	            	actual  : false
           	Test:       	TestWithErrorCheck/with_errCheck_true
   ...
  

• TestHandleS3Operation - gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/internal/span_pointers - Details

  Expand for error

   Failed
   
   === RUN   TestHandleS3Operation
   --- FAIL: TestHandleS3Operation (0.01s)
  

• TestHandleS3Operation/basic_operation - gopkg.in/DataDog/dd-trace-go.v1/contrib/aws/internal/span_pointers - Details

  Expand for error

   Failed
   
   === RUN   TestHandleS3Operation/basic_operation
   Adding link...
       --- FAIL: TestHandleS3Operation/basic_operation (0.00s)
   panic: implement me [recovered]
   	panic: implement me
   
   goroutine 84 [running]:
   testing.tRunner.func1.2({0x1206a80, 0x14e75e0})
   ...
  

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-14 20:17:40

Comparing candidate commit 02da8e3 in PR branch nicholas.hulston/s3-span-pointers with baseline commit 27624f6 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 1 unstable metrics.

[nhulston]

the aws span should live until we receive a response, not when the request is sent. (this is how AWS spans work in other tracers)

From my testing this has barely any actual impact on the span end time (does anyone know why? I'd expect there to be a big difference?)

Therefore, I'm moving span.Finish() to deserialize. I also had to do this to get unit tests to pass