Setting DOTNET_SHARED_STORE For Function Deployed In CloudFormation #222

hounddog22030 · 2022-06-10T15:56:13Z

hounddog22030
Jun 10, 2022

We are creating a Lambda Layer in codebuild.

The command we are using is dotnet lambda publish-layer

Then we do a dotnet lambda package with --function-layers (_layer arn_)

This command emits the warning:

Warning: You must set the DOTNET_SHARED_STORE environment variable when deploying the package. If not set the layers specified will not be located by the .NET Core runtime. The trailing '/' is required.

We then deploy the function using CF with a AWS::Lambda::Function and set an environment variable on the funciton of DOTNET_SHARED_STORE to /opt/dotnetcore/store/

But, in the function's CloudWatch log, we get: System.IO.FileNotFoundException: Could not load file or assembly 'Amazon.Lambda.Core

The documentation reads as below:

The .NET Core runtime in Lambda needs to be configured to look for the NuGet assemblies from the layer. This is done by setting the DOTNET_SHARED_STORE environment variable to the paths of where the runtime package store layer will be extracted. If multiple layers are used with different opt directories then each directory should be added to the environment variable separated by colons. A trailing / is required for the DOTNET_SHARED_STORE .NET Core environment variable.

What might we be doing wrong?

The other weird part is the error message is Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Amazon.Lambda.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

We are using Amazon.Lambda.Core Version 2.1.0

The layer manifest has <Package Id="Amazon.Lambda.Core" Version="2.1.0" />

The function failing has no direct reference to Amazon.Lambda.Core but includes <PackageReference Include="AWSSDK.CloudFront" Version="3.7.4.39" /> which depends on Amazon.Lambda.Core 3.7.11.13

Answered by ashishdhingra

Sep 5, 2023

UPDATE:
@hounddog22030 Apologies for the delay.

Looks like you were using incorrect value net60 for --framework parameter in your dotnet lambda publish-layer command. The correct command is dotnet lambda publish-layer MyLayer --s3-bucket testbucket --framework net6.0 --layer-type runtime-package-store. I noticed this when running dotnet lambda list-layers where it was not showing value for Compatible Runtimes for the published layer. After correcting the value of --framework parameter to net6.0, it displayed value dotnet6 for Compatible Runtimes when using dotnet lambda list-layers command.

Kindly refer Target frameworks in SDK-style projects for valid TFM values.

After the new version of…

View full answer

ashishdhingra · 2022-06-14T20:51:49Z

ashishdhingra
Jun 14, 2022
Maintainer

@hounddog22030 Are you referring to the documentation at https://github.com/aws/aws-extensions-for-dotnet-cli/blob/master/docs/Layers.md? As mentioned in the documentation, it specifies that the deploy-function, deploy-serverless and package-ci commands will take care of setting the DOTNET_SHARED_STORE environment variable. It also specified that The package command will output the value that DOTNET_SHARED_STORE must be set to when the deployment bundle is created by the package command. When the deployment bundle is deployed ensure the DOTNET_SHARED_STORE environment variable is set to the value the package command outputted.. Could you manually set this environment variable and see if the error is gone?

0 replies

hounddog22030 · 2022-06-14T21:55:57Z

hounddog22030
Jun 14, 2022
Author

Could you manually set this environment variable and see if the error is gone?<

Set the environment variable where? We are already setting an environment variable on the function of DOTNET_SHARED_STORE to /opt/dotnetcore/store/.

deploy-function, deploy-serverless and package-ci<<

We are not using those commands, we are using dotnet lambda package.

Let me try to review the full picture:
We would like to:

Package a layer in codebuild
Upload that layer to the artifact store
Package a function that uses that layer in codebuild
Deploy the layer (using AWS::Lambda::LayerVersion) and the function (from step 3 - using AWS::LambdaFunction) in a cloudformation template.

We haven't even really tried that, we published the layer from codebuild to the Lambda Function Layers store and then attempted step 3 and 4 and got to this spot, so we haven't even attempted pulling the layer as a package using AWS::Lambda::LayerVersion

2 replies

ashishdhingra Jun 15, 2022
Maintainer

@hounddog22030 Could you please share the minimal code solution and the CloudFormation template for steps 3 and 4, to investigate further?

hounddog22030 Jun 15, 2022
Author

How do you want the code shared? I have it ready.

normj · 2022-06-16T06:42:17Z

normj
Jun 16, 2022
Maintainer

In the CloudFormation template are you setting the Layer's arn on the AWS::Lambda::Function resource?

"Get": {
    "Type": "AWS::Serverless::Function",
    "Properties": {
    "Handler": "AWSServerless92::AWSServerless92.Functions::Get",
    "Runtime": "dotnet6",
    "CodeUri": "",
    "MemorySize": 256,
    "Timeout": 30,
    "Role": null,
    "Policies": [
        "AWSLambdaBasicExecutionRole"
    ],
    "Layers" : [
        "<layer-arn>"
    ]
    }
}

0 replies

hounddog22030 · 2022-06-16T11:45:47Z

hounddog22030
Jun 16, 2022
Author

@normj

Without know where to post "all" the code for a minimal repo, here's the steps I just did.

dotnet lambda publish-layer (on a referenced library that contains AWSSDK.S3 - recorded the output ARN
added the above ARN to the template as below

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Resources": {
        "LF2FYH5": {
            "Type": "AWS::Lambda::Function",
            "Properties": {
              "Handler" : "Functions::Functions.Function::FunctionHandler",
              "Runtime": "dotnet6",
              "Code": "C:\\Users\\17034\\source\\repos\\LayerRepo\\Functions\\bin\\Release\\net6.0\\Functions.zip",
              "Role": "arn:aws:iam::768033286672:role/yadayada-master-deploy-MasterPipelineRole-ECL43U7BVEYY",
              "Environment" : {
                "Variables" : { "DOTNET_SHARED_STORE":"/opt/dotnetcore/store/"}
                },
                "Layers" : ["arn:aws:lambda:us-east-1:768033299999:layer:Layer1:1"]
              }
        }
    }
}

performed aws cloudformation package --template-file usinglayers.template --s3-bucket ainsley3d --output-template-file output.template
performed aws cloudformation deploy --template-file C:\Users\17034\source\repos\LayerRepo\output.template --stack-name usinglayers
Executed the lambda function in the template via the console "Test" pane

Received error message 2022-06-16T11:36:04.981Z f70f4b39-d89e-42ca-8664-f769a25baf2d fail System.IO.FileNotFoundException: Could not load file or assembly 'AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'. The system cannot find the file specified.

The layer artifact.xml looks correct:

<StoreArtifacts>
  <Package Id="AWSSDK.Core" Version="3.7.12" />
  <Package Id="AWSSDK.S3" Version="3.7.9.16" />
</StoreArtifacts>

0 replies

hounddog22030 · 2022-07-01T15:28:21Z

hounddog22030
Jul 1, 2022
Author

This is what I have done, I don't know if I am missing something or there's a bug?

Here's some very easy to follow steps to reproduce without having to provide a code sample. These steps are created by following as closely as possible to One Of @normj Posts

Most steps can be completed from a command line. There are assumptions about editing aws-lambda-tools-defaults.json to make this work in your environment that anyone helping with this will know are necessary and how to complete.

mkdir layer-test
cd layer-test
dotnet new serverless.AspNetCoreWebAPI
cd src
dotnet new classlib --name Dependencies
dotnet new sln
dotnet sln add layer-test\layer-test.csproj
dotnet sln add Dependencies\Dependencies.csproj
dotnet add package Amazon.Lambda.AspNetCoreServer --source https://api.nuget.org/v3/index.json
dotnet add Dependencies\Dependencies.csproj package Amazon.Lambda.AspNetCoreServer --source https://api.nuget.org/v3/index.json
cd Dependencies
dotnet lambda publish-layer MyLayer --s3-bucket MyBucket --framework net60 --layer-type runtime-package-store
record the results from step 12 above (e.g. arn:aws:lambda:us-east-1:768099999672:layer:MyLayer:2) - you will use in a few steps below
Update the function in the template, adding Layers:["arn:aws:lambda:us-east-1:768099999672:layer:MyLayer:2"] to the "AspNetCoreFunction.Properties" json.
cd layer-test
dotnet lambda deploy-serverless Xyz --function-layers arn:aws:lambda:us-east-1:768099999672:layer:MyLayer:2
copy the url output from the deployment output in step 16 and navigate in your browser https://xxxxxx.execute-api.us-east-1.amazonaws.com/Prod/api/values

Observed Behavior:
Browser: HTTP 500: {"message": "Internal server error"}
CloudWatch Log: Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Amazon.Lambda.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

Expected Behavior:
Browser: HTTP 200: {"value1","value2"}
CloudWatch Log: No Errors

FYI: The manifest in S3 contains:

<StoreArtifacts>
  <Package Id="Amazon.Lambda.APIGatewayEvents" Version="2.4.1"/>
  <Package Id="Amazon.Lambda.ApplicationLoadBalancerEvents" Version="2.1.0"/>
  <Package Id="Amazon.Lambda.AspNetCoreServer" Version="7.2.0"/>
  <Package Id="Amazon.Lambda.Core" Version="2.1.0"/>
  <Package Id="Amazon.Lambda.Logging.AspNetCore" Version="3.1.0"/>
  <Package Id="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.0"/>
  <Package Id="Microsoft.Extensions.Configuration" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.Configuration.Abstractions" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.Configuration.Binder" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.DependencyInjection.Abstractions" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.Logging" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.Logging.Abstractions" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.Options" Version="2.1.0"/>
  <Package Id="Microsoft.Extensions.Primitives" Version="2.1.0"/>
  <Package Id="System.Runtime.CompilerServices.Unsafe" Version="4.5.0"/>
</StoreArtifacts>

4 replies

hounddog22030 Jul 13, 2022
Author

Can anyone help with this?

hounddog22030 Sep 13, 2022
Author

Can anyone help with this? I spent a good amount of time trying to lay out the cleanest steps to reproduce. Thanks.

ashishdhingra Sep 5, 2023
Maintainer

Reproducible even after setting environment via command line dotnet lambda deploy-serverless Xyz --function-layers arn:aws:lambda:us-east-2:<<account-id>>:layer:MyLayer:1 --environment-variables DOTNET_SHARED_STORE=/opt/dotnetcore/store/ and also ensuring that this environment variable is set at function configuration level. Getting error Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Amazon.Lambda.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.. Unsure why it is looking for Amazon.Lambda.Core, Version=1.0.0.0 when version 2.1.0 is indirectly referenced by Amazon.Lambda.AspNetCoreServer version 8.1.0 package.

CC @normj

EDIT: Amazon.Lambda.Core, Version=1.0.0.0 is the assembly version, not the package version.

ashishdhingra Sep 5, 2023
Maintainer

UPDATE:
@hounddog22030 Apologies for the delay.

Looks like you were using incorrect value net60 for --framework parameter in your dotnet lambda publish-layer command. The correct command is dotnet lambda publish-layer MyLayer --s3-bucket testbucket --framework net6.0 --layer-type runtime-package-store. I noticed this when running dotnet lambda list-layers where it was not showing value for Compatible Runtimes for the published layer. After correcting the value of --framework parameter to net6.0, it displayed value dotnet6 for Compatible Runtimes when using dotnet lambda list-layers command.

Kindly refer Target frameworks in SDK-style projects for valid TFM values.

After the new version of layer was published:

modified the serverless.template to set the new value for Layers and DOTNET_SHARED_STORE environment variable.

"Properties": {
        "Handler": "layer-test::layer_test.LambdaEntryPoint::FunctionHandlerAsync",
        ...,
        "Layers":["arn:aws:lambda:us-east-2:<<account-id>>:layer:MyLayer:2"],
        "Environment":{
            "Variables": {
                "DOTNET_SHARED_STORE": "/opt/dotnetcore/store/"
            }
        },
        ...
      }

Executed the dotnet lambda deploy-serverless Xyz --s3-bucket testbucket command.
Opened URL https://<<some-id>>.execute-api.us-east-2.amazonaws.com/Prod/api/values in web browser. It works fine now.

Thanks,
Ashish

@github-actions proposed-answer

Answer selected by ashishdhingra

2023-12-01T20:07:38Z

github-actions[bot]
bot Dec 1, 2023

Hello! Reopening this discussion to make it searchable.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setting DOTNET_SHARED_STORE For Function Deployed In CloudFormation #222

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Setting DOTNET_SHARED_STORE For Function Deployed In CloudFormation #222

hounddog22030 Jun 10, 2022

Replies: 6 comments · 6 replies

ashishdhingra Jun 14, 2022 Maintainer

hounddog22030 Jun 14, 2022 Author

ashishdhingra Jun 15, 2022 Maintainer

hounddog22030 Jun 15, 2022 Author

normj Jun 16, 2022 Maintainer

hounddog22030 Jun 16, 2022 Author

hounddog22030 Jul 1, 2022 Author

hounddog22030 Jul 13, 2022 Author

hounddog22030 Sep 13, 2022 Author

ashishdhingra Sep 5, 2023 Maintainer

ashishdhingra Sep 5, 2023 Maintainer

github-actions[bot] bot Dec 1, 2023

hounddog22030
Jun 10, 2022

Replies: 6 comments 6 replies

ashishdhingra
Jun 14, 2022
Maintainer

hounddog22030
Jun 14, 2022
Author

ashishdhingra Jun 15, 2022
Maintainer

hounddog22030 Jun 15, 2022
Author

normj
Jun 16, 2022
Maintainer

hounddog22030
Jun 16, 2022
Author

hounddog22030
Jul 1, 2022
Author

hounddog22030 Jul 13, 2022
Author

hounddog22030 Sep 13, 2022
Author

ashishdhingra Sep 5, 2023
Maintainer

ashishdhingra Sep 5, 2023
Maintainer

github-actions[bot]
bot Dec 1, 2023