Skip to content

07 container demo

Jump to bottom
Paul Duvall edited this page Feb 15, 2021 · 63 revisions

Deploying Container Security Checks in a Pipeline

From your AWS CloudShell Environment in the us-east-2 region, run the following commands:

aws s3api list-buckets --query 'Buckets[?starts_with(Name, `csoa-7-`) == `true`].[Name]' --output text | xargs -I {} aws s3 rb s3://{} --force
sudo rm -rf ~/csoa-7
mkdir ~/csoa-7
cd ~/csoa-7
aws s3 mb s3://csoa-7-$(aws sts get-caller-identity --output text --query 'Account')
git clone https://github.com/PaulDuvall/csoa.git
cd csoa/lesson7-deployment-arch
zip csoa-lesson7.zip *.*
aws s3 sync ~/csoa-7/csoa/lesson7-deployment-arch s3://csoa-7-$(aws sts get-caller-identity --output text --query 'Account')

Now, run the commands to launch the CloudFormation stack.

aws cloudformation create-stack --stack-name csoa-7-container-devsecops --template-body file:///home/cloudshell-user/csoa-7/csoa/lesson7-deployment-arch/csoa-7-container-devsecops.yml --parameters ParameterKey=EmailAddress,ParameterValue=you@example.com ParameterKey=CodeCommitS3Bucket,ParameterValue=csoa-7-$(aws sts get-caller-identity --output text --query 'Account') ParameterKey=CodeCommitS3Key,ParameterValue=csoa-lesson7.zip --capabilities CAPABILITY_NAMED_IAM --disable-rollback --region us-east-2

It takes about 2 minutes to launch the container security pipeline and related resources in your AWS account.

  • Go to the AWS CloudFormation console in the us-east-2 region.
  • Once the CloudFormation stack is complete, go to AWS CodePipeline and choose the csoa-7-container-devsecops pipeline. Both of the actions in the Build stage will fail. This is because you need to configure some files.
  • To do this, go to csoa-7-container-devsecops AWS CodeCommit repository.
  • Create a new Dockerfile file (Source) and add the following code and commit the changes:
# ===== Flawed Dockerfile, do not use for running a Docker Container =====

FROM continuumio/miniconda:latest
RUN apt-get update \
&& apt-get install -y \
   gcc \
   fortunes \
   cowsay \
&& pip install apache-airflow[crypto,postgres]
CMD /usr/games/fortune | /usr/games/cowsay
  • From the same AWS CodeCommit repository, double click on the secrets_config.json file and add the following code snippet and commit the file to CodeCommit:
{
    "Slack Token": "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})",
    "GitHub": "[g|G][i|I][t|T][h|H][u|U][b|B].*['|\"][0-9a-zA-Z]{35,40}['|\"]",
    "Google Oauth": "(\"client_secret\":\"[a-zA-Z0-9-_]{24}\")",
    "Generic Secret": "[s|S][e|E][c|C][r|R][e|E][t|T].*['|\"][0-9a-zA-Z]{32,45}['|\"]",
    "AWS API Key": "AKIA[0-9A-Z]{16}",
    "Slack Webhook": "https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}",
    "Google (GCP) Service-account": "\"type\": \"service_account\"",
    "Password in URL": "[a-zA-Z]{3,10}://[^/\\s:@]{3,20}:[^/\\s:@]{3,20}@.{1,100}[\"'\\s]"
}
  • Go back to AWS CodePipeline to see the pipeline running.
  • You should receive an error in the DockerLint CodeBuild action that looks something like this:
Ranges can only match single chars (mentioned due to duplicates).

Pin versions in apt get install. Instead of `apt-get install <package>` use `apt-get install <package>=<version>

Delete the apt-get lists after installing something

Pin versions in pip. Instead of `pip install <package>` use `pip install <package>==<version>

Avoid additional packages by specifying `--no-install-recommends`
  • Go back to AWS CodeCommit again.
  • Double click on the Dockerfile file and replace the contents of the file with the following code snippet and commit the changes:
FROM python:latest

LABEL maintainer="CSOA"

COPY . /app

WORKDIR /app
RUN pip install -r requirements.txt

ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2

EXPOSE 5000

USER root

CMD python ./index.py
  • Go back to AWS CodePipeline to see the pipeline running.
  • The DockerLint action should now be successful, but you should be receiving an error in the Secrets CodeBuild action that looks something like this:

Reason: AWS API Key

  • From the same AWS CodeCommit repository, double click on the index.py file and remove the line that starts with access_key and commit the file to CodeCommit.

As it goes through the actions in the pipeline, it will perform static analysis of the Dockerfile via Hadolint and check for any secrets that have been commited to the source code repository using trufflehog. It will fail again because the access keys are still in your Git history. To fix this, run the following steps:

  1. Go to the IAM Console.
  2. Click on your current userid.
  3. Select the Security credentials tab.
  4. Within the HTTPS Git credentials for AWS CodeCommit section, click on the Generate credentials button.
  5. Click the Download credentials button and save the file.
  6. From AWS CloudShell, type the following:
cd ~
sudo rm -rf csoa-7-container-devsecops
git clone https://git-codecommit.us-east-2.amazonaws.com/v1/repos/csoa-7-container-devsecops

# Prompt to enter username and password

Complete the next set of steps.

cd csoa-7-container-devsecops

git config --global user.name "NAME"
git config --global user.email your@email.com
git commit --amend --reset-author

Complete the remaining steps.

git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch index.py" \
  --prune-empty --tag-name-filter cat -- --all

git push origin --force --all

# Prompt to enter username and password

CodePipeline will intitate a new pipeline run and it should successfully pass all stages in the pipeline.

View the Source Files

  • Go to AWS CodeCommit and review all of the source files in each of the repositories.

Delete all Resources

  • Go back to AWS CloudShell in the us-east-2 region and run the command below to delete the resources provisioned in this demo. It takes over 10 minutes for all resources to be deleted from your account.
aws s3api list-buckets --query 'Buckets[?starts_with(Name, `csoa-`) == `true`].[Name]' --output text | xargs -I {} aws s3 rb s3://{} --force

# Wait until the S3 buckets have been terminated

aws cloudformation delete-stack --stack-name csoa-7-container-devsecops --region us-east-2

You can verify the stacks have been terminated by going to the CloudFormation Console in the us-east-2 region.

Resources

Clone this wiki locally