[PPT] - How to Automated testing for: terraform test docker packer PowerPoint Presentation

SLIDE 1

Automated testing for: ✓ terraform ✓ docker ✓ packer ✓ kubernetes ✓ and more Passed: 5. Failed: 0. Skipped: 0. Test run successful.

How to test infrastructure code

SLIDE 2

The DevOps world is full of Fear

SLIDE 3

Fear of outages

SLIDE 4

Fear of security breaches

SLIDE 5

Fear of data loss

SLIDE 6

Fear of change

SLIDE 7

“Fear leads to

anger. Anger

leads to hate. Hate leads to suffering.”

Scrum Master Yoda

SLIDE 8

And you all know what suffering leads to, right?

SLIDE 9

SLIDE 10

Credit: Daniele Polencic

SLIDE 11

SLIDE 12

SLIDE 13

Many DevOps teams deal with this fear in two ways:

SLIDE 14

1) Heavy drinking and smoking

SLIDE 15

2) Deploying less frequently

SLIDE 16

Sadly, both of these just make the problem worse!

SLIDE 17

SLIDE 18

There’s a better way to deal with this fear:

SLIDE 19

Automated tests

SLIDE 20

Automated tests give you the confidence to make changes

SLIDE 21

Fight fear with confidence

SLIDE 22

We know how to write automated tests for application code…

SLIDE 23

resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... }

But how do you test your Terraform code deploys infrastructure that works?

SLIDE 24

apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers:

name: hello-world-app

image: gruntwork-io/hello-world-app:v1 ports:

containerPort: 8080

How do you test your Kubernetes code configures your services correctly?

SLIDE 25

This talk is about how to write tests for your infrastructure code.

SLIDE 26

I’m I’m Yev Yevgeni eniy Br Brikman

ybrikman.com

SLIDE 27

Co-founder of Gruntwork

gruntwork.io

SLIDE 28

Author

SLIDE 29

1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion

Outline

SLIDE 30

1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion

Outline

SLIDE 31

Static analysis: test your code without deploying it.

SLIDE 32

Stati Static an anal alysis

1. 1. Co Compiler / / par arser / / interpreter 2. 2. Li Linte ter 3. 3. Dr Dry run

SLIDE 33

Stati Static an anal alysis

1. 1. Co Compiler / / par arser / / interpreter 2. 2. Li Linte ter 3. 3. Dr Dry run

SLIDE 34

Statically check your code for syntactic and structural issues

SLIDE 35

Tool Command

Terraform

terraform validate

Packer

packer validate <template>

Kubernetes

kubectl apply -f <file> --dry-run --validate=true

Examples:

SLIDE 36

Stati Static an anal alysis

1. 1. Co Compiler / / par arser / / interpreter 2. 2. Li Linte ter 3. 3. Dr Dry run

SLIDE 37

Statically validate your code to catch common errors

SLIDE 38

Tool Linters

Terraform

1. conftest
2. terraform_validate
3. tflint

Docker

1. dockerfile_lint
2. hadolint
3. dockerfilelint

Kubernetes

1. kube-score
2. kube-lint
3. yamllint

Examples:

SLIDE 39

Stati Static an anal alysis

1. 1. Co Compiler / / par arser / / interpreter 2. 2. Li Linte ter 3. 3. Dr Dry run

SLIDE 40

Partially execute the code and validate the “plan”, but don’t actually deploy

SLIDE 41

Tool Dry run options

Terraform

1. terraform plan
2. HashiCorp Sentinel
3. terraform-compliance

Kubernetes

kubectl apply -f <file> --server-dry-run

Examples:

SLIDE 42

1. Static analysis
2. Unit tests
3. Integration tests
4. End-to-end tests
5. Conclusion

Outline

SLIDE 43

Unit tests: test a single “unit” works in isolation.

SLIDE 44

Un Unit test sts

1. 1. Un Unit testing basics 2. 2. Ex Exampl ple: Terra rrafo form rm unit t te tests ts 3. 3. Ex Exampl ple: Docker/ r/Kube bern rnete tes unit t te tests ts 4. 4. Cl Clean aning up af after tests

SLIDE 45

Un Unit test sts

1. 1. Un Unit testing basics 2. 2. Ex Exampl ple: Terra rrafo form rm unit t te tests ts 3. 3. Ex Exampl ple: Docker/ r/Kube bern rnete tes unit t te tests ts 4. 4. Cl Clean aning up af after tests

SLIDE 46

You can’t “unit test” an entire end- to-end architecture

SLIDE 47

Instead, break your infra code into small modules and unit test those!

module module module module module module module module module module module module module module module

SLIDE 48

With app code, you can test units in isolation from the outside world

SLIDE 49

resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... }

But 99% of infrastructure code is about talking to the outside world…

SLIDE 50

resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... }

If you try to isolate a unit from the

utside world, you’re left with nothing!

SLIDE 51

So you can only test infra code by deploying to a real environment

SLIDE 52

Key takeaway: there’s no pure unit testing for infrastructure code.

SLIDE 53

Therefore, the test strategy is:

1. Deploy real infrastructure 2. Validate it works

(e.g., via HTTP requests, API calls, SSH commands, etc.)

3. Undeploy the infrastructure

(So it’s really integration testing of a single unit!)

SLIDE 54

Tool Deploy / Undeploy Validate Works with Terratest

Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc.

kitchen-terraform

Yes Yes Terraform

Inspec

No Yes Servers, Cloud APIs

Serverspec

No Yes Servers

Goss

No Yes Servers

Tools that help with this strategy:

SLIDE 55

Tool Deploy / Undeploy Validate Works with Terratest

Yes Yes Terraform, Kubernetes, Packer, Docker, Servers, Cloud APIs, etc.

kitchen-terraform

Yes Yes Terraform

Inspec

No Yes Servers, Cloud APIs

Serverspec

No Yes Servers

Goss

No Yes Servers

In this talk, we’ll use Terratest:

SLIDE 56

Un Unit test sts

1. 1. Un Unit testing basics 2. 2. Ex Exampl ple: Terra rrafo form rm unit t te tests ts 3. 3. Ex Exampl ple: Docker/ r/Kube bern rnete tes unit t te tests ts 4. 4. Cl Clean aning up af after tests

SLIDE 57

Sample code for this talk is at:

github.com/gruntwork-io/infrastructure-as-code-testing-talk

SLIDE 58

An example of a Terraform module you may want to test:

SLIDE 59

infrastructure-as-code-testing-talk └ examples └ hello-world-app └ main.tf └ outputs.tf └ variables.tf └ modules └ test └ README.md

hello-world-app: deploy a “Hello, World” web service

SLIDE 60

resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... }

Under the hood, this example runs on top of AWS Lambda & API Gateway

SLIDE 61

$ terraform apply Outputs: url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com $ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com Hello, World!

When you run terraform apply, it deploys and outputs the URL

SLIDE 62

Let’s write a unit test for hello-world-app with Terratest

SLIDE 63

infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ README.md

Create hello_world_app_test.go

SLIDE 64

func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) }

The basic test structure

SLIDE 65

func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) }

1. Tell Terratest where your Terraform

code lives

SLIDE 66

func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) }

2. Run terraform init and terraform

apply to deploy your module

SLIDE 67

func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) }

3. Validate the infrastructure works.

We’ll come back to this shortly.

SLIDE 68

func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) }

4. Run terraform destroy at the end of

the test to undeploy everything

SLIDE 69

func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) }

The validate function

SLIDE 70

func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) }

1. Run terraform output to get the web

service URL

SLIDE 71

func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) }

2. Make HTTP requests to the URL

SLIDE 72

func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) }

3. Check the response for an expected

status and body

SLIDE 73

func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) }

4. Retry the request up to 10 times, as

deployment is asynchronous

SLIDE 74

Note: since we’re testing a web service, we use HTTP requests to validate it.

SLIDE 75

Infrastructure Example Validate with… Example

Web service

Dockerized web app HTTP requests Terratest http_helper package

Server

EC2 instance SSH commands Terratest ssh package

Cloud service

SQS Cloud APIs Terratest aws or gcp packages

Database

MySQL SQL queries MySQL driver for Go

Examples of other ways to validate:

SLIDE 76

$ export AWS_ACCESS_KEY_ID=xxxx $ export AWS_SECRET_ACCESS_KEY=xxxxx

To run the test, first authenticate to AWS

SLIDE 77

$ go test -v -timeout 15m -run TestHelloWorldAppUnit …

-- PASS: TestHelloWorldAppUnit (31.57s)

Then run go test. You now have a unit test you can run after every commit!

SLIDE 78

Un Unit test sts

1. 1. Un Unit testing basics 2. 2. Ex Exampl ple: Terra rrafo form rm unit t te tests ts 3. 3. Ex Exampl ple: Docker/ r/Kube bern rnete tes unit t te tests ts 4. 4. Cl Clean aning up af after tests

SLIDE 79

What about other tools, such as Docker + Kubernetes?

SLIDE 80

infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ Dockerfile └ deployment.yml └ modules └ test └ README.md

docker-kubernetes: deploy a “Hello, World” web service to Kubernetes

SLIDE 81

FROM ubuntu:18.04 EXPOSE 8080 RUN DEBIAN_FRONTEND=noninteractive apt-get update && \ apt-get install -y busybox RUN echo 'Hello, World!' > index.html CMD ["busybox", "httpd", "-f", "-p", "8080"]

Dockerfile: Dockerize a simple “Hello, World!” web service

SLIDE 82

apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-app-deployment spec: selector: matchLabels: app: hello-world-app replicas: 1 spec: containers:

name: hello-world-app

image: gruntwork-io/hello-world-app:v1 ports:

containerPort: 8080

deployment.yml: define how to deploy a Docker container in Kubernetes

SLIDE 83

$ cd examples/docker-kubernetes $ docker build -t gruntwork-io/hello-world-app:v1 . Successfully tagged gruntwork-io/hello-world-app:v1 $ kubectl apply -f deployment.yml deployment.apps/hello-world-app-deployment created service/hello-world-app-service created $ curl localhost:8080 Hello, World!

Build the Docker image, deploy to Kubernetes, and check URL

SLIDE 84

Let’s write a unit test for this code.

SLIDE 85

infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ docker_kubernetes_test.go └ README.md