How to implement continuous delivery and testing Tamas Csako - - PowerPoint PPT Presentation

1

How to implement continuous delivery and testing

Tamas Csako

November 2019

• 10+ years experience in IT
• 6+ years experience in managing and

leading teams

• Wide technology overview
• Community builder, job interviewer,

mentor, coach, team player, consultant TAMAS CSAKO, PH.D.

Senior Software Testing Manager Head of Cloud & DevTestSecOps Practice EPAM Hungary

3

Agenda

1

I N T R O D U C T I O N A N D O V E R V I E W

2 3 4

O U R B E S T P R A C T I C E S C O N T I N U O U S D E L I V E R Y A N D T E S T I N G S U M M A R Y

3

4

EPAM Cloud and DevTestSecOps Practice

KEY FACTS PARTNERSHIPS SERVICES MOSTLY USED TOOLS AND TECHNOLOGIES

Active Projects

20+ 400+

Global Delivery Centers Cloud, DevOps & Test engineers

8,000

Cloud enablement, migration and engineering DevOps Transformation, SDLC Automation Cyber Security AIOps - AI for IT operations Application & Infrastructure management services (AMS & IMS) Testing

5

Continuous Testing Blueprint

CI / Build Server

Builds / Status Notifications

Artefact Repository

Dashboard QA Cloud

Version Control

PROD APPROVAL Production Deployment Mock / Unit Tests Сode Analysis Code Review

UAT, INT PERF PROD

Security check Performance Test Integration Test Functional & Regression Test

• Fully Automated CI/CD Pipeline
• Built-in quality and Security gates
• Frequent and fast feedback

Feedback

Developer

6

Cloud services - AWS

Fully automated infra as code. Build everything. No manual change in important accounts. Do regular POCs of the services as there are numerous changes and new services coming out monthly. Watch the news!

SEPARATE DEV/QA/UAT/PROD ENVIRONMENTS

• At least in VPC level but best keep them in separate AWS account.
• This way you can use the same code to build each of them and test.
• Have separate account for manual changes, POCs, sandbox environment.

COMPONENTS MIGHT BE BUILT BY DIFFERENT TOOLS

• Terraform or CloudFormation for fundamental layer
• Packer or Ansible for EC2 AMI build

IAM SSO INTEGRATION IS ESSENTIAL FOR BIG COMPANIES

• Separate role for users, services, build tools, TechOps-admin groups
• Maintain Least Privilege Rule could be hard because of frequent changes. Use AWS managed policies whenever

possible.

7

Infrastructure as a code - Terraform

HANDLE INFRASTRUCTURE CODE LIKE AN APPLICATION CODE

• Versioning your infrastructure
• Keep coding standards
• Use code review for knowledge sharing
• Testing, testing and testing.
• Linting, unit and smoke tests (Terratest is a Go library for infrastructure testing)

MAKE THE MODULES REUSABLE (WHETHER ACROSS MULTIPLE PROJECTS) AS MUCH AS POSSIBLE

• Try to define well designed units and make modules for implementing the required functionality
• Well defined input and output parameters
• Separation between the logic and the parameters

PUSH YOUR INFRASTRUCTURE CODE THROUGH EACH OF YOUR ENVIRONMENTS

• Development, Testing, Staging and Production

PREPARE ROLLBACK AND DISASTER RECOVERY PLAN

• Terraform is a great tool, but it has his own limitation
• No automatic rollback capability
• It could easily destroy any of your cloud resources, if you are not aware how it works behind the scenes

8

Provisioning and configuration management - Ansible

• Ansible is like the Swiss Army Knife of DevOps.
• Keep re-usability in mind, separate roles even in code base so it's easier to share for even different projects, teams etc.

But keep it simple.

• Think declaratively. Try to build roles that they can build from zero but even update an existing env using the same code.

(= Idempotence.) Add state to tasks whether possible.

• Check if Ansible Galaxy might already have what you try to achieve. Contribute back your improvement ideas.
• Naming convention is nice to have for variables, roles, role directory structure and playbooks. Always name tasks in roles. Add

comments when your code needs explanation.

• Ansible tags are handy for filtering within role tasks so it's good to have them. Group tasks when they have the same tags or

when clause.

• Use dynamic inventory whenever possible e.g. in cloud environment
• Variable precedence is always important to keep in mind so always use variables in the right level but

try to keep them in role level. Avoid hard-coded values but use variables and templates.

• Secrets should be vaulted. Use at least ansible-vault but central vault is even better.
• Clean up debug messages before commit to version control but log and monitor performance of your

playbooks.

• Try to build roles which can run in multiple OS families.
• Gather facts could slow down playbook runtime.

References:

• https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html

9

Orchestration management - Kubernetes

CONTAINERIZE YOUR OPERATIONS

• "Run to completion" style automations:
• Versioned DB schema upgrades
• Database population with versioned test data
• Regular cleanup/garbage collection to keep your environments healthy

THIRD-PARTY COMPONENTS (OR DEVELOPING YOUR OWN): DO NOT REINVENT THE WHEEL!

• Install via Helm charts (the agentless Helm 3 has just been released!) - Infrastructure as Code concept

Go for shopping curated charts: github.com/helm/charts/tree/master/stable

• Install via Operators and let Kubernetes do maintenance work for you – Infrastructure as Software concept

When tasks like updating, scaling, reconfiguring, continuous supervising are not trivial or hard to cover their automation with built-in functionalities of Kubernetes. You can find a lot here: OperatorHub.io

• Out of the box solutions doesn't suit you?

Try extending functionalities with Sidecar containers instead of altering an upstream

• Always think about Security: What kind of container images are pulled into your clusters with a chart?

You can validate them (Docker Bench, Claire, Anchore, etc.), then host them from own registry (Harbor, JFrog, Nexus) or even sign them (Notary)

10

Containerization - Docker

CI/CD

• Prepare for externalized configuration:
• containerized application
• config override mechanism

SECURE YOUR IMAGES

• Build upon trusted and minimal base images, strictly avoid "pushed" images
• Minimalize your image content to bare-minimal (reduce attack surface)
• Limit user permissions inside containers:
• run your processes as non-root users
• advanced: use uid/gid mapping (docker user namespaces)

OPERATIONS/DEBUGGING

• Provide healthcheck for your containerized services to be able to detect readiness and liveness
• Containerized software should print logs to STDIN/STDERR
• Docker logs are ephemeral by default, collect them centrally (ELK/EFK, AWS Cloudwatch, etc)
• Attach specially prepared debug container and get a shell just like you have entered to target container with exec but with

an extended toolkit

CONTAINERIZED TESTING

• You can create reproducible and quick test environments even for UI testing (Selenium)

11

Monitoring - Grafana and Prometheus

HOW TO DO MONITORING IN A FAST-CHANGING ENVIRONMENT REQUIREMENTS:

• Automation
• Scalability
• On-demand

DISCOVERY:

• Design a flexible tagging schema
• Service type, Environment, Custom settings

VISUALIZATION:

• Compose a dashboard as needed
• Use variables to make the dashboard

more dynamic

12

DevTestSecOps Transformation Requires a Holistic Approach

• Coding Standards
• Code Review
• Static Code Analysis
• Code Quality Gates
• Version Control, Branching
• Unit Testing
• Agile Practices
• Engineering Culture
• CI/CD
• DevTestOps
• Environment Provisioning
• Automated Pipeline with Quality

Gates

• Micro services
• Cloud Native Development
• Application Transformation
• Refactoring

DevTestSecOps Transformation

• Test Pyramid
• Test Design Practices
• Living Documentation
• Test Data Management Services
• Service Virtualization
• Shift Left Non-functional Testing
• Shift Left Integration Testing
• Production Testing and Monitoring

13

How We Are Achieving Our Testing Vision

Clear vision of the target state and production-proven experience Automation of various test types within the iteration

TESTING PYRAMID CONTINUOUS TESTING SHIFT LEFT IN-SPRINT TEST AUTOMATION TOOLS & ACCELERATORS METRIC-BASED CONTROL

Holistic delivery transformation incl. infrastructure and architecture EPAM custom tools and accelerators covering the full test pyramid scope (Automated) quality gates to drive quality in earlier development phases Continuous automated measurment and improvement of QA metrics

14

Continuous Testing Blueprint

CI / Build Server

Builds / Status Notifications

Artefact Repository

CI Dashboard QA Cloud

Version Control

PROD APPROVAL Production Deployment Mock / Unit Tests Сode Analysis Code Review

UAT, INT PERF PROD

Security check Performance Test Integration Test Functional & Regression Test

• Fully Automated CI/CD Pipeline
• Built-in quality and Security gates
• Frequent and fast feedback

Feedback

Developer

15

TAMAS_CSAKO@EPAM.COM WWW.LINKEDIN.COM/IN/TAMAS-CSAKO