Introduction digitalocean.com What does DO do? Simple, - - PowerPoint PPT Presentation

digitalocean.com

Introduction

digitalocean.com

What does DO do?

Simple, Developer-focused Cloud Hosting

digitalocean.com

What are we using Ansible for?

digitalocean.com

Example Deployment

digitalocean.com

Example Project Layout

digitalocean.com

Project Layout

• Inventories
• Local Module Library
• Group Variables / Host Variables
• Roles

○ Component roles ○ Project Specific Roles

• Playbooks

○ Server Templates ○ Cluster Configuration ○ Actions

• Makefiles

digitalocean.com

Inventories

• List hosts (by environment)
• Define groups
• Guardrails

ansible-playbook -i inventories/development ...

digitalocean.com

Inventories

• List hosts (by environment)
• Define groups
• Guardrails

all: children: mysql: children: mysql_managed: hosts: test-mysql-0[1:3].atlantic.com: test-mysql-0[1:6].pacific.com: mysql_unmanaged:

digitalocean.com

Inventories

• List hosts (by environment)
• Define groups
• Guardrails

ansible-playbook ... --extra-vars="target_env=development” ... Playbook:

• hosts: mysql:!mysql_unmanaged:&{{ target_env }}

...

digitalocean.com

Inventories: Constructed Groups

plugin: constructed strict: false groups: dev: inventory_hostname.startswith('dev-')

digitalocean.com

Inventories: Constructed Groups

plugin: constructed strict: false groups: dev_mysql: (group_names|intersect(['mysql', 'dev']))|length >= 2

digitalocean.com

Inventories: Ordering

inventories

• development
• 10_mysql.yml
• 90_environment.yml
• 99_dev_mysql.yml
• production
• staging

digitalocean.com

Variable Order of Precedence

1. command values (eg “-u user”) 2. role defaults 3. inventory file or script group vars 4. inventory group_vars/all 5. playbook group_vars/all 6. inventory group_vars/* 7. playbook group_vars/* 8. inventory file or script host vars 9. inventory host_vars/* 10. playbook host_vars/* 11. host facts / cached set_facts 12. play vars 13. play vars_prompt 14. play vars_files 15. role vars (defined in role/vars/main.yml) 16. block vars (only for tasks in block) 17. task vars (only for the task) 18. include_vars 19. set_facts / registered vars 20. role (and include_role) params 21. include params 22. extra vars (always win precedence)

digitalocean.com

Variable Management

• Role defaults interface with the role
• Define project level generic variables applicable to all environments

○ playbook group_vars/all ○ playbook group_vars/*

• Host specific overrides

○ inventory host_vars/*

• Variables we construct

○ role vars / include_vars / set_facts

• Functional role variables

○ role (and include_role) params

• Guardrails

○ extra vars (always win precedence)

digitalocean.com

Variable Management Example - Defaults

• ### proxysql install

proxysql_create_image: "{{ global_create_image | default(false) }}" proxysql_download_src: https://github.com/sysown/proxysql/releases/download proxysql_version: 1.4.10 proxysql_mysql_client_version: 5.7 proxysql_user: proxysql proxysql_group: proxysql proxysql_datadir: /var/lib/proxysql proxysql_restart_missing_heartbeats: 10 ... # autocommit proxysql_mysql_autocommit_false_is_transaction: false proxysql_mysql_autocommit_false_not_reusable: false proxysql_mysql_enforce_autocommit_on_reads: false proxysql_mysql_forward_autocommit: false ...

digitalocean.com

Variable Management Example - Vars

• ...

### percona required packages proxysql_release: "{{ proxysql_download_src }}/v{{ proxysql_version }}/proxysql_{{ proxysql_version }}-ubuntu18_amd64.deb" ... proxysql_mysql_variables: autocommit_false_is_transaction: variable: "autocommit_false_is_transaction" variable_value: "{{ proxysql_mysql_autocommit_false_is_transaction | to_json }}" autocommit_false_not_reusable: variable: "autocommit_false_not_reusable" variable_value: "{{ proxysql_mysql_autocommit_false_not_reusable | to_json }}" client_found_rows: variable: "client_found_rows" variable_value: "{{ proxysql_mysql_client_found_rows | to_json }}" ...

digitalocean.com

Variable Management Example - Config

#jinja2: lstrip_blocks: "true" datadir="{{ proxysql_datadir }}" restart_on_missing_heartbeats={{ proxysql_restart_missing_heartbeats }} admin_variables= { {% for config_item in proxysql_admin_variables|dictsort %} {% if config_item.1.variable_value is not none %} {{ config_item.1.variable }}={{ config_item.1.variable_value | to_json }} {% endif %} {% endfor %} } mysql_variables= { {% for config_item in proxysql_mysql_variables|dictsort %} {% if config_item.1.variable_value is not none %} {{ config_item.1.variable }}={{ config_item.1.variable_value | to_json }} {% endif %} {% endfor %} }

digitalocean.com

Anatomy of a Role

digitalocean.com

Anatomy of a Role

• A role should be map to a single unit of functionality that utilise a common set of variables.
• Roles should be intuitive, and wherever possible mimic a common structure.
• Role Variable Management

○ Where possible, a [component] role should be generic, and any variables should map to sensible defaults. ○ The interface into role customisation should be via scalar role defaults. ○ Role variables should be used for variables that shouldn't be overridden in normal circumstance, or as syntactic sugar to construct variables internal to the role.

• A role should have repeatable logic and should avoid logical branching that might be

non-repeatable.

digitalocean.com

Component Roles

digitalocean.com

Role Versioning

• name: role_mysql_proxysql

src: git+ssh://git@github.pacific.com/ansible/role_mysql_proxysql.git version: 1.1.1

digitalocean.com

Example ProxySQL Deployment

digitalocean.com

Testing Roles

digitalocean.com

• pip install --user molecule

○ pip install --user molecule[ec2] ○ pip install --user molecule[docker]

Molecule

digitalocean.com

• create / destroy / list / cleanup
• prepare
• dependency
• login

Molecule Commands

digitalocean.com

Anatomy of a Role

digitalocean.com

• lint
• syntax
• idempotence
• verify
• check

Molecule Commands

digitalocean.com

• converge
• test
• side-effects

Molecule Commands

digitalocean.com

Role Development with Molecule

digitalocean.com

Testing ProxySQL Example

digitalocean.com

Molecule Configuration

dependency: name: galaxy driver: name: docker lint: name: yamllint platforms:

• name: host1

image: "geerlingguy/docker-${MOLECULE_DISTRO:-ubuntu1804}-ansible:latest" command: ${MOLECULE_DOCKER_COMMAND:-""} volumes:

• /sys/fs/cgroup:/sys/fs/cgroup:ro

privileged: true pre_build_image: true provisioner: name: ansible lint: name: ansible-lint

digitalocean.com

Molecule Configuration

scenario: name: default converge_sequence: # - dependency

• create

# - prepare

• converge

test_sequence:

• lint
• destroy

# - dependency

• syntax
• create

# - prepare

• converge
• idempotence

# - side_effect

• verify
• destroy

digitalocean.com

Molecule Configuration

verifier: name: testinfra env: PYTHONWARNINGS: "ignore:.*U.*mode is deprecated:DeprecationWarning"

• ptions:

v: 1 lint: name: flake8

digitalocean.com

Functional Testing with TestInfra

proxysql_file_attributes = ("proxysql_file," "proxysql_file_user," "proxysql_file_group," "proxysql_file_mode") @pytest.mark.parametrize(proxysql_file_attributes, [ ("/root/.my.cnf", None, None, 0o600), ("/etc/proxysql.cnf", "proxysql", "proxysql", 0o644), ]) def test_proxysql_files(host, proxysql_file, proxysql_file_user, proxysql_file_group, proxysql_file_mode): f = host.file(proxysql_file) assert f.exists assert f.is_file if proxysql_file_user: assert f.user == proxysql_file_user if proxysql_file_group: assert f.group == proxysql_file_group if proxysql_file_mode: assert f.mode == proxysql_file_mode

digitalocean.com

Functional Testing with TestInfra

• Host fixture

○ host.file ○ host.package ○ host.service ○ host.run

digitalocean.com

Continuous Integration Pipeline

digitalocean.com

User Management

digitalocean.com

• Deployment and maintenance of individual and Service users
• Maintain user (dynamic) privileges
• Manage secrets (securely)
• Manage across both MySQL and ProxySQL

User Management Story

digitalocean.com

Manual Worst Case Scenario

sammy sammy@10.21.% sammy@10.22.% sammy@10.23.% sammy@10.24.% sammy@10.25.%

digitalocean.com

User Management Requirements

• Deploy user control manifest to Ansible Project Role
• Generate & Encrypt secrets in Ansible Vault
• Consistent Delivery across technologies / tenancies / environments
• Scalable solution

digitalocean.com

Request > New UMC > Gen Secret > commit/PR > Peer Review > Dry Run > Deploy

New User Deploy Chain

+ >> >> >>

digitalocean.com

User Control Manifest

sammy_ro: state: present active: true default_shard: atlantic default_hostgroup: swimming_pool default_schema: baby_shark enabled_schemas:

• baby_shark
• left_shark

cluster_privs: atlantic: env: dev:

• '%’

privs:

• 'baby_shark.*:SELECT'

pacific: env: dev:

• '%’

privs:

• 'left_shark.*:SELECT,INSERT,UPDATE,DELETE'

digitalocean.com

docs.ansible.com/ansible/latest/modules/list_of_database_modules.html?#proxysql proxysql_mysql_users

proxysql_query_rules proxysql_replication_hostgroups proxysql_scheduler

ProxySQL Ansible Module

proxysql_backend_servers proxysql_global_variables proxysql_manage_config

digitalocean.com

Secret Storage

digitalocean.com

Secret storage

Ansible Vault

• Used for shared secrets
• Fine-grained access control
• Simple, no dependencies
• Stored in repo w/ inventory &

playbooks

digitalocean.com

Ansible Vault

Vault file per environment (dev/stage/production)

• monitoring_password: ooxalohquaiK8aidai0x

backup_password: ahvowooG9dohfashaCho ansible-vault encrypt $ANSIBLE_VAULT;1.2;AES256;production 3538653766323031333335306239343730333830 6536303737313838633736386164616334383631 3566613638356633666138326338356637636135 3637636231623034380a32633831636239353836 6663346365666465353266633865653963343063 6266656230626232316662386639343233656439

digitalocean.com

Ansible Vault

sammy_ro: dev: '*100569F51F55F3599CECBCABB9AC59AB29F30283' stage2: '*712E505DB01097F24D1B72509A820E254A525528' production: '*1A0174A0F2F4692917994C19C158F96B1914A53F' sammy_rw: dev: '*0A5B11132F429A38988540A351C60A620DE5BBFA' stage2: '*8B5CCFF05B945DED591D603DF936805476E77837' production: '*C0BF0057A4AC079E7ABE2FEC54DC87A1680B95DC'

Two vault files for all password hashes

• Passwords distributed to users via LastPass
• Ansible never needs password, just hash

digitalocean.com

Ansible Vault

Challenges of ansible-vault approach:

• Merge conflicts
• Visibility / discoverability

digitalocean.com

diffing Ansible vaults $ git diff individuals.yml $ANSIBLE_VAULT;1.1;AES256

• 6530653733663966316630613666393
• 6139303463613133633262643034633
• 3235346162366237366538653361653

... +3938303335363632343730656164356 +3433373263346136653730313636336 +3862633932353039616334303631376

digitalocean.com

diffing Ansible vaults

.gitconfig: [diff "ansible-vault"] textconv = ansible-vault view cachetextconv = false .gitattributes: individuals.yml diff=ansible-vault

digitalocean.com

diffing Ansible vaults $ git diff individuals.yml sammy_rw: dev: '*0A5B11132F429A3898854' stage2: '*8B5CCFF05B945DED59837'

• production: '*C0BF0057A4AC0780B95DC'

+ production: '*B1E8174F9DCE654C25608'

digitalocean.com

Similar config for merge

/usr/local/bin/ansible-vault-merge .gitconfig: [merge "ansible-vault"] name = ansible-vault merge driver driver = /usr/local/bin/ansible-vault-merge -- %O %A %B %P .gitattributes: individuals.yml diff=ansible-vault merge=ansible-vault

See github.com/building5/ansible-vault-tools

digitalocean.com

Finding variables in vault

Where is backup_password defined?

• Variable prefix

○ In vars.yml: backup_password: "{{ vault_backup_password }}" ○ In vault.yml: vault_backup_password: ahvowooG9dohfashaCho

• git grep:

$ git grep --textconv backup_password vars.yml:backup_password: "{{ vault_backup_password }}" vault.yml:vault_backup_password: ahvowooG9dohfashaCho

digitalocean.com

Ansible vault passwords

• Store in local file

○ Exclude from git! ○ Can be executable

• Can define multiple vault-ids:

[defaults] vault_identity_list = production@vault-keyring-client.py,stage@vault-keyring-client.py

digitalocean.com

HashiCorp Vault For secrets shared with others

digitalocean.com

HashiCorp Vault

# Set VAULT_ADDR $ vault login # Set VAULT_TOKEN $ vault write secret/sammy pass=Za6Uoy $ vault read secret/sammy Key Value

• -- -----

pass Za6Uoy

digitalocean.com

HashiCorp Vault

• hashi_vault lookup plugin

○ https://docs.ansible.com/ansible/latest/plugins/lookup/hashi _vault.html

• Various auth methods

○ We use approle and token

• Auth secret stored in environment variable or in

Ansible vault

digitalocean.com

HashiCorp Vault

• copy:

content: >- {{ lookup('hashi_vault', 'secret=secret/sammy_cert:cert url=https://vault.example.com:8200 auth_method=approle role_id={{ role_id }} secret_id={{ secret_id }} ) }} dest: /etc/nginx/ssl/certificate.crt

digitalocean.com

HashiCorp Vault

Why not HashiCorp Vault for everything?

• External dependency
• Changing policies requires another commit and

approval by another team

digitalocean.com

Ansible Performance

digitalocean.com

Performance Visibility

ansible.cfg callback_whitelist = profile_tasks, profile_roles, timer stdout_callback = actionable

digitalocean.com

Performance

• SSH optimisations
• Gathering Facts
• Play Logic

digitalocean.com

Performance

• SSH optimisations
• Gathering Facts
• Play Logic

digitalocean.com

A mitogen is a chemical substance, usually a protein, that induces a cell to begin cell division

Mitogen for Ansible

An Ansible Plugin that replaces the use of SSH

digitalocean.com

Mitogen for Ansible

• Redesigned UNIX connection layer and module runtime for Ansible
• Easy installation, minimal configuration
• Immediate gains (immediately) 1% - 7% improvement in run duration

*unless you have and it didn’t help (Windows, Networking equipment)

digitalocean.com

Performance Challenges

• Loopy Loops

digitalocean.com

Module Override

digitalocean.com

With pushdown Without mitogen

digitalocean.com

With pushdown With mitogen

digitalocean.com

Without pushdown Without mitogen

digitalocean.com

Without pushdown With mitogen

digitalocean.com

Mitogen SSH Only Standard mysql_user 4m 23s 10m 6s Patched mysql_user 7m 33s 8m 19s

We’re Hiring!

https://www.digitalocean.com/careers