Human-friendly DNS powered by Golang We love to work in a casual - - PowerPoint PPT Presentation

Human-friendly DNS powered by Golang

We love to work in a casual environment that energizes us to revolutionize the fitness industry.

Site Reliability Engineering @ eGym

• Scale

Site Reliability Engineering @ eGym

Site Reliability Engineering @ eGym

Site Reliability Engineering @ eGym

Room for more SREs :)

Site Reliability Engineering @ eGym

• Scale
• Automate
• Own reliability

○ 24/7 Monitoring ○ On-call

• Consult teams

DNS Basics

DNS Basics

Names

www egym de $NULL . . .

separator label

DNS Hosts

www egym de . .

domain host Top-level domain

DNS Name Spacing

int egym de . .

zone Sub zone

DNS Name Spacing

https://en.wikipedia.org/wiki/Domain_Name_System

DNS @ eGym

• 20+ domains

○ and counting...

• 18 delegations to sub zones

○ “team spaces”

• 700+ resource records

○ without sub zone records Registrar Hoster Cloud DNS Delegation (NS RR) Zone File Deployment Change Scripts Scripts Scripts Read/Parse

Rethinking DNS @ eGym

Single Source Of Truth Production DNS (Cloud DNS) Tools Single Source of Truth

• Version Controlled
• Human Readable
• Non-repetitive

Cloud DNS

• Scales
• API

Tools

• Rollbacks
• Replays
• Automated (non-interactive mode)
• Safeguards

Rethinking DNS @ eGym

Single Source Of Truth Production DNS (Cloud DNS) Tools Single Source of Truth

• Version Controlled
• Human Readable
• Non-repetitive

Cloud DNS

• Scales
• API

Tools

• Rollbacks
• Replays
• Automated (non-interactive mode)
• Safeguards

YAML Templates

DNS Tooling

• PaPuDNS

○ Parses YAML-formatted zone information ○ Parses YAML-formatted templates ○ Applies templates ○ In-Memory database with all resource records ○ Fetches current zone information from Cloud DNS via API ○ Calculates difference ○ Pushes the changes (atomically)

• dns-check

○ Parses YAML-formatted “expectations” ○ Checks via live DNS if expectations are meet ○ Does not (yet) use the same format/database

Zones

github.com/egymgmbh/papudns zones:

• zone: egym.coffee

description: Test zone. ttl: 300 templates:

• gmail
• website

names:

• name: '@'

texts: data:

• foobar-site-verification-123456
• name: paloalto

forwarding: ttl: 60 target: flaky.cloud.example.com.

• name: losangeles

addresses: literals:

• 192.0.2.99
• 2001:db8:200::99
• Sets zone TTL
• Pulls in “names” from templates
• TXT resource record
• CNAME resource record

(custom TTL)

• A and AAAA resource records

Templates

templates:

• template: gmail

description: > This template adds Google mailservers to a zone. names:

• name: '@'

mail: ttl: 604800 # 1 week = 604800 seconds mailservers:

• mailserver: aspmx.l.google.com.

priority: 10

• mailserver: alt1.aspmx.l.google.com.

priority: 20

• name: google._domainkey

texts: data:

• >

v=DKIM1; k=rsa; p=foobar123456

www egym de $NULL . . .

github.com/egymgmbh/papudns

Go and YAML

• Package yaml.v2

○ gopkg.in/yaml.v2

• func Unmarshal(in []byte, out interface{}) (err error)

○ Byte stream to custom struct type ○ Struct fields are only unmarshalled if they are exported (have an upper case first letter), and are unmarshalled using the field name lowercased as the default key.

• See demo!

Go and DNS

• Pure Go resolver

○

export GODEBUG=netdns=go # force pure Go resolver

• Cgo resolver

○

export GODEBUG=netdns=cgo # force cgo resolver

• Raw DNS queries

○ github.com/miekg/dns/

By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread. https://golang.org/pkg/net/#Resolver

On the wire: Pure Go resolver vs. Cgo resolver

16:33:52.097709 IP (tos 0x0, ttl 64, id 53695, offset 0, flags [DF], proto UDP (17), length 88) force.59722 > google-public-dns-a.google.com.domain : [bad udp cksum 0x7757 -> 0x35c8!] 18325+ [1au] AAAA? danrl.com. ar: . OPT UDPsize=4096 DO (60) 16:36:26.279509 IP (tos 0x0, ttl 64, id 15282, offset 0, flags [DF], proto UDP (17), length 88) force.55245 > google-public-dns-a.google.com.domain : [bad udp cksum 0x7757 -> 0x6724!] 10166+ [1au] AAAA? danrl.com. ar: . OPT UDPsize=4096 DO (60)

Resolvers

• See demo!

CNAME

www.egym.de. dualstack.egym-server-1779992439.eu-west-1.elb.amazonaws.com. 2a01:578:3::36d9:cf8b 2a01:578:3::2e89:6e8a 2a01:578:3::36f7:bfb5 CNAME AAAA

Putting it all together

• DNS check

○ Install dns-check ○ Define expectations ○ Reality check

• PaPuDNS

○ Install PaPuDNS ○ Define zone information ○ Deploy!

Summary

• We started building tools in Go

○ SRE ❤฀ Go

• We intentionally NOT use goroutines for critical deployments

○ We want humans (slow) to be able to veto the tool’s actions (fast)

• TODO: Use goroutines/channels for dns-check
• TODO: Use single source of truth for monitoring, too

○ And see if that works well (debatable)

• career.egym.com (we are growing!)
• code.egym.de (dev blog)
• github.com/egymgmbh/papudns (deploy tool)
• github.com/egymgmbh/dns-tools (monitoring tool)
• @danrl_com (twitter)
• danrl.com (my homepage)
• github.com/danrl/playground-2017-08-gopher-meetup

(today’s code) What time is it? Time to socialize!