Layers Yelps Mission Connecting people with great local - - PowerPoint PPT Presentation

Layers

Yelp’s Mission

Connecting people with great local businesses.

(Abstraction) Layers

(related: separation of concerns)

Abstractions?

"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." - Edsger Dijkstra All non-trivial abstractions, to some degree, are leaky. - Joel Spolsky Any problem in computer science can be solved with another layer of indirection. But that usually will create another problem - David Wheeler

Danger: I don’t know as much as Rich Hickey

Identify the Author’s Purpose

A. Persuade the audience that his/her design recommendations are better than the current status quo B. Inform others about the history of architecture decisions in computer science C. Inspire engineers to think outside the box when designing systems D. Entertain the audience with historic anecdotes E. None of the above

DevOps

DevOps

• Est. 2009

Let’s Talk About Devops

Features Wireframes Tasks Code / Unit Tests Build + Release Deployment | Monitoring | Alerting QA | Acceptance Tests PM UX Lead Dev Devs QA Release Engineering Operations

“What if we got rid of some of the layers?”

?

Let’s Talk About Devops (no-ops?)

Features Wireframes Tasks Code / Unit Tests QA | Acceptance Tests Build + Release Deployment | Monitoring | Alerting PM UX Devs

Separation Of Concerns

Why?

• Increased “Performance” (deploys per day)
• Better ownership of issues
• Harder to hire
• Massive increase in developer complexity
• Large inter-team communication overhead

Filesystems

Let’s talk about filesystems

files ext4 dm-crypt Software Raid Raw Block Devices LVM /mnt/stuff/foo /mnt/stuff /dev/mapper/luks-17c8... /dev/mapper/lv0 /dev/md0 /dev/sd{a,b}

“What if we got rid of some of the layers?”

Let’s talk about zfs

files zfs (zraid) (encryption) (checksums) (pools) (vdevs) Raw Block Devices /mnt/stuff/foo /mnt/stuff /dev/sd{a,b}

Separation Of Concerns

Now What?

• Better error detection/recovery
• We can deduplicate things
• More efficient snapshots / Repair
• All new tooling is not interoperable

New Concept: “Sympathetic Abstraction”

• “Opposite” of “Leaky”?
• https://mechanical-sympathy.blogspot.com/

~ $ ssh -vv leb1.xkyle.com -- true OpenSSH_7.7p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017 debug1: Reading configuration data /home/kyle/.ssh/config debug1: /home/kyle/.ssh/config line 1: Applying options for leb1.xkyle.com debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "leb1.xkyle.com" port 4242 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to leb1.xkyle.com [2607:8b00:0:96::d0eb:ec0a] port 4242. ... debug1: Entering interactive session. debug1: pledge: network debug2: channel_input_open_confirmation: channel 0: callback start debug2: fd 3 setting TCP_NODELAY debug2: client_session2_setup: id 0 debug1: Sending environment. debug1: Sending env LANG = en_US.UTF-8 debug2: channel 0: request env confirm 0 debug1: Sending command: true ... Transferred: sent 2696, received 2516 bytes, in 0.1 seconds Bytes per second: sent 23959.5, received 22359.8 debug1: Exit status 0

https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html?hl=commitlog_sync commit_failure_policy (Default: stop) Policy for commit disk failures:

• die
• Shut down gossip and Thrift and kill the JVM, so the node can be replaced.
• stop
• Shut down gossip and Thrift, leaving the node effectively dead, available for inspection using JMX.
• stop_commit
• Shut down the commit log, letting writes collect but continuing to service reads (as in pre-2.0.5 Cassandra).
• ignore
• Ignore fatal errors and let the batches fail.

disk_optimization_strategy (Default: ssd) The strategy for optimizing disk reads. Possible values: ssd or spinning. disk_failure_policy (Default: stop) Sets how Cassandra responds to disk failure. Recommend settings: stop or best_effort. Valid values:

• die
• Shut down gossip and Thrift and kill the JVM for any file system errors or single SSTable errors, so the node can be

replaced.

• stop_paranoid

Kyle’s Abstraction Law:

“Over time, layers tend toward becoming more sympathetic (or merged) in the pursuit of performance.”

HTTP(S)

OSI Layer Model History

• Published in 1984 under ISO7498 (35 years ago)
• Mostly by Charles Bachman from his experience with ARPANET, etc
• Written with the purpose of giving protocol designers a Map to follow

Layer 6: Presentation Layer 5: Session Layer 4: Transport Layer 2: Data link Layer 1: Physical Layer 3: Network SSL N/A (Right? HTTP is “stateless”) TCP (Transmission Control Protocol) IP (Internet Protocol) 802.11 (Wi-Fi?) 802.11 (Wi-Fi?) Layer 7: Application HTTP (Hypertext Transfer Protocol)

HTTPS in the OSI Network Model

https://en.wikipedia.org/wiki/Internet_protocol_suite#Comparison_of_TCP/IP_and_OSI_layering

The IETF protocol development effort is not concerned with strict layering. Some of its protocols may not fit cleanly into the OSI model, although RFCs sometimes refer to it and often use the old OSI layer numbers. The IETF has repeatedly stated that Internet protocol and architecture development is not intended to be OSI-compliant. RFC 3439, addressing Internet architecture, contains a section entitled: "Layering Considered Harmful".

RFC 1122, Internet STD 3 (1989) Cisco Academy[28] Kurose,[29] Forouzan[30] Comer,[31] Kozierok[32] Stallings[33] Arpanet Reference Model (RFC 871) OSI model Four layers Four layers Five layers Four+one layers Five layers Three layers Seven layers "Internet model" "Internet model" "Five-layer Internet model" or "TCP/IP protocol suite" "TCP/IP 5-layer reference model" "TCP/IP model" "Arpanet reference model" OSI model Application Application Application Application Application Application/Process Application Presentation Session Transport Transport Transport Transport Host-to-host or transport Host-to-host Transport Internet Internetwork Network Internet Internet Network Link Network interface Data link Data link (Network interface) Network access Network interface Data link Physical (Hardware) Physical Physical

Layer 6: Presentation Layer 5: Session Layer 4: Transport Layer 2: Data link Layer 1: Physical Layer 3: Network SSL N/A (Right? HTTP is “stateless”) TCP (Transmission Control Protocol) IP (Internet Protocol) 802.11 (Wi-Fi?) 802.11 (Wi-Fi?) Layer 7: Application HTTP (Hypertext Transfer Protocol)

HTTP in the OSI Network Model

“What if we got rid of some of the layers?”

Transport Layer (OSI 4) Link Layer (OSI 1,2) Internet Layer (OSI 3) UDP (User Datagram Protocol) IP (Internet Protocol) 802.11 (Wi-Fi) Application Layer (OSI 5,6,7) Encryption Sessions Multiplexing Error Correction QUIC (Quick UDP Internet Connections)

QUIC in the IP Model

Separation Of Concerns

Now What?

• Roaming sessions
• 0-RRT Handshakes
• Better throughput
• Better loss recovery
• You can’t debug via tcpdump anymore!
• Only a few companies has dared to ship this!
• Very poor tooling / ecosystems around it!
• Must have faith in QUIC-crypto!

(Micro) Services

Rails App Model View Controller Users Mysql

“What if we ADDED some more layers?”

Users API Gateway Users Service Search Tweets Ads Moments Ads Mysql User Mysql Search Infra

Moments SQL

Tweets NoSQL Followers

Graph DB

Timeline

Now What?

• No more atomic cross-cutting changes
• Simplified deploy process? (Simple for who?)
• Polyglot stack (easy to hire for)
• No more unified tooling
• You scale out components (scale to millions of rps)
• You scale out development (1000 engineers)

k8s

Layer 3: Application Code Layer 1: IaaS VM Layer 0: IaaS Provider Bare Metal Layer 2: OS Java Jar Ubuntu x86_64 c5.xlarge N/A

Traditional Cloud Deployment

“What if we ADDED EVEN MORE layers?”

Layer 4: Pod Layer 3: Container Orchestration Layer 1: IaaS VM Layer 0: IaaS Provider Bare Metal Layer 2: OS pod-foo kublet Ubuntu x86_64 c5.xlarge N/A

Layers of a k8s Infra

Layer 5: Container docker Layer 6: Process java -jar

Layer 5: k8s federation Layer 4: k8s cluster Layer 3: Deployment / ReplicaSet Layer 1: Docker Container Layer 0: Process Layer 2: Pod Prod k8s federation uswest1-prod Frontend replicaset myapp-pod FROM apache2 httpd

K8s Layers

“What if we got rid of some of the layers?”

AWS Metal Instances

Layer 0/1: IaaS Provider Bare Metal Hardware

Future Layers

Layer 0/1: IaaS Provider Bare Metal Hardware

Future Layers Extreme Example - Unikernels?

Layer 2-7: Unikernel (No OS, just a directly booting application) MirageOS, etc

Layer 0/1: IaaS Provider Bare Metal Hardware

Future Layers - AWS Lambda / FaaS?

Layer 2-5: Amazon / k8s Stuff Same Stuff As Before Layer 7: Your Process Your Code Layer 6: Container Firecracker / Docker

What? We Are Back To Where We Started?

• Where is the mechanical sympathy?
• What happened to Layer-Smashing, Kyle?
• Three Reasons why we are moving this direction...

Layer 0/1: IaaS Provider Bare Metal Hardware

Reason 1: You have a layer for just the stuff you care about

Layer 2-5: Amazon / k8s Stuff Same Stuff As Before Layer 7: Your Process Your Code Layer 6: Container Firecracker / Docker

k8s pods Apache API Gateway + FaaS Controller (Request Routing) (Compute Dispatch)

Reason 2: Sympathetic routing and dispatch Or “Functions as a Service (Serverless)” Layer Smashing

Autoscaling Group (Compute Dispatch) Apache Elastic Load Balancer (Request Routing) Apache

Reason 3: Sympathetic Compute Scheduling Layers (k8s operators)

Autoscaling Group MySQL MySQL ? K8s MySQL Operator MySQL MySQL

I know what I must do

Conclusions?

• 1. “Sympathetic Layers” are the best layers!
• 2. k8s operators and FaaS as re-slicing where the layers are

(for the greater good!)

• 3. You should watch “Simple Made Easy” before changing

your mind

https://www.infoq.com/presentations/Simple-Made-Easy

Hold for (shameless) applause, Then Questions?

SEPARATION OF CONCERNS IS OVERRATED

Attribution

• OSI Layers - http://www.gargasz.info/osi-model-how-internet-works/
• Simple Made Easy - The Art of Unix Programming (2003) - Eric S. Raymond

X

Linux Kernel GPU X Server

X Client (GUI App) X11 Protocol over Socket /tmp/.X11-unix/X0 Ioctl via libdrm /dev/dri/card0 3d Game Ioctl via libdrm /dev/dri/card0 Window Manager

“Mechanism, not policy”

“What if we got rid of some of the layers?”

Now What?

• Lower latency GUI Interactions
• NO TEARING