PRACTICAL MTLS Ying Li @cyli PROBLEM TYPICAL MICROSERVICE - - PowerPoint PPT Presentation

▶
practical mtls
SMART_READER_LITE
LIVE PREVIEW

PRACTICAL MTLS Ying Li @cyli PROBLEM TYPICAL MICROSERVICE - - PowerPoint PPT Presentation

Mar 09, 2023 113 likes •740 views

MINIMIZING THE WINDOW OF COMPROMISE PRACTICAL MTLS Ying Li @cyli PROBLEM TYPICAL MICROSERVICE ARCHITECTURE VPC S1 DB S2 S1 S3 PROBLEM VLAN-TASTIC MICROSERVICE ARCHITECTURE S1 DB S2 S1 S3 PROBLEM CORRECT MICROSERVICE ARCHITECTURE

slide-1
SLIDE 1

PRACTICAL MTLS

MINIMIZING THE WINDOW OF COMPROMISE

Ying Li @cyli

slide-2
SLIDE 2

TYPICAL MICROSERVICE ARCHITECTURE

PROBLEM

S1 S1 S2 S3 DB

VPC

slide-3
SLIDE 3

VLAN-TASTIC MICROSERVICE ARCHITECTURE

PROBLEM

S1 S1 S2 S3 DB

slide-4
SLIDE 4

CORRECT MICROSERVICE ARCHITECTURE

PROBLEM

S1 S1 S2 S3 DB

slide-5
SLIDE 5

APPLICATION TLS LIFECYCLE

Bootstrap Revoke Renew

PROBLEM

slide-6
SLIDE 6
  • CSR ➡ CA
  • Configuration

BOOTSTRAP

PROBLEM

slide-7
SLIDE 7
  • Schedule

RENEW

PROBLEM

slide-8
SLIDE 8
  • Schedule
  • CSR ➡ CA
  • Configuration

RENEW

PROBLEM

slide-9
SLIDE 9
  • Schedule
  • CSR ➡ CA
  • Configuration
  • Restart

PROBLEM

RENEW

slide-10
SLIDE 10
  • CRL
  • OCSP [Stapling]

REVOKE

PROBLEM

slide-11
SLIDE 11

AUTOMATE, AUTOMATE, AUTOMATE

  • Promotes adoption of mTLS

PRINCIPLE

slide-12
SLIDE 12

AUTOMATE, AUTOMATE, AUTOMATE

  • Promotes adoption of mTLS
  • Single location for private key

PRINCIPLE

slide-13
SLIDE 13

AUTOMATE, AUTOMATE, AUTOMATE

  • Promotes adoption of mTLS
  • Single location for private key
  • Shorter certificate expiry

PRINCIPLE

slide-14
SLIDE 14

SWARMKIT OVERVIEW

https://github.com/docker/swarmkit

slide-15
SLIDE 15

SWARMKIT OVERVIEW

Worker Manager Manager Manager Worker Worker Worker Worker Worker

CLUSTER

slide-16
SLIDE 16

SWARMKIT OVERVIEW

Worker Manager Manager Manager Worker Worker Worker Worker Worker

CLUSTER

Node Node Node Node Node Node

slide-17
SLIDE 17

SWARMKIT OVERVIEW

Worker Manager Manager Manager Worker Worker Worker Worker Worker raft store

CLUSTER

slide-18
SLIDE 18

SWARMKIT OVERVIEW

Node CA CA CA Node Node Node Node Node raft store

CLUSTER

slide-19
SLIDE 19

SWARMKIT’S IMPLEMENTATION

slide-20
SLIDE 20

BOOTSTRAP

SWMTKN-1-mx8suomaom825bet6-cm6zts22rl4hly2 Known Prefix Token Version Hash

  • f Root CA

Random Secret

SWARMKIT’S IMPLEMENTATION

slide-21
SLIDE 21
  • 1. Retrieve, validate Root CA

certificate.

BOOTSTRAP

1

CA

Node

SWARMKIT’S IMPLEMENTATION

slide-22
SLIDE 22
  • 1. Retrieve, validate Root CA

certificate.

  • 2. CSR + secret token ➡ CA. (TLS)

BOOTSTRAP

1 2

CA

Node

SWARMKIT’S IMPLEMENTATION

slide-23
SLIDE 23
  • 1. Retrieve and validate Root CA

Public key material.

  • 2. CSR + secret token ➡ CA. (TLS)
  • 3. Get certificate. (TLS)

BOOTSTRAP

1 2 3

CA

Node

SWARMKIT’S IMPLEMENTATION

slide-24
SLIDE 24

RENEW

Valid From Valid Until 50% 80%

SWARMKIT’S IMPLEMENTATION

slide-25
SLIDE 25
  • 1. CSR + ➡ CA. (mTLS)

2.Get certificate. (mTLS)

RENEW

1 2

CA

Node

SWARMKIT’S IMPLEMENTATION

slide-26
SLIDE 26

RENEW

Restart

  • 1. Trigger extra leader election
  • 2. Workers all need to reconnect

to managers

  • 3. Reschedule work

SWARMKIT’S IMPLEMENTATION

slide-27
SLIDE 27

RENEW

SWARMKIT’S IMPLEMENTATION

slide-28
SLIDE 28

RENEW

SWARMKIT’S IMPLEMENTATION

slide-29
SLIDE 29

RENEW

Server

Existing connections New connections

SWARMKIT’S IMPLEMENTATION

slide-30
SLIDE 30

RENEW

Client

Existing connections New connections

SWARMKIT’S IMPLEMENTATION

slide-31
SLIDE 31

REVOKE

SWARMKIT’S IMPLEMENTATION

slide-32
SLIDE 32

REVOKE

SWARMKIT’S IMPLEMENTATION

REMOVE

CRLS, OCSP [Stapling]

slide-33
SLIDE 33

REMOVE

SWARMKIT’S IMPLEMENTATION

NODE BLACKLIST

Node ID Certificate Expiry

a8h1vsk3k9o5nwea858ty9kma 2017-08-26 01:02:52 UTC k80l2au3yq9f7x6r2oca13vwt 2017-07-15 11:35:23 UTC n970d5be9ccgnreg4iti4jho3 2017-08-01 22:59:05 UTC

slide-34
SLIDE 34

REMOVE

SWARMKIT’S IMPLEMENTATION

Worker/Manager Manager

Request Validate node ID against blacklist Authorize role Perform work Response

Worker/Manager Manager

slide-35
SLIDE 35

REMOVE

SWARMKIT’S IMPLEMENTATION

BLACKLIST VS WHITELIST

slide-36
SLIDE 36

REMOVE

SWARMKIT’S IMPLEMENTATION

Manager Manager Manager delayed join

slide-37
SLIDE 37

REMOVE

SWARMKIT’S IMPLEMENTATION

Manager Manager Manager

slide-38
SLIDE 38

Rotate CA

PROBLEM

slide-39
SLIDE 39

CA ROTATION

PROBLEM

  • (conf.) All nodes: trust old and new CA
  • (wait.) Verify all nodes

1

slide-40
SLIDE 40

CA ROTATION

PROBLEM

  • (conf.) All nodes: trust old and new CA
  • (wait.) Verify all nodes
  • (conf.) All nodes: renew certificates
  • (wait.) Verify all nodes

1 2

slide-41
SLIDE 41

CA ROTATION

PROBLEM

  • (conf.) All nodes: trust old and new CA
  • (wait.) Verify all nodes
  • (conf.) All nodes: renew certificates
  • (wait.) Verify all nodes
  • (conf.) All nodes: trust new CA only
  • (wait.) Verify all nodes

1 2 3

slide-42
SLIDE 42

CROSS-SIGNED INTERMEDIATE

Root A

Key Info: A Signed by: A

Root

B Key Info: B Signed by: B

Root

B X Leaf cert: X Signed by: B Root: B

PRINCIPLE

slide-43
SLIDE 43

CROSS-SIGNED INTERMEDIATE

Root A

Key Info: A Signed by: A

Root

B Key Info: B Signed by: B

Root

A X Leaf cert: X Signed by: B Root: A

PRINCIPLE

slide-44
SLIDE 44

CROSS-SIGNED INTERMEDIATE

Root A

Key Info: A DN: A Signed by: A

Root

B Key Info: B DN: B Signed by: B Key Info: B DN: B Signed by: A

Root A Root A

Intermediate

B’

PRINCIPLE

slide-45
SLIDE 45

Leaf cert: X Signed by: B’ Root: A

Root A Root A

Intermediate

B’ X

CROSS-SIGNED INTERMEDIATE

PRINCIPLE

slide-46
SLIDE 46

Root A Root A

Intermediate

B’

Root B

Leaf cert: X Signed by: B Root: B X

CROSS-SIGNED INTERMEDIATE

PRINCIPLE

slide-47
SLIDE 47

CA ROTATION

SWARMKIT’S IMPLEMENTATION

  • (conf.) All nodes: trust old and new CA
  • (wait.) Verify all nodes
  • Generate cross-signed intermediate
slide-48
SLIDE 48

CA ROTATION

SWARMKIT’S IMPLEMENTATION

  • Generate cross-signed intermediate
  • (conf.) All nodes: renew certificates
  • (wait.) Verify all nodes

1

slide-49
SLIDE 49

CA ROTATION

SWARMKIT’S IMPLEMENTATION

  • Generate cross-signed intermediate
  • (conf.) All nodes: renew certificates
  • (wait.) Verify all nodes
  • (conf.) All nodes: trust new CA
  • (wait.) Verify all nodes
  • Throw away cross-signed intermediate

1 2

slide-50
SLIDE 50

CA ROTATION: BEFORE ROTATION

SWARMKIT’S IMPLEMENTATION

Node Trust Root: Node TLS Certificate: Cluster Trust Root: Cluster Cert Issuer:

Root A Root A Root A Root A Root A

Z

slide-51
SLIDE 51

CA ROTATION: START ROTATION

SWARMKIT’S IMPLEMENTATION

Node Trust Root: Node TLS Certificate:

Root A Root A Root A

Intermediate

B’ Cluster Trust Root: Cluster Cert Issuer:

Root A Root A Root A

Z

slide-52
SLIDE 52

CA ROTATION: NODE CERT RENEWAL

SWARMKIT’S IMPLEMENTATION

Node Trust Root: Node TLS Certificate:

Root A Root A Root A

Intermediate

B’ Cluster Trust Root: Cluster Cert Issuer:

Root A

Root

A

Root A

Intermediate

B X

slide-53
SLIDE 53

CA ROTATION: NODE CERT RENEWAL

SWARMKIT’S IMPLEMENTATION Node1 Node2 Node3 Node4 Node5 Trust Root TLS Certificate

Root A

Root A Z Root

A

Ro A

Intermediate

B X

Root

A

Ro A

Intermediate

B X

Root

A

Ro A

Intermediate

B X

Root A Z

Root A Root A Root A Root A

slide-54
SLIDE 54

CA ROTATION: ROTATE TRUST ROOT

SWARMKIT’S IMPLEMENTATION

Node Trust Root: Node TLS Certificate:

Root B

Cluster Trust Root: Cluster Cert Issuer:

Root B

Root A

Root A

Intermediate

B

Root

B X

Root A Root A

Intermediate

B’

slide-55
SLIDE 55

CA ROTATION: ROTATE TRUST ROOT

SWARMKIT’S IMPLEMENTATION Node1 Node2 Node3 Node4 Node5 Trust Root TLS Certificate

Root B Root B Root B

Root

A

Ro A

Intermediate

B X

Root

A

Ro A

Intermediate

B X

Root

A

Ro A

Intermediate

B X

Root

A

Ro A

Intermediate

B X

Root

A

Ro A

Intermediate

B X

Root A Root A

slide-56
SLIDE 56

CA ROTATION: FINISH ROOT ROTATION

SWARMKIT’S IMPLEMENTATION

Node Trust Root: Node TLS Certificate:

Root B

Cluster Trust Root: Cluster Cert Issuer:

Root B Root B

Root A

Root A

Intermediate

B

Root

B X

slide-57
SLIDE 57

DEMO

slide-58
SLIDE 58

SUMMARY

MINIMIZING THE WINDOW OF COMPROMISE

slide-59
SLIDE 59

SUMMARY

MINIMIZING THE WINDOW OF COMPROMISE

  • automatic bootstrap, renewal
  • short certificate expiry
slide-60
SLIDE 60

SUMMARY

MINIMIZING THE WINDOW OF COMPROMISE

  • automatic bootstrap, renewal
  • short certificate expiry
  • certificate revocation
  • CA rotation
slide-61
SLIDE 61

SUMMARY

MORE INFORMATION

https://github.com/docker/swarmkit

https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/

https://github.com/cloudflare/cfssl

(@cyli)