BLESS: Better Security and Ops for SSH Access Bryan D. Payne, - - PowerPoint PPT Presentation

▶
bless better security and ops for ssh access
SMART_READER_LITE
LIVE PREVIEW

BLESS: Better Security and Ops for SSH Access Bryan D. Payne, - - PowerPoint PPT Presentation

Jan 14, 2023 357 likes •1.11k views

BLESS: Better Security and Ops for SSH Access Bryan D. Payne, Director of Product Security June 2017 Post by Ryan McGeehan 1 2 3 4 5 Phishing & Lateral Data Zero Day Backdoor Exfiltrate Movement Gathering Attack Several users

slide-1
SLIDE 1

Bryan D. Payne, Director of Product Security June 2017

BLESS: Better Security and Ops for SSH Access

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6

Post by Ryan McGeehan

slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10

1 2 3 4 5

Phishing & Zero Day Attack Backdoor Lateral Movement Data Gathering Exfiltrate Several users are targeted by phishing

  • attacks. At

least one succeeds. Victim machine is accessed remotely by adversary. Attack elevates access and propagates throughout the network. It exploits any privileges and information discovered along the way. Data is collected, prepared, and staged for exfiltration. Encrypted data is exfiltrated, typically to another compromised system that is external to the

  • rganization.
Adapted from https:/ /blogs.rsa.com/anatomy-of-an-attack/.
slide-11
SLIDE 11

What’s the Problem?

slide-12
SLIDE 12
slide-13
SLIDE 13

LDAP

slide-14
SLIDE 14

LDAP

slide-15
SLIDE 15

Operator 2 App A Instances App B Instances App C Instances Operator 3 Operator 1

slide-16
SLIDE 16

Operator 2 Bastion App A Instances App B Instances App C Instances Operator 3 Operator 1

slide-17
SLIDE 17

What about single use SSH keys?

slide-18
SLIDE 18

What if they left great clues behind?

slide-19
SLIDE 19

And offered strong protections?

slide-20
SLIDE 20

Netflix’s Solution

slide-21
SLIDE 21

SSH Authentication

slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26

Bastion’s Lambda Ephemeral Ssh Service

slide-27
SLIDE 27
slide-28
SLIDE 28

def my_handler(event, context): message = 'Hello {} {}!'.format(event['first_name'], event['last_name']) return { 'message' : message } Invoke Lambda

ClientContext

Lambda Response

Status + Payload
slide-29
SLIDE 29

Bastion BLESS Invoke BLESS BLESS Response

Certificate Certificate Request
slide-30
SLIDE 30

Bastion BLESS Invoke BLESS

Certificate Request

BLESS Response

Certificate

AWS KMS

Decrypt SSH CA private key
slide-31
SLIDE 31

Bastion BLESS Invoke BLESS

Certificate Request

BLESS Response

Certificate

AWS KMS

Decrypt SSH CA private key

Instances

SSH with certificate
slide-32
SLIDE 32

SSH Certificates

slide-33
SLIDE 33

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

slide-34
SLIDE 34

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

User or Host Certificates

slide-35
SLIDE 35

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

Control over what is logged by SSHd

slide-36
SLIDE 36

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

Short-lived certs reduce risk

slide-37
SLIDE 37

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

Valid for a single target (account, app, username, etc)

slide-38
SLIDE 38

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

Valid from a single host

slide-39
SLIDE 39

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

Control what the SSH session can be used for

slide-40
SLIDE 40

Scoping Credentials

slide-41
SLIDE 41

Bastion BLESS Instances Developer Access to Bastion == Access to Instances

slide-42
SLIDE 42

Bastion BLESS Bar App Developer App Defines Access List Foo App

slide-43
SLIDE 43

Bastion BLESS Bar App Developer Foo App App Defines Multiple Roles

slide-44
SLIDE 44

Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

instance_user:aws_account:app_name

slide-45
SLIDE 45

# Entries to enable BLESS TrustedUserCAKeys /etc/ssh/bless_user_ssh_cas.pub AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u

Config File

/etc/ssh/sshd_config

slide-46
SLIDE 46

bless_demo_instances:bless_demo_instances:123456789012:i-18badf00ddeadbeef

Config File

/etc/ssh/authorized_principals/blessdemo

slide-47
SLIDE 47

Operational Wins

slide-48
SLIDE 48

Bastion BLESS Invoke BLESS

Certificate Request

BLESS Response

Certificate

AWS KMS

Decrypt SSH CA private key

Instances

SSH with certificate
slide-49
SLIDE 49
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-50
SLIDE 50

Key Secrecy

Personal Keys Expiration Shared Keys

slide-51
SLIDE 51

Key Rotation

vs

Human Machine

slide-52
SLIDE 52

Logging Context

Jun 22 00:20:55 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA-CERT ID request[##################] for[user_name] from[10.0.1.1] command[test:us- east-1:bless_demo_instances:bles s_demo_instances-v001:oq-ssh] ssh_key[RSA de:ad:be:ef: 00:00:00:00:00:de:ad:be] ca[arn:aws:lambda:region:account :function:name] valid_to[2017/06/22 00:25:53] (serial 0) CA RSA SHA256:8badf00d000000008bad Jun 22 00:20:34 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA SHA256:de:ad:be:ef: 00:00:00:00:00:de:ad:be

Traditional SSH certificates with BLESS

slide-53
SLIDE 53

Availability Wins

LDAP

slide-54
SLIDE 54

Yes, It’s Open Source!

slide-55
SLIDE 55

https:/ /github.com/Netflix/bless

slide-56
SLIDE 56

https:/ /github.com/Netflix/bless

slide-57
SLIDE 57

https:/ /github.com/Netflix/bless

slide-58
SLIDE 58

https:/ /github.com/Netflix/bless

slide-59
SLIDE 59

https:/ /github.com/Netflix/bless

slide-60
SLIDE 60

Demo Time

slide-61
SLIDE 61

User Experience

slide-62
SLIDE 62
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-63
SLIDE 63
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-64
SLIDE 64
slide-65
SLIDE 65

Bastion Using BLESS

slide-66
SLIDE 66
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-67
SLIDE 67
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-68
SLIDE 68
slide-69
SLIDE 69

Instance SSHd Setup

slide-70
SLIDE 70
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-71
SLIDE 71
  • 5. AWS ssh tool: Use
session credentials to request a certificate BLESS AWS KMS
  • 6. BLESS: Decrypt SSH CA
private key with KMS
  • 7. BLESS: Generate and
sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs
  • 8. BLESS: Return
a short lived certificate Instances
  • 9. AWS ssh tool:
ssh with certificate
  • 10. sshd: Validate
certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace
  • 3. Pilgrim:
Generate Keypair Request SSH Cert
  • 2. AWS SSH tool:
Take request, determine user, application, instance
  • 4. Sshaman
Daemon: Determine calling user information. Use session credentials to request a certificate. Developer
  • 1. SSH: Auth to
Bastion Pilgrim Logs Sshaman Logs sshd Logs
slide-72
SLIDE 72
slide-73
SLIDE 73

Related Work

  • Lyft
  • Uses BLESS with client that runs on laptops
  • https:/
/eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d
  • Facebook
  • Leverages signed certificates with principals
  • https:/
/code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/
  • Wikimedia
  • SSH-agent proxy to protect private key on bastion
  • https:/
/blog.wikimedia.org/2017/03/22/keyholder/
slide-74
SLIDE 74

Questions?

bryanp@netflix.com https:/ /bryanpayne.org

[PS… I’m hiring!]