Route Server Automation and ROV Nick Pratley nick@ix.asn.au Life - - PowerPoint PPT Presentation

▶
route server automation and rov
SMART_READER_LITE
LIVE PREVIEW

Route Server Automation and ROV Nick Pratley nick@ix.asn.au Life - - PowerPoint PPT Presentation

Apr 01, 2023 562 likes •703 views

Route Server Automation and ROV Nick Pratley nick@ix.asn.au Life Under Lockdown: how to stop heists, hijacks, and hostages Timeline When Metrics and Reporting to Members NOW 17 th 18 th August RS1 Upgrades 24th 25 th August RS2

slide-1
SLIDE 1

Route Server Automation and ROV

Nick Pratley nick@ix.asn.au Life Under Lockdown: how to stop heists, hijacks, and hostages

slide-2
SLIDE 2

Timeline

When

Metrics and Reporting to Members NOW RS1 Upgrades 17th – 18th August RS2 Upgrades 24th – 25th August Route Server Automation (yes, including daily AS-SET updates) 1st September Drop Invalid Routes 1st September

slide-3
SLIDE 3

Metrics and Reporting

  • What are we going to drop when we enable ROV?
  • No RPKI support on current software, no easy & fast way to get and validate rib
  • Emails to members with invalid routes, do you know about them, can we help?

Hopefully after tonight everyone is just going to go and do it? ;-)

  • Hacked some python scripts:
  • SSH to route servers & get rib: “ssh -c ’sudo birdc show route table master’ rs1.*.ix.asn.au”
  • Validate routes using Cloudflare's rpki.json (https://rpki.cloudflare.com/rpki.json) with a

Radix tree in python for searches

  • Export data to influxdb as a telegraf input running hourly
  • Graph with Grafana
  • https://metrics.ix.asn.au/d/58WdNHGMk/ix-rpki
slide-4
SLIDE 4
slide-5
SLIDE 5

So – what will we drop?

AS10AS10214 10214 121.200.32.0/24 AS10214 10214 121.200.33.0/24 AS132405 132405 2001:df0:2c7:100::/64 AS132405 132405 2001:df0:2c7:1::/64 AS132405 132405 2001:df0:2c7:200::/64 AS132405 132405 2001:df0:2c7:3001::/64 AS132405 132405 43.250.92.0/24 AS132405 132405 43.250.93.0/24 AS132405 132405 43.250.94.0/24 AS132405 132405 43.250.95.0/24 AS13335 13335 103.21.244.0/24 AS13335 13335 2606:4700:7000::/48 AS134090 134090 103.106.90.0/24 AS134090 134090 103.106.91.0/24 AS134090 59256 2401:9cc0:200::/48 AS134090 59256 2401:9cc0:300::/48 AS135513 17741 114.31.103.0/24 AS136001 136001 202.179.134.0/24 AS139609 45891 202.1.160.0/20 AS139609 45891 202.1.176.0/20 AS23838 23838 116.90.135.0/24 AS23838 23838 2401:f000:2:200::/56 AS23838 23838 2401:f000:2::/56 AS23838 23838 2401:f000:32:101::/64 AS23838 23838 2401:f000:32:103::/64 AS23838 23838 2401:f000:32:13::/64 AS23838 23838 2401:f000:32:16::/64 AS23838 23838 2401:f000:32:18::/64 AS23838 23838 2401:f000:32:2::/64 AS23838 23838 2401:f000:32:27::/64 AS23838 23838 2401:f000:32:28::/64 AS23838 23838 2401:f000:32:29::/64 AS23838 23838 2401:f000:32:37::/64 AS23838 23838 2401:f000:32:4::/64 AS23838 23838 2401:f000:32:5::/64 AS23838 23838 2401:f000:32:6::/64 AS23838 23838 2401:f000:32:86::/64 AS23838 23838 2401:f000:32:87::/64 AS23838 23838 2401:f000:32:92::/64 AS23838 23838 2401:f000:32:99::/64 AS23838 23838 2401:f000:32:c000::1/128 AS23838 23838 2401:f000:32:c000::2/128 AS23838 23838 2401:f000:32:c000::/64 AS23838 23838 2401:f000:32:c001::105/128 AS23838 23838 2401:f000:32:c001::11/128 AS23838 23838 2401:f000:32:c002::2/128 AS23838 23838 2401:f000:32:c002::3/128 AS23838 23838 2401:f000:32:c003::1/128 AS23838 23838 2401:f000:32:c005::/64 AS23838 23838 2401:f000:5:1::/64 AS23838 23838 2401:f000:5::/64 AS23838 23838 2402:1c00:0:1::/64 AS23838 23838 2402:1c00:1000:1::/64 AS23838 23838 2402:1c00:10::/126 AS23838 23838 2402:1c00:2000:1001::/64 AS23838 23838 2402:1c00::/64 AS23838 23838 2402:1c00:dead:beef::/64 AS23838 23838 2402:1c00:fffe::/127 AS23838 23838 2402:1c00:fffe::4/127 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:a0/124 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:c0/124 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:c2/127 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:c6/127 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:d0/124 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:e0/124 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:f0/127 AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:f2/127 AS3356 132199 120.28.146.0/24 AS3356 132199 120.28.165.0/24 AS3356 132199 120.28.252.0/22 AS3356 132199 180.190.84.0/24 AS3356 132199 180.191.192.0/23 AS3356 132199 180.191.194.0/23 AS3356 132199 180.191.196.0/22 AS3356 132199 180.191.224.0/23 AS3356 132199 180.191.227.0/24 AS3356 132199 180.191.228.0/22 AS3356 132199 180.191.232.0/22 AS3356 132199 180.191.236.0/22 AS3356 132199 180.191.240.0/21 AS3356 132199 180.191.248.0/22 AS3356 132199 180.191.252.0/22 AS3356 132199 180.191.32.0/22 AS3356 132199 180.191.36.0/22 AS3356 132199 180.191.40.0/22 AS3356 132199 180.191.44.0/22 AS3356 132199 180.191.48.0/24 AS3356 132199 180.191.49.0/24 AS3356 132199 180.191.50.0/23 AS3356 132199 180.191.52.0/22 AS3356 132199 180.191.58.0/23 AS3356 132199 180.191.60.0/22 AS3356 132199 222.127.196.0/24 AS3356 138197 103.126.150.0/24 AS3356 18190 120.28.15.0/24 AS3356 27281 2620:116:800e::/48 AS3356 45731 103.20.190.0/24 AS36351 36351 185.147.58.0/24 AS36351 36351 185.147.59.0/24 AS36351 7489 27.100.39.0/24 AS38195 134409 2407:c280:ffff::/48 AS38220 38220 2403:cc00:4000::/36 AS38561 38561 2402:f00::/32 AS45177 38220 2403:cc00:4000::/36 AS45177 45177 150.107.32.0/23 AS45177 45177 150.107.34.0/23 AS45280 45280 2402:7e00:0:102::/64 AS45280 45280 2402:7e00:10:100::/56

slide-6
SLIDE 6

So – what will we drop?

AS4826 10214 121.200.32.0/24 AS4826 10214 121.200.33.0/24 AS4826 133075 2407:f100:4::/48 AS4826 13335 2606:4700:7000::/48 AS4826 17918 122.252.150.0/24 AS4826 17918 122.252.151.0/24 AS4826 56030 45.118.190.0/24 AS4826 56068 131.203.76.0/23 AS4826 56304 131.203.63.0/24 AS4826 59256 2401:9cc0:200::/48 AS4826 59256 2401:9cc0:300::/48 AS4826 9503 2402:6000:106::/48 AS4826 9503 2402:6000:109::/48 AS4826 9503 2402:6000:10d::/48 AS4826 9503 2402:6000:10e::/48 AS4826 9503 2402:6000:201::/48 AS4826 9503 2402:6000:202::/48 AS4826 9790 2404:4400:1000::/36 AS4826 9790 2404:4400:2000::/36 AS4826 9790 2404:4408:8::/48 AS56030 56030 45.118.190.0/24 AS58511 134409 2407:c280:ffff::/48 AS64098 38220 2403:cc00:4000::/36 AS64098 64098 2403:780:f::/48 AS7175 7175 2402:c00:2:a00::/56 AS7545 137079 103.107.247.0/24 AS7545 18405 122.200.160.0/20 AS7575 7575 2001:388:70d2::/48 AS7600 9297 103.74.188.0/24 AS7600 9297 103.74.189.0/24 AS9297 9297 103.74.188.0/24 AS9297 9297 103.74.189.0/24 AS9500 9500 2407:7000:f300::/48 AS9790 56068 131.203.76.0/23 AS9790 56304 131.203.63.0/24 AS9790 56304 2400:3d80:2000:100::/56 AS9790 56304 2400:3d80:2000:200::/56 AS9790 56304 2400:3d80:2000:400::/56 AS9790 56304 2400:6900:1030:200::/56 AS9790 56304 2400:6900:1030:3::/64 AS9790 56304 2400:6900:1030:462::/64 AS9790 56304 2400:6900:1030:464::/64 AS9790 56304 2400:6900:1030:500::/56 AS9790 56304 2400:6900:1030:5::/64 AS9790 56304 2400:6900:1030:600::/64 AS9790 56304 2400:6900:1030:6::/64 AS9790 56304 2400:6900:1030:700::/56 AS9790 56304 2400:6900:1030:e00::/56 AS9790 56304 2400:6900:2000:1000::/56 AS9790 56304 2400:6900:2000:1200::/56 AS9790 56304 2400:6900:2000:1500::/56 AS9790 56304 2400:6900:2000:1800::/56 AS9790 56304 2400:6900:2000:1900::/56 AS9790 56304 2400:6900:2000:1b00::/56 AS9790 56304 2400:6900:2000:1c00::/56 AS9790 56304 2400:6900:2000:1d00::/56 AS9790 56304 2400:6900:2000:1e00::/56 AS9790 56304 2400:6900:2000:1f00::/56 AS9790 56304 2400:6900:2000:2300::/56 AS9790 56304 2400:6900:2000:300::/56 AS9790 56304 2400:6900:2000:3800::/56 AS9790 56304 2400:6900:2000:3900::/56 AS9790 56304 2400:6900:2000:3a00::/56 AS9790 56304 2400:6900:2000:400::/56 AS9790 56304 2400:6900:2000:4e00::/56 AS9790 56304 2400:6900:2000:500::/56 AS9790 56304 2400:6900:2000:600::/56 AS9790 56304 2400:6900:2000:800::/56 AS9790 56304 2400:6900:2000:900::/56 AS9790 56304 2400:6900:2000:a00::/56 AS9790 56304 2400:6900:2000:c00::/56 AS9790 56304 2400:6900:2000:e00::/56 AS9790 56304 2400:6900:2000:f00::/56 AS9790 56304 2400:6900:2030:9::/64 AS9790 56304 2400:6900:2030:e::/64 AS9790 56304 2400:6900:2:2::/64 AS9790 56304 2400:6900:2:3::/64 AS9790 56304 2400:6900:2::/64 AS9790 56304 2400:6900:2:ff00::/64 AS9790 56304 2400:6900:3010:200::/56 AS9790 56304 2400:6900:3010:4::/64 AS9790 56304 2400:6900:ffff:100::/56 AS9790 56304 2400:6900:ffff:10::/60 AS9790 56304 2400:6900:ffff:1::/64 AS9790 56304 2400:6900:ffff:200::/56 AS9790 56304 2400:6900:ffff:2::/64 AS9790 56304 2400:6900:ffff:3::/64 AS9790 56304 2400:6900:ffff:a::/64 AS9790 56304 2400:6900:ffff:b::/64 AS9790 56304 2400:6900:ffff:f300::/56 AS9790 56304 2400:6900:ffff:f500::/56 AS9790 56304 2400:6900:ffff:f::/64 AS9790 56304 2400:6900:ffff:fe00::/56 AS9790 56304 2400:6900:ffff:fffd::/64 AS9790 56304 2400:6900:ffff:fffe::/64 AS9790 56304 2401:9480:300:10d::/64 AS9790 9503 202.53.182.0/24

slide-7
SLIDE 7

Route Server Software Upgrades

  • Current Route Server Software
  • Ubuntu 16.04 LTS – End of Support in April 2021
  • BIRD 1.6.2 - Released September 2016
  • No RPKI / ROV support (without mangling prefix lists – ew)
  • New Software
  • Ubuntu 20.04 LTS – Released April 2020
  • BIRD 2.0.7 – Released October 2019
  • RTR support
  • Plan
  • There are two route servers in each exchange (are you peering with both)?
  • Pick a time window, disable rs1, upgrade Ubuntu, upgrade bird, deploy automated BIRD config
  • 2 hours per route server – apt full-upgrade takes ageeeeeees, actual downtime of around 10 minutes for

reboot and software ugprade

  • Rinse and repeat for second RS
  • Rinse and repeat for each exchange
slide-8
SLIDE 8

Route Server Automation – Current State

  • We have an operations portal that is the source of

truth

  • It has a restful json API to query data
  • Slack chatops to generate all network and RS

configs

  • errbot
  • More hacky python scripts
  • Jinja2 templates
  • AS-SET updates are manual still
  • Tried early alpha automation (cronjob re-generating

some tagged services), lead to some interesting edge cases of broken configs

  • Disabled that, and started working on version 2
slide-9
SLIDE 9

Route Server Automation – Version 2

  • ARouteServer (https://github.com/pierky/arouteserver)
  • Codify Route Server Config
  • YAML clients built from Portal API
  • Community config defined in git

repo

  • Takes ~15 minutes to deploy globally
  • Rundeck
  • Less moving parts than Stackstorm
  • Runs workflow at 3pm AEST daily
  • Generates config for all exchanges

using ARouteServer

  • Test and Deploy Route Server config

via SSH

slide-10
SLIDE 10

Route Origin Validation - Tooling

  • OctoRPKI (https://github.com/cloudflare/cfrpki#octorpki)
  • Running on 5 servers, one in each exchange
  • Generates a json list of Route Object Authorization (ROA)
  • Performs all crypto validation
  • Exports ROA as signed curated json list to GoRTR instances
  • GoRTR (https://github.com/cloudflare/gortr)
  • Run locally on each Route Server
  • Uses json from OctoRPKI sources and streams into special BIRD tables for

validation

  • Basically no logic and no overhead required, thus running on route server is

acceptable

slide-11
SLIDE 11

Route Origin Validation – BIRD Test

  • BIRD 2.0.7 supports RTR protocol
  • Connect to local GoRTR instance & instance

running on partner route server

  • Validate routes from peers, before importing
  • Accept valid and unknown routes
  • Drop invalids before importing into master table
slide-12
SLIDE 12

Route Origin Validation – Example Config

slide-13
SLIDE 13

Route Server Automation and ROV

Nick Pratley nick@ix.asn.au Life Under Lockdown: how to stop heists, hijacks, and hostages