Bankrupt Covert Channel: Turning Network Predictability into Vulnerability Dmitrii Ustiugov , Plamen Petrov, Siavash Katebzadeh, Boris Grot University of Edinburgh This work is supported by ARM Center of Excellence at University of Edinburgh
Data Breaches Never Been More Relevant 2
Containing Data Breaches in Public Cloud Data breaches happen ☹ Spyware, side channels, … ● Secure cloud environment Outside world Spy with Receiver How to extract secret? secret Cloud vendors strive to contain stolen info Firewalls, authentication, … ● 3
Containing Data Breaches in Public Cloud Data breaches happen ☹ Spyware, side channels, … ● Secure cloud environment Outside world Isolation layers Spy with Receiver How to extract secret? secret Cloud vendors strive to contain stolen info Image by Lynn Willis, source Process and virtual machine isolation Firewalls, authentication, … ● ● Physical server isolation ● Virtual network isolation ● 4
Containing Data Breaches in Public Cloud Virtual network isolation Data breaches happen ☹ Physical server isolation Spyware, side channels, … ● VM isolation Spy with Receiver Secret extraction? secret Cloud vendors strive to contain stolen info Firewalls, authentication, … ● Are secrets safe now? 5
Covert Channels Definition : Communication without using legitimate data transfer mechanisms Usually via resource sharing (e.g., CPU cache) ● Example: Timing channel via access latency modulation ● High latency for transmitting “1”, low for “0” ○ Image by Rick Leche (flipped vertically), source Covert channels allow bypassing isolation layers 6
Network Covert Channels Virtual network isolation Allow communication across cluster/datacenter Physical server isolation Breach many isolation layers at once VM isolation Spy with Receiver Secret extraction secret Stereotypical thinking: Networks are noisy ⇨ low accuracy and low throughput channels But… Are modern networks noisy? 7
Emerging Networks in Public Cloud Node A One-sided CPU RDMA read Remote Direct Memory Access (RDMA) Memory Data Today most cloud providers offer RDMA networks NIC ● AWS, Azure, Alibaba, Oracle, … ○ RDMA network RDMA network packets bypass destination CPU Node B NIC Low round-trip latency: 2-4μsec ● High BW with commodity NICs: 100+Gb/s Memory ● Remote region for A Data Nodes use one-sided reads/writes to CPU their private data in remote node’s memory 8
Network BW vs. Memory BW Discrepancy Node A CPU First glance at bandwidth in modern servers Memory RDMA NICs offer 100-200Gb/s NIC ● Memory delivers >100GB/s (=800Gb/s) ● Network BW 100Gb/s Expectation: Memory BW always much larger Memory BW Node B NIC 100GB/s=800Gb/s Memory CPU 9
Network BW vs. Memory BW Discrepancy Node A CPU First glance at bandwidth in modern servers Memory RDMA NICs offer 100-200Gb/s NIC ● Memory delivers >100GB/s (=800Gb/s) ● Network BW 100Gb/s Expectation: Memory BW always much larger Wrong! Memory BW Node B NIC 100GB/s=800Gb/s Memory has 100s of internal devices (banks) ● Memory 10Gb/s Each bank delivers just ~10Gb/s ● … 100s of banks E.g., same for both Micron DDR2, DDR4 ○ CPU Bank behaves as FIFO: ~50ns fixed service time ○ Network traffic can easily congest one memory bank 10
Bankrupt: RDMA Intra-Cluster Covert Channel Bursts of RDMA reads Sender Receiver Key features Probes (individual RDMA reads) No direct communication between Sender and Receiver ● RDMA network Extremely stealthy! ● Intermediary NIC Basic idea Memory Bank Sender transmits the secret by modulating ● … the latency of one memory bank on an Intermediary node Receiver probes the bank latency and decodes the message ● CPU Intermediary is unrelated innocuous node ● No shared memory between Sender and Receiver Bank delay ○ High 1 0 1 0 1 0 Low Timeline 11
Constructing Bankrupt Bursts of RDMA reads Sender Receiver 1. Search for addresses that map to target bank Probes Challenge: CPU hashes addresses to determine the bank (individual RDMA reads) ● Sender and Receiver search addresses independently ● RDMA network Addresses different for Sender and Receiver ● Recall: No memory sharing! ○ Intermediary NIC Memory Bank 2. Determining communication parameters … Sender side ● How many RDMA reads per burst? ○ CPU Transmission frequency? ○ Receiver side ● Bank delay Receiving (probes) frequency? ○ High 1 0 1 0 1 0 Low Timeline 12
Finding Addresses in Same Bank 13
Virtual Memory Addressing Page offset Virtual address (VA) Cache block offset Virtual addresses translated to physical upon access … p p p p p p p c c c c c c Translation at page granularity 63 5 0 ● Same mechanism for local and remote (over RDMA) accesses ● Translate VA->PA Physical address (PA) ? ? ? ? ? ? ? … ? ? p p p p p p p c c c c c c Arbitrary bits, defined by OS All bits within page are same Within a page, physical address bits same as in virtual address 14
Bank Location Physical address (PA) ? ? ? ? ? ? ? … ? ? p p p p p p p c c c c c c Some physical address bits, “ bank bits ”, define bank Low-order bits to maximize bank-level parallelism Bank bits define bank location ● XOR function How to find addresses in same bank? These addresses have same bank bits ● Bank location Need to find exact bank bits positions 15
Same-Bank Addresses Search: Iteration 1 In page (e.g., 1GB) Virtual address (VA) In cache block Attacker (Sender and Receiver independently): … p p p p p p p 0 0 0 0 0 0 1. Chooses arbitrary addresses in remote memory Reads to same cache blocks (64B) coalesced ○ Throughput ⇨ set {5:0} bits to 0 2. Issues RDMA reads to chosen addresses and measures network throughput Network BW = number of serving banks x10Gb/s ○ … Single bank’s BW = ~10Gb/s (can vary slightly across vendors) Measurements 16
Same-Bank Addresses Search: Iteration 2 In page (e.g., 1GB) Virtual address (VA) In cache block Attacker (Sender and Receiver independently) … p p p p p p 0 0 0 0 0 0 0 1. Reduces subset of addresses Bank bit! a. Set {6:0} bits to 0 Throughput 2. Issues RDMA reads & measures throughput Throughput dropped by 2x ⇨ bit 6 is bank bit Single bank’s BW = ~10Gb/s (can vary slightly across vendors) Measurements 17
Same-Bank Addresses Search: Iteration 3 In page (e.g., 1GB) Virtual address (VA) In cache block Attacker (Sender and Receiver independently) … p p p p p 0 0 0 0 0 0 0 0 1. Reduces subset of addresses Not a bank bit! a. Set {7:0} bits to 0 Throughput 2. Issues RDMA reads & measures throughput Same throughput ⇨ bit 7 is NOT bank bit Single bank’s BW = ~10Gb/s (can vary slightly across vendors) Measurements 18
Same-Bank Addresses Search: Iteration 4 In page (e.g., 1GB) Virtual address (VA) In cache block Attacker (Sender and Receiver independently) … p p p p 0 0 0 0 0 0 0 0 0 1. Reduces subset of addresses a. Set {8:0} bits to 0 Throughput 2. Issues RDMA reads & measures throughput Throughput dropped by 2x ⇨ bit 8 is bank bit … Single bank’s BW = ~10Gb/s (can vary slightly across vendors) Measurements 19
Same-Bank Addresses Search: Iteration N In page (e.g., 1GB) Virtual address (VA) In cache block Attacker (Sender and Receiver independently) … p p 0 0 0 0 0 0 0 0 0 0 0 1. Reduces subset of addresses a. Set {N-6:0} bits to 0 Throughput 2. Issues RDMA reads & measures throughput Knowing bank bits locations, choose arbitrary addresses with bank bits equal to 0 ● … Throughput saturated cache block bits equal to 0 ● ⇨ all bank bits zeroed Single bank’s BW = ~10Gb/s (can vary slightly across vendors) Measurements Trivial complexity: Remote attacker finds addresses in <1 second 20
Bursts of RDMA reads Sender Receiver Probes (individual RDMA reads) RDMA network Determining Intermediary NIC Communication Parameters Memory Bank … CPU Bank delay High 1 0 1 0 1 0 Low Timeline 21
Sender Side Optimization Space Key parameter : Sender’s burst size Transmission accuracy Larger bursts more pronounced ● ⇨ higher accuracy Especially in noisy networks ○ Smaller bursts drain quicker ● Transmission frequency ⇨ higher frequency Burst size 22
Receiver Side Probe Round-Trip Delay Transmission period estimation Transmission period Transmitted packets comprise fixed-size preamble and payload ● Example: 32-bit preamble & 200-bit payload ○ Receiver iteratively determines the transmission period by ● looking for pre-agreed preamble value 1 0 Measurements Key parameter : Probing frequency Several probes (measurements) per transmission period ● Timeline Found little sensitivity on decoding accuracy with probing ● frequency > 2MHz (1/0.5μseconds) 23
Evaluation Platforms Private Cluster Public Cloud: CloudLab Utah (isolated and loaded network ) (80% utilized during measurements) Cluster size: 6 nodes Cluster size: 200 nodes Infiniband Infiniband CPU: Xeon E5-2630v4 (Broadwell) CPU: Xeon E5-2640v4 (Broadwell) RAM: 64GB, DDR4-2400 RAM: 64GB, DDR4-2400 NIC: Mellanox CX-5, 56Gb/s NIC: Mellanox CX-4, 50Gb/s 24
Recommend
More recommend
