XASM: A Cross-Enclave Composition Mechanism for Exascale System - - PowerPoint PPT Presentation

▶
xasm a cross enclave composition mechanism for exascale
SMART_READER_LITE
LIVE PREVIEW

XASM: A Cross-Enclave Composition Mechanism for Exascale System - - PowerPoint PPT Presentation

Oct 26, 2022 250 likes •647 views

XASM: A Cross-Enclave Composition Mechanism for Exascale System Software Noah Evans, Kevin Pedretti, Brian Kocoloski, John Lange, Michael Lang, Patrick G. Bridges nevans@sandia.gov 6/1/16 Sandia National Laboratories is a multi-program

slide-1
SLIDE 1

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND NO. 2011-XXXXP

XASM: A Cross-Enclave Composition Mechanism for Exascale System Software

Noah Evans, Kevin Pedretti, Brian Kocoloski, John Lange, Michael Lang, Patrick G. Bridges nevans@sandia.gov 6/1/16

slide-2
SLIDE 2

Outline

▪ Application composition and why it matters ▪ Hobbes: System software support for application composition ▪ XASM: Cross Enclave Shared Memory ▪ Conceptual modifications needed for Hobbes ▪ Implementation on the Kitten lightweight kernel ▪ Performance evaluation ▪ Future work ▪ Conclusions

2

slide-3
SLIDE 3

Composition Use Cases in Next-Generation HPC

▪ End-to-end science workflows ▪ Coupled simulation, analysis, and tools ▪ In-situ and in-transit analytics ▪ Multi-physics applications ▪ Application Introspection ▪ Performance analysis, concurrency throttling ▪ Debugging ▪ This presentation concentrates on co-located simulation and analytics workloads

3

slide-4
SLIDE 4

Why Composition is Important

▪ Data movement is expensive ▪ Writes to filesystem especially ▪ Need to compartmentalize complexity ▪ Jamming everything into one executable is a pain, fragile

4

Application Visualization File System Display Bad: Insufficient bandwidth Application Visualization File System Display Bad: Inefficient use of compute infrastructure Application/Visualization File System Display Good! But Application and Visualization have different OS/R requirements

slide-5
SLIDE 5

Example: SNAP and Spectrum Analysis

▪ SNAP ▪ Neutronics proxy, based on PARTISN ▪ Simulates reactor using sweep3d ▪ Spectrum analysis ▪ After each timestep ▪ Two separate processes communicating

5

slide-6
SLIDE 6

Outline

▪ Application composition and why it matters ▪ Hobbes: System software support for application composition ▪ XASM: Cross Enclave Shared Memory ▪ Conceptual modifications needed for Hobbes ▪ Implementation for Linux and Kitten lightweight kernel ▪ Performance evaluation ▪ Future work ▪ Conclusions

6

slide-7
SLIDE 7

Hobbes Project: Systems Software Support for Composition

▪ Application level composition difficult for application writer ▪ Lots of research on how to support (Adios ’10, Gold-rush ’13)

7

  • Goals
  • Minimize Data movement in

composition

  • Optimizing the scheduling of

composed workloads

slide-8
SLIDE 8

Hobbes Project: Why Systems Software Should Support Composition

8

Linux Producer Consumer Kitten

physical memory pool

Cow Region Pinned Snapshot

Xemem

slide-9
SLIDE 9

Hobbes Project: Why Systems Software Should Support Composition

9

Linux Producer Consumer Kitten

physical memory pool

Cow Region Pinned Snapshot

Xemem

  • Space sharing and time sharing

virtualization using “Enclaves”

slide-10
SLIDE 10

Hobbes Project: Why Systems Software Should Support Composition

10

Linux Producer Consumer Kitten

physical memory pool

Cow Region Pinned Snapshot

Xemem

  • Space sharing and time sharing

virtualization using “Enclaves”

  • Communicate using
  • ptimized transports
slide-11
SLIDE 11

Outline

▪ Application composition and why it matters ▪ Hobbes: System software support for application composition ▪ XASM: Cross Enclave Shared Memory ▪ Conceptual modifications needed for Hobbes ▪ Implementation on the Kitten lightweight kernel ▪ Performance evaluation ▪ Future work ▪ Conclusions

11

slide-12
SLIDE 12

XASM: Optimizing Data Movement for Composition

▪ Transparent: No changes to APIs ▪ Consistent: Neither side, producer or consumer, sees changes made by the other ▪ Asynchronous: No locking needed

12

  • Fig. 4: TCASM Architecture
slide-13
SLIDE 13

Trick: Copy On Write

▪ Allows “lazy” copying of data - can avoid the copy in some situations ▪ OS notified when process trying to modify shared page

13

  • No modification = no copy
  • Modification incurs the extra cost of a

page fault

slide-14
SLIDE 14

Outline

▪ Application composition and why it matters ▪ Hobbes: System software support for application composition ▪ XASM: Cross Enclave Shared Memory ▪ Conceptual modifications needed for Hobbes ▪ Implementation on the Kitten lightweight kernel ▪ Performance evaluation ▪ Future work ▪ Conclusions

14

slide-15
SLIDE 15

Kitten Implementation

▪ Implementations heavily dependent on virtual memory systems ▪ How are the virtual to physical mappings are maintained will affect contention and allocation policy ▪ Need to optimize contention and allocation tradeoffs for performance

15

slide-16
SLIDE 16

Kitten Virtual Memory

▪ In Kitten, user allocates physical memory explicitly ▪ User chooses own virtual to physical mappings ▪ Kitten flat-mapped, no page faults! ▪ Additions to Kitten needed for XASM: ▪ Add a page fault handler to Kitten ▪ Add a mechanism to make physical memory pools available to individual processes

16

slide-17
SLIDE 17

Kitten XASM

Producer Consumer

Kitten

// PRODUCER arena_map_backed_region_anywhere(my_aspace, &region, …); for(i=0; i < datalen; i++) simulate(data[i]); aspace_copy(id, &dst, 0);

slide-18
SLIDE 18

Kitten XASM

Producer Consumer

Kitten region

// PRODUCER arena_map_backed_region_anywhere(my_aspace, &region, …); for(i=0; i < datalen; i++) simulate(data[i]); aspace_copy(id, &dst, 0);

pool

slide-19
SLIDE 19

Kitten XASM

// PRODUCER arena_map_backed_region_anywhere(my_aspace, &region, …); for(i=0; i < datalen; i++) simulate(data[i]); aspace_copy(id, &dst, 0);

Producer Consumer

Kitten pool region

slide-20
SLIDE 20

Kitten XASM

// PRODUCER arena_map_backed_region_anywhere(my_aspace, &region, …); for(i=0; i < datalen; i++) simulate(data[i]); aspace_copy(id, &dst, 0);

Producer Consumer

Kitten pool region region

slide-21
SLIDE 21

Kitten XASM

// PRODUCER arena_map_backed_region_anywhere(my_aspace, &region, …); for(i=0; i < datalen; i++) simulate(data[i]); aspace_copy(id, &dst, 0);

Producer Consumer

Kitten pool region region

slide-22
SLIDE 22

Kitten XASM

Producer Consumer

Kitten

// CONSUMER aspace_smartmap(xasm_id, my_id, SMARTMAP_ALIGN, SMARTMAP_ALIGN); for(i=0; i < datalen; i++) analyze(data[i]); aspace_unsmartmap(xasm_id, my_id, …); aspace_destroy(xasm_id);

Kitten pool region region

slide-23
SLIDE 23

Kitten XASM

Producer Consumer

Kitten

// CONSUMER aspace_smartmap(xasm_id, my_id, SMARTMAP_ALIGN, SMARTMAP_ALIGN); for(i=0; i < datalen; i++) analyze(data[i]); aspace_unsmartmap(xasm_id, my_id, …); aspace_destroy(xasm_id);

Kitten Kitten region pool region

slide-24
SLIDE 24

Kitten XASM

Producer Consumer

// CONSUMER aspace_smartmap(xasm_id, my_id, SMARTMAP_ALIGN, SMARTMAP_ALIGN); for(i=0; i < datalen; i++) analyze(data[i]); aspace_unsmartmap(xasm_id, my_id, …); aspace_destroy(xasm_id);

Kitten Kitten Kitten region pool region

slide-25
SLIDE 25

Kitten XASM

Producer Consumer

// CONSUMER aspace_smartmap(xasm_id, my_id, SMARTMAP_ALIGN, SMARTMAP_ALIGN); for(i=0; i < datalen; i++) analyze(data[i]); aspace_unsmartmap(xasm_id, my_id, …); aspace_destroy(xasm_id);

Kitten Kitten Kitten pool region

slide-26
SLIDE 26

Outline

▪ Application composition and why it matters ▪ Hobbes: System software support for application composition ▪ Xasm: Transparently Consistent Asynchronous Shared Memory ▪ Conceptual modifications need for Hobbes ▪ Implementation for Linux and Kitten lightweight kernel ▪ Performance evaluation ▪ Future work ▪ Conclusions

26

slide-27
SLIDE 27

Performance Evaluation

▪ Need to show that it works with minimal performance

  • verhead

▪ Questions to answer: ▪ What is the overhead of page fault handling? ▪ How does the overhead of Xasm compare to base case and synchronized shared memory?

27

slide-28
SLIDE 28

Experimental Design

▪ Sandy bridge 2.2 GHz,12 core, 2 socket system, 24 GB (Hyper-threading off) ▪ Hobbes environment on Linux ▪ Use cycle counter for kernel measurements of page faults ▪ SNAP + Spectrum Analysis as macro benchmark ▪ Compare worst case (xpmem+spin locks), Xasm, best case (no analytics) ▪ Inter-enclave on Kitten (6 trials per size) ▪ x*y*z = 96, 200, 324, 490, 768, 6144

28

slide-29
SLIDE 29

0.00 0.25 0.50 0.75 1.00 3000 6000 9000 12000

Cycles Density

Operating System Kitten Linux

Distribution of Cycles In Page Fault Handler

Kitten faults less noisy

29

slide-30
SLIDE 30

Linux slower 25% of time

30

0.00 0.25 0.50 0.75 1.00 5000 10000 15000

Cycles Percentage Faults Completed

Operating System Kitten Linux

CDF of Cycles In Page Fault Handler

slide-31
SLIDE 31

XASM Overhead Negligible Between Processes

31

0.000 0.005 0.010 100 1000

Problem Size (Bytes) Time In Situ (Seconds)

Composed Applications

SNAP no analytics SNAP w/ Analytics Linux via TCASM SNAP w/ Analytics Linux via XPMEM

Time Spent In Situ

slide-32
SLIDE 32

<4% In Worst Case

32

  • 0.00

0.01 0.02 0.03 100 1000 10000

Problem Size (Bytes) Percentage of Time In Situ

Composed Applications

  • SNAP no analytics

SNAP w/ Analytics Linux via TCASM SNAP w/ Analytics Linux via XPMEM

Percentage of Time In Situ

slide-33
SLIDE 33

Address Space Copies Expensive but Still Manageable

33

  • 1

2 3 100 1000

Problem Size (x*y*z) Time In Situ (Seconds)

Composed Applications

  • SNAP Kitten w/ Analytics Linux via XASM

Time Spent In Situ (Seconds)

slide-34
SLIDE 34

Still Less Than 3% of Execution Time with Scale

34

  • 0.00

0.03 0.06 0.09 0.12 100 1000

Problem Size (x*y*z) Percentage of Time In Situ

Composed Applications

  • SNAP Kitten w/ Analytics Linux via XASM

Percentage of Time Spent In Situ

slide-35
SLIDE 35

Outline

▪ Application composition and why it matters ▪ Hobbes: System software support for application composition ▪ XASM: Transparently Consistent Asynchronous Shared Memory ▪ Conceptual modifications need for Hobbes ▪ Implementation for Linux and Kitten lightweight kernel ▪ Performance evaluation ▪ Future work ▪ Conclusions

35

slide-36
SLIDE 36

Future work

▪ We know what systems software can do in this case but we don’t know what it should do ▪ For a Copy on Write mechanism what does fault tolerance and flow control look like? ▪ Copy on Write not necessarily good for some HPC applications ▪ Anything that touches every page ▪ Wait free queue instead? ▪ Integrate XASM with in-situ runtimes like ADIOS

36

slide-37
SLIDE 37

Conclusions

▪ Systems software can simplify composition ▪ XASM is one potential mechanism ▪ XASM is working in the Hobbes OS/R stack ▪ Performance is reasonable and using XASM is easy ▪ Potential piece of exascale infrastructure

37

slide-38
SLIDE 38

Thank You

38