CS 744: SNOWFLAKE Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - - PowerPoint PPT Presentation

▶
cs 744 snowflake
SMART_READER_LITE
LIVE PREVIEW

CS 744: SNOWFLAKE Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - - PowerPoint PPT Presentation

Oct 16, 2022 332 likes •569 views

CS 744: SNOWFLAKE Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - Assignment 1 grades out! - Assignment 2 by mid-week - Midterm this week! - Project Proposal Peer review AEFIS FEEDBACK How has your experience been reading papers? Are

slide-1
SLIDE 1

CS 744: SNOWFLAKE

Shivaram Venkataraman Fall 2020

slide-2
SLIDE 2

ADMINISTRIVIA

  • Assignment 1 grades out!
  • Assignment 2 by mid-week
  • Midterm this week!
  • Project Proposal Peer review
slide-3
SLIDE 3

AEFIS FEEDBACK

How has your experience been reading papers? Are the lectures useful for learning? How are the discussion groups? Did you get to know students in the class? Would it help to have the same group each time? Anything else we could improve for the second half?

slide-4
SLIDE 4

Machine Learning SQL Applications

slide-5
SLIDE 5

CLOUD COMPUTING STACK

Scalable Storage Systems Computational Engines Machine Learning SQL

slide-6
SLIDE 6

SNOWFLAKE: GOALS

Software-as-a-Service Elastic Highly Available Semi-Structured Data

slide-7
SLIDE 7

SNOWFLAKE DESIGN

slide-8
SLIDE 8

STORAGE VS COMPUTE

Shared Nothing Multi Cluster, Shared Data

slide-9
SLIDE 9

STORAGE: HYBRID COLUMNAR

Alice 32 Bob 22 Eve 24 Victor 27

Alice,32,Bob,22 Eve,24,Victor,27 Alice, Bob, 32,22 Eve, Victor,24,27 Row-oriented Hybrid Columnar

slide-10
SLIDE 10

VIRTUAL WAREHOUSES

Elasticity, Isolation Local caching, Stragglers

slide-11
SLIDE 11

CLOUD SERVICES

Concurrency Control Pruning

slide-12
SLIDE 12

FAULT TOLERANCE

slide-13
SLIDE 13

SEMI STRUCTURED DATA

{ first_name: “john”, last_name: “doe”,

  • rder_id: “1234”,

} { first_name: “bucky”, last_name: “badger”,

  • rder_id: “52342”,
  • rder_date: “3/3/2020”,

}

Extraction operation Flattening Infer types, Pruning

slide-14
SLIDE 14

TIME TRAVEL?

Multiple versions of table (MVCC) Undo accidental deletes Cheap to clone / snapshot a table

slide-15
SLIDE 15

SECURITY

Hierarchical key management Key rotation, re-keying

slide-16
SLIDE 16

SUMMARY, TAKEAWAYS

Snowflake

  • Cloud computing à Elastic data warehouse
  • Key idea: Separation of compute and storage!
  • Hybrid columnar storage format
  • Elastic compute with virtual warehouses
  • Pruning, semi-structured optimizations, fault tolerant
slide-17
SLIDE 17

AEFIS FEEDBACK

slide-18
SLIDE 18

DISCUSSION

https://forms.gle/ZFosdUnizXYABAE86

slide-19
SLIDE 19

We see how Snowflake leads to the design of an elastic data warehouse. If we were to similarly design an Elastic PyTorch for training how would the design look? What are some design trade-offs compared to existing PyTorch?

slide-20
SLIDE 20

NEXT STEPS

Next class: Midterm! AEFIS feedback Project proposal peer feedback assignments

slide-21
SLIDE 21