CS 839: Design the Next-Generation Database Lecture 24: HTAP - - PowerPoint PPT Presentation

▶
cs 839 design the next generation database lecture 24 htap
SMART_READER_LITE
LIVE PREVIEW

CS 839: Design the Next-Generation Database Lecture 24: HTAP - - PowerPoint PPT Presentation

Aug 06, 2023 242 likes •553 views

CS 839: Design the Next-Generation Database Lecture 24: HTAP Xiangyao Yu 4/16/2020 1 Announcements Vote on the topic of the last lecture Option 1: Streaming [required] Discretized Streams: Fault-Tolerant Streaming Computation at Scale

slide-1
SLIDE 1

Xiangyao Yu 4/16/2020

CS 839: Design the Next-Generation Database Lecture 24: HTAP

1

slide-2
SLIDE 2

Announcements

2

Vote on the topic of the last lecture

Option 1: Streaming

  • [required] Discretized Streams: Fault-Tolerant Streaming Computation at Scale
  • [optional] Apache FlinkTM: Stream and Batch Processing in a Single Engine
  • [optional] The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and

Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing

Option 2: Time series

  • [required] Gorilla: A Fast, Scalable, In-Memory Time Series Database
  • [optional] Time Series Management Systems: A Survey
slide-3
SLIDE 3

Discussion Highlights

3

FaaS vs. BaaS for databases

  • BaaS advantages: simplifies communication and state sharing, caching
  • BaaS disadvantages: potentially lower CPU and memory utilization
  • FaaS advantages: fine-granularity pricing model, auto-scaling
  • FaaS disadvantages: overhead of inter-function coordination, functions have

limited resources and execution time, communication through S3, inherently designed for small functions

What can BaaS (e.g., Snowflake) borrow from FaaS?

  • Auto-scaling: Dynamically resource allocation and fine-grained pricing

Benefits and limiting factors of running OLTP on serverless computing?

  • Benefits: Elastic scaling based on demand, transactions are inherently short-

lived

  • Limiting factors: S3 has no read-after-write consistency, concurrency control

is hard due to lack of communication

slide-4
SLIDE 4

Today’s Paper

4

ICDE 2011

slide-5
SLIDE 5

HTAP: Hybrid Transactional/Analytical Processing

Hybrid transactional/analytical processing (HTAP), a term created by Gartner Inc in 2014: Key advantage: reducing time to insight

5

Hybrid transactional/analytical processing (HTAP) is an emerging application architecture that "breaks the wall" between transaction processing and analytics. It enables more informed and "in business real time" decision making.

slide-6
SLIDE 6

OLTP vs. OLAP (Slide from L2)

OLTP database Transactions OLAP database

(Update Intensive) (Read Intensive, rare updates)

  • Takes hours for

conventional databases

  • Takes seconds for Hybrid

transactional/analytical processing (HTAP) systems

6

slide-7
SLIDE 7

HTAP Design Options [1]

7

Single System for OLTP and OLAP

  • Using Separate Data Organization for OLTP and OLAP
  • Same Data Organization for both OLTP and OLAP

Separate OLTP and OLAP Systems

  • Decoupling the Storage for OLTP and OLAP
  • Using the Same Storage for OLTP and OLAP

[1] Özcan, Fatma, Yuanyuan Tian, and Pinar Tözün. "Hybrid transactional/analytical processing: A survey." ICMD, 2017.

Hyper

slide-8
SLIDE 8

Background: Through the Looking Glass [2]

8

[2] Harizopoulos, S., Abadi, D. J., Madden, S., & Stonebraker, M. OLTP through the looking glass, and what we found there. SIGMOD 2008

slide-9
SLIDE 9

Background: H-STORE [3]

9

[3] Kallman, R., et al. H-store: a high-performance, distributed main memory transaction processing system. VLDB 2008

Single partition transactions are sequentially executed Multi-partition transactions lock entire partitions Support short, stored-procedure transactions

slide-10
SLIDE 10

Background: VoltDB

H-Store is commercialized into VoltDB VoltDB has some cool features

  • Active-active replication (deterministic execution)
  • Command logging

10

slide-11
SLIDE 11

Hyper

11

Execute analytical queries without blocking transactions

slide-12
SLIDE 12

Virtual Memory Snapshots

12

Create consistent database snapshot for OLAP queries to read Transactions run with copy-on-write to avoid polluting the snapshots

slide-13
SLIDE 13

Fork()

13

fork() creates a new process by duplicating the calling process. The new process is referred to as the child process. The calling process is referred to as the parent process.

Linux Programmer's Manual Does not copy all the memory pages Does copy the parent’s page table (all pages set to readonly mode) Copy-on-write (COW)

  • If any page is modified by either parent or child process, a new

page is created for the corresponding process

slide-14
SLIDE 14

Cost of Fork()

14

Cost of fork() is proportional to the page table size, which depends on

  • Database size
  • Page size
slide-15
SLIDE 15

Fork-Based Virtual Snapshots

15 Page tables OLTP process OLAP process Page OLTP process OLAP process Page Page’

ref=2 ref=1 ref=1

slide-16
SLIDE 16

Multiple OLAP Session

16 OLTP process

OLAP Session: Group of OLAP queries that access the same snapshot

B ref=1 A ref=1

slide-17
SLIDE 17

Multiple OLAP Session

17 Snapshot 1

OLAP Session: Group of OLAP queries that access the same snapshot

B ref=2 A ref=2 OLTP process

slide-18
SLIDE 18

Multiple OLAP Session

18 Snapshot 1 B ref=2 A ref=1

OLAP Session: Group of OLAP queries that access the same snapshot

A’ ref=1 OLTP process

slide-19
SLIDE 19

Multiple OLAP Session

19 Snapshot 1 B ref=3 A ref=1

OLAP Session: Group of OLAP queries that access the same snapshot

A’ ref=2 Snapshot 2 OLTP process

slide-20
SLIDE 20

Multiple OLAP Session

20 Snapshot 1 B ref=2 A ref=1

OLAP Session: Group of OLAP queries that access the same snapshot

A’ ref=2 Snapshot 2 OLTP process

slide-21
SLIDE 21

Multi-Threaded OLTP Processing

21

Single-partition transaction

  • Sequential execution within partition
  • Different partitions run in parallel

Multi-partition transaction

  • System-wide sequential execution
slide-22
SLIDE 22

Multi-Threaded OLTP Processing

22

Single-partition transaction

  • Sequential execution within partition
  • Different partitions run in parallel

Multi-partition transaction

  • System-wide sequential execution

When to fork()?

  • Option 1: Fork after quiescing all threads
  • Option 2: Fork in the middle of a transaction and then undo the transaction’s

changes

slide-23
SLIDE 23

Logging and Checkpointing

Logging

  • Logical redo logging

Checkpointing

  • Based on the same VM snapshot

mechanism

23

slide-24
SLIDE 24

Evaluation – Performance Comparison

24

Config 1 Config 2 Config 3

slide-25
SLIDE 25

Evaluation – Memory Consumption

25

slide-26
SLIDE 26

Hyper Today?

26

slide-27
SLIDE 27

HTAP – Q/A

State-of-the-art in HTAP? Overhead of Hyper? Row-format has the same performance as column-format for OLTP? Really necessary to do real-time analytical work? What if data does not fit in memory? (Anti-caching) Why not using shared memory and a concurrency control? Why logical logging is a problem in conventional system? Evaluation is weak Analytical data no longer fits in memory in 2020

27

slide-28
SLIDE 28

Topic of the Last Lecture

28

Option 1: Streaming

  • [required] Discretized Streams: Fault-Tolerant Streaming Computation at Scale
  • [optional] Apache FlinkTM: Stream and Batch Processing in a Single Engine
  • [optional] The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and

Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing

Option 2: Time series

  • [required] Gorilla: A Fast, Scalable, In-Memory Time Series Database
  • [optional] Time Series Management Systems: A Survey
slide-29
SLIDE 29

Group Discussion

What are the challenges of applying the VM-snapshot idea to a shared-memory OLTP system? Fork() replicates the page table, which is expensive when the database is large. Can you think of any approach to reduce this cost? Given the four possible designs of HTAP ({single system, separate system} x {shared data, separate data}), which one is the most promising in your opinion? What if you have infinite number of machines?

29