Reducing Remote Read Footprint Bartek Plotka @bwplotka Getting - - PowerPoint PPT Presentation

▶
reducing remote read footprint
SMART_READER_LITE
LIVE PREVIEW

Reducing Remote Read Footprint Bartek Plotka @bwplotka Getting - - PowerPoint PPT Presentation

Feb 11, 2023 279 likes •423 views

Reducing Remote Read Footprint Bartek Plotka @bwplotka Getting samples from TSDB /api/v1/query /api/v1/query_range Prometheus PromQL Engine Select() Select() /read Queryable Interface TSDB @bwplotka Remote Read: Response /read

slide-1
SLIDE 1

Reducing Remote Read Footprint

Bartek Plotka @bwplotka

slide-2
SLIDE 2

@bwplotka

Prometheus PromQL Engine TSDB Queryable Interface Select()

Getting samples from TSDB

Select() /api/v1/query /api/v1/query_range /read

slide-3
SLIDE 3

@bwplotka

Remote Read: Response

/read

<proto> message QueryResult { repeated prometheus.TimeSeries timeseries = 1; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; }

SAMPLES

https://developers.google.com/protocol-buffers over HTTP

slide-4
SLIDE 4

Prometheus

Chunks Chunks Series C

@bwplotka

TSDB

Remote Read: Bottlenecks

Select()

Chunks Chunks Chunks Series B Chunks Chunks Chunks Series A

Filling… Huge RemoteRead Response

Decode samples

slide-5
SLIDE 5
slide-6
SLIDE 6

Prometheus

@bwplotka

Remote Read: Bottlenecks

Done! Huge RemoteRead Response

Marshall

proto binary

Compress

slide-7
SLIDE 7

@bwplotka

Remote Read Request: Allocations on heap

Single request for 8h, 10 000 series Prometheus: 1.5GB Thanos: 2GB

slide-8
SLIDE 8

@bwplotka

Remote Read: Response (Prometheus 2.13.0)

/read

<proto> message QueryResult { repeated prometheus.TimeSeries timeseries = 1; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; } <proto> message ChunkedReadResponse { repeated prometheus.ChunkedSeries chunked_series = 1; } message ChunkedSeries { repeated Label labels = 1; repeated Chunk chunks = 2; }

SAMPLES STREAMED_XOR_CHUNKS [NEW]

slide-9
SLIDE 9

@bwplotka

Remote read: Response (Prometheus 2.13.0)

/read

<proto> message QueryResult { repeated prometheus.TimeSeries timeseries = 1; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; } <proto> message ChunkedReadResponse { repeated prometheus.ChunkedSeries chunked_series = 1; } message ChunkedSeries { repeated Label labels = 1; repeated Chunk chunks = 2; }

SAMPLES STREAMED_XOR_CHUNKS [NEW]

sample{Value, Timestamp} XOR chunk (~120 samples)

slide-10
SLIDE 10

@bwplotka

Chunked protocol (Prometheus v2.13.0)

message ChunkedReadResponse { repeated prometheus.ChunkedSeries chunked_series = 1; } big-endian uint32 Checksum of ChunkedReadResponse uvarint64 of ChunkedReadResponse

1MB Max

slide-11
SLIDE 11

Prometheus

Chunks Chunks Series E

@bwplotka

TSDB

Remote Read (Prometheus v2.13.0)

Select()

Chunks Chunks Chunks Chunks Chunks Chunks Series D Series C Chunked Read Response

Frame Marshall & Compress

1MB

slide-12
SLIDE 12

@bwplotka

Remote Read Request: Heap Allocs

Prometheus: 150MB (was 1.5GB) Single request for 8h, 10 000 series Thanos: ~0B (was 2GB)

~2.5x reduced latency ~2x less CPU time

slide-13
SLIDE 13

Bartek Plotka @bwplotka

Thanks!

Questions?

Links:

  • APIs: https://prometheus.io/docs/prometheus/latest/querying/api/
  • Remote read: https://github.com/prometheus/prometheus/blob/master/prompb/remote.proto#L27
  • Go stream client: https://github.com/prometheus/prometheus/blob/master/storage/remote/chunked.go
  • Blog: https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/