Reducing Remote Read Footprint Bartek Plotka @bwplotka Getting - PowerPoint PPT Presentation

Reducing Remote Read Footprint Bartek Plotka @bwplotka

Getting samples from TSDB /api/v1/query /api/v1/query_range Prometheus PromQL Engine Select() Select() /read Queryable Interface TSDB @bwplotka

Remote Read: Response /read SAMPLES <proto> message QueryResult { repeated prometheus.TimeSeries timeseries = 1; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; } https://developers.google.com/protocol-buffers over HTTP @bwplotka

Remote Read: Bottlenecks Prometheus Select() TSDB Filling… Series C Series A Series B Chunks Chunks Chunks Chunks Chunks Chunks Chunks Chunks Huge RemoteRead Response Decode samples @bwplotka

Remote Read: Bottlenecks Prometheus Done! proto binary Marshall Compress Huge RemoteRead Response @bwplotka

Remote Read Request: Allocations on heap Single request for 8h, 10 000 series Thanos: 2GB Prometheus: 1.5GB @bwplotka

Remote Read: Response (Prometheus 2.13.0) /read SAMPLES <proto> message QueryResult { repeated prometheus.TimeSeries timeseries = 1; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; } STREAMED_XOR_CHUNKS [NEW] <proto> message ChunkedReadResponse { repeated prometheus.ChunkedSeries chunked_series = 1; } message ChunkedSeries { repeated Label labels = 1; repeated Chunk chunks = 2; } @bwplotka

Remote read: Response (Prometheus 2.13.0) /read SAMPLES <proto> sample{Value, Timestamp} message QueryResult { repeated prometheus.TimeSeries timeseries = 1; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; } STREAMED_XOR_CHUNKS [NEW] <proto> XOR chunk (~120 samples) message ChunkedReadResponse { repeated prometheus.ChunkedSeries chunked_series = 1; } message ChunkedSeries { repeated Label labels = 1; repeated Chunk chunks = 2; } @bwplotka

Chunked protocol (Prometheus v2.13.0) message ChunkedReadResponse { repeated prometheus.ChunkedSeries chunked_series = 1; } 1MB Max big-endian uint32 Checksum of ChunkedReadResponse uvarint64 of ChunkedReadResponse @bwplotka

Remote Read (Prometheus v2.13.0) Prometheus Select() 1MB TSDB Series E Series C Series D Marshall & Frame Chunks Chunks Chunks Chunks Chunks Chunks Chunks Chunks Compress Chunked Read Response @bwplotka

Remote Read Request: Heap Allocs Single request for 8h, 10 000 series Prometheus: 150MB (was 1.5GB) Thanos: ~0B (was 2GB) ~2.5x reduced latency ~2x less CPU time @bwplotka

Thanks! Questions? Links: APIs: https://prometheus.io/docs/prometheus/latest/querying/api/ ● Remote read: https://github.com/prometheus/prometheus/blob/master/prompb/remote.proto#L27 ● Go stream client: https://github.com/prometheus/prometheus/blob/master/storage/remote/chunked.go ● Blog: https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/ ● Bartek Plotka @bwplotka