Experiences Building InfluxDB in Go Paul Dix paul@influxdb.com - - PowerPoint PPT Presentation

▶
experiences building influxdb in go
SMART_READER_LITE
LIVE PREVIEW

Experiences Building InfluxDB in Go Paul Dix paul@influxdb.com - - PowerPoint PPT Presentation

May 06, 2023 455 likes •968 views

Experiences Building InfluxDB in Go Paul Dix paul@influxdb.com @pauldix Who am I? informs my experience working with a language like Go co-founder, CEO, programmer Author Languages worked with Professionally in order VBScript Delphi

slide-1
SLIDE 1

Experiences Building InfluxDB in Go

Paul Dix paul@influxdb.com @pauldix

slide-2
SLIDE 2

Who am I?

informs my experience working with a language like Go

slide-3
SLIDE 3

co-founder, CEO, programmer

slide-4
SLIDE 4

Author

slide-5
SLIDE 5

Languages worked with Professionally in order

  • VBScript
  • Delphi
  • C# and VB.NET
  • Ruby & Javascript
  • Java
  • C (only a little)
  • Ruby & Javascript
  • Scala
  • Go
slide-6
SLIDE 6

Mix of static and dynamic, but most time in dynamic languages

slide-7
SLIDE 7

Open source time series database written in Go

slide-8
SLIDE 8

What’s a time series?

slide-9
SLIDE 9

Stock trades and quotes

slide-10
SLIDE 10

Metrics

slide-11
SLIDE 11

Analytics

slide-12
SLIDE 12

Events

slide-13
SLIDE 13

Sensor data

slide-14
SLIDE 14

Two kinds of time series data…

slide-15
SLIDE 15

Regular time series

t0 t1 t2 t3 t4 t6 t7 Samples at regular intervals

slide-16
SLIDE 16

Irregular time series

t0 t1 t2 t3 t4 t6 t7 Events whenever they come in

slide-17
SLIDE 17

Why’d we pick Go?

slide-18
SLIDE 18

Some project requirements

  • Self contained binary install
slide-19
SLIDE 19

Some project requirements

  • Self contained binary install
  • Performance
slide-20
SLIDE 20

previous experience with it

slide-21
SLIDE 21

faster development than working with C/C++

slide-22
SLIDE 22

growing community

slide-23
SLIDE 23

simplicity of the language

significant advantage for picking up new programmers and contributors

slide-24
SLIDE 24

InfluxDB Project Stats

  • First Commit - September 26th, 2013
  • 176 Contributors
  • 68,000 LOC
slide-25
SLIDE 25

Team background

  • Java
  • Scala
  • Python
  • Ruby
  • C++
slide-26
SLIDE 26

What has been great

slide-27
SLIDE 27

Static typing

can’t believe I’m saying this

slide-28
SLIDE 28

Community

slide-29
SLIDE 29

Performance

slide-30
SLIDE 30

What surprised us

slide-31
SLIDE 31

GC hasn’t been a problem

slide-32
SLIDE 32

Contributors with no previous Go experience

simplicity of the language wins again

slide-33
SLIDE 33

Haven’t really missed generics

slide-34
SLIDE 34

except when I do…

slide-35
SLIDE 35

// Sort methods func (a Values) Len() int { return len(a) } func (a Values) Swap(i, j int) { a[i], a[j] = a[j], a[i] } func (a Values) Less(i, j int) bool { return a[i].Time().UnixNano() < a[j].Time().UnixNano() }

slide-36
SLIDE 36

Duplicate code for each data type

select percentile(90, value) from cpu where time > now() - 1d group by time(10m)

slide-37
SLIDE 37

// Iterator represents an iterator over a series. type Iterator interface { SeekTo(seek int64) (key int64, value interface{}) Next() (key int64, value interface{}) Ascending() bool }

we have to cast this later

slide-38
SLIDE 38

Costs on performance

f := val.(float64) // do some math

slide-39
SLIDE 39

// FloatIterator represents an iterator over a series. type FloatIterator interface { SeekTo(seek int64) (key int64, value float64) Next() (key int64, value float64) Ascending() bool }

slide-40
SLIDE 40

Implement for every function

func MapMean(input *MapInput) interface{} { if len(input.Items) == 0 { return nil }

  • ut := &meanMapOutput{}

for _, item := range input.Items {

  • ut.Count++

switch v := item.Value.(type) { case float64:

  • ut.Total += v

case int64:

  • ut.Total += float64(v)
  • ut.ResultType = Int64Type

} } return out }

slide-41
SLIDE 41

Could use interfaces, but there’s a performance penalty there too…

slide-42
SLIDE 42

What has been bad

slide-43
SLIDE 43

Dependency management

slide-44
SLIDE 44

Experiences migrating to 1.5

slide-45
SLIDE 45

Compile times longer

  • n a project this size it’s noticeable
slide-46
SLIDE 46

Performance more about our code

slide-47
SLIDE 47

Go 1.5 issue forced us to revert

https://github.com/golang/go/issues/12233

slide-48
SLIDE 48

Unit and integration tests won’t save you.

if you operate at scale, only full blown scale tests will tell you anything and even then you may not find it

slide-49
SLIDE 49

Q&A

paul@influxdb.com @pauldix