Go(lang) to Python Europython 2019 Basel, Stefan Baerisch - - PowerPoint PPT Presentation

▶
go lang to python
SMART_READER_LITE
LIVE PREVIEW

Go(lang) to Python Europython 2019 Basel, Stefan Baerisch - - PowerPoint PPT Presentation

Jun 15, 2023 45 likes •603 views

Go(lang) to Python Europython 2019 Basel, Stefan Baerisch stefan@stbaer.com 1 Why this Talk? Good Qt Bindings Frontend Very Expressive / Productive Python Fast Enough Good For Experiments Good Library Support Good Cross Plattform

slide-1
SLIDE 1

Go(lang) to Python

Europython 2019 Basel, Stefan Baerisch stefan@stbaer.com

  • 1
slide-2
SLIDE 2

Why this Talk?

  • 2

Frontend Python Backend ???

Good Performance Good Concurrency / Parallelism Expressive Enough Good Qt Bindings Very Expressive / Productive Fast Enough Good For Experiments Good Library Support Good Cross Plattform Support Low “Frustration” Factor

slide-3
SLIDE 3

Nice Things about Go

  • 3

Go has:

Automatic Memory Management Lightweight, Managed Go-Routines

No Sync/ASync

Very Quick Compiler Quite Simple Language (“The Programming language for people that don’t like programming

languages”)

slide-4
SLIDE 4

Why Go with Python? Not {C++Rust} ?

  • 4

<Your Factor Here> Likely Good ? ? Ok Great Low Frustration Factor Good Great Good Great Language Expressiveness Ok Great

Performance / Concurrency / Parallelism

Good Cpp / Rust Go Python

Low Friction

slide-5
SLIDE 5

Some Spoilers

  • 5

Image: Professor Butts and the Self-Operating Napkin (1931), Wikipedia, Public Domain

  • Things work, but are currently far from elegant.
  • They are, however, quite interesting
slide-6
SLIDE 6

Wrapping Go - The Basics

  • 6
slide-7
SLIDE 7

Options for Using Go from Python

  • 7

…

Python Go

Binary RPC (gRPC / Thrift) Call External Command Bindings goPy

Roll me Own

Same Process? Published Api? Transport Mechamisn Transport Format Questions…

…. REST / JSON RPC

slide-8
SLIDE 8

Wrapping Go with CGO

  • 8

https://golang.org/cmd/cgo/ https://dave.cheney.net/2016/01/18/cgo-is-not-go

Slower Build Times More Complex Builds When calling Go from C Significant Overhead Restrictions in Sharing Data CGO has some drawbacks…

Go C

Go Routines, scheduled by runtime Go Segmented Stacks Go Calling Convention Threads Fixed-size Stacks C-Calling Convention

CGO

CGO makes bindings possible…

slide-9
SLIDE 9

High Level Architecture

  • 9

Python Go CGO Cython C / C++ Support Code Go Support Code

Idiomatic Go Code we want to use in Python Export C API. Convert Parameters and Result Values Manage Lifetime of Exported Go Data C/C++ Representation

  • f Value-based

parameters and results Define Extention Types and Wrappers for CGO C API

slide-10
SLIDE 10

Build Process

  • 10

Go Files CGO File Go Build Static Libary Header Files C / C++ Support python / C Static Libary Header Files setup.py Cython File Python Module Step 1 Step 2

slide-11
SLIDE 11

Wrapping a Simple Function

  • 11
slide-12
SLIDE 12

Simple Function - Goals

  • 12

Go CGO Cython Python

func Add(v, v2 int) int {} //export cgo_Add func cgo_Add(cgo_v1, cgo_v2 C.int) C.int {} cpdef int add(int v1, int v2): cdef extern from "cgo_lib/cgo_lib.h": int cgo_Add(int p0, int p1)

Code we want to use Provide C API Wrap C API Use Functionality

slide-13
SLIDE 13

Simple Function - Go Code

  • 13

Result, can be multiple values, similar to tuples

  • Parameters. name(s), then type

Function name, exported if it starts with capital letter

slide-14
SLIDE 14

Simple Function - CGO

  • 14

make available in C lib C parameter / result types Type conversation Actual work

Build the Package Name of C Lib / Header Build Library

slide-15
SLIDE 15

Excerpt from the Header File

  • 15

Our Function CGO generated C Header

slide-16
SLIDE 16

Setup.py for Cython

  • 16

Build Command Our C Library

slide-17
SLIDE 17

Simple Function - Cython

  • 17

Declare Function Wrapper Function Using it in Python

slide-18
SLIDE 18

Wrapping User Defined Types

  • 18
slide-19
SLIDE 19

User Defined Types in Go and Python

  • 19

Go Python

Classes / Extention Types User Defined Types Exported Methods Methods Constructors / Factoryfunctinos Functions Destructor (__del__, __dealloc___,…) Garbage Collection Exceptions Error Result Values <ignored for now> Interfaces, Channels

slide-20
SLIDE 20

Wrapping User Types - Go Code

  • 20

Strings Object Pointer / Reference Methods

slide-21
SLIDE 21

Wrapping User Types - References

  • 21

Data Cython Data Cython Proxy Jobs of the Proxy Keep the Go Garbage Collector away from the data shared with Python Translate the Go Pointer / Value into something we can share with Python Not Possible: Direct Sharing of Go Data Possible: Using a Proxy How to use Data References in Python?

NOT OK

OK

slide-22
SLIDE 22
  • 22

Go Data Pointer ID Mapping 1 Go Data 2 1 Cython Wrapper Member Member Method Method Member Member Method Method

…

1 Cython Wrapper Cython Instance Management

slide-23
SLIDE 23

Wrapping User Types - References

  • 23

Id <=> Pointer Mapping Functions Package Level Variable Bookkeeping

slide-24
SLIDE 24

Wrapping User Types - CGO

  • 24

Convert Parameters / Results Go Function Call

“Constructor” “Destructor”

slide-25
SLIDE 25

Wrapping User Types - CGO

  • 25

Convert Parameters Convert Results Get Object Convert Parameters / Results

slide-26
SLIDE 26

Wrapping User Types - Headers

  • 26

CGO generated Headers Cython Function Declaration

slide-27
SLIDE 27

User Types - Cython Constructor

  • 27

Manage Objects Only Member - ID of the underlying Go objects Convert Parameters Request new or register existing object Register Object

slide-28
SLIDE 28

Cython Method

  • 28

Get Name as C String from Go, via CGO

slide-29
SLIDE 29

Python Again

  • 29
slide-30
SLIDE 30

Wrapping Maps, Lists, etc. as Results and Parameters Also: Errors

  • 30
slide-31
SLIDE 31

Complex Parameters/Results - Go

  • 31

Golang Error Return Objects Return Slice of Strings Return Map

slide-32
SLIDE 32

Complex Results - How?

  • 32

How to express slices/list or maps/dicts on the way from Go to Python (and back)?

error exception ?

  • bject id

extension type user type map ? dict ? slice list string *char str int, uint… (unsigned) int int Cython / Python Cgo / C Go

slide-33
SLIDE 33

Complex Results - Idea

  • 33

C++ has std::vector, std::map Cython can wrap C++

errors <=> exception

  • utput parameter

<=> <=> (unsign ed) int Cython <=> <=>

  • bject id

extension type user type map *void to std::map, helper functions dict *void to std::vector, helper functions slice list string *char str int, uint… (unsigned) int int Python Cgo / C Go

Output parameters are ok for errors, for now

slide-34
SLIDE 34

Complex Results - C API

  • 34
slide-35
SLIDE 35

Complex Results - C++

  • 35
slide-36
SLIDE 36

Complex Results -CGO

  • 36
slide-37
SLIDE 37

Complex Results - Cython

  • 37
slide-38
SLIDE 38

Errors -CGO

  • 38
slide-39
SLIDE 39

Errors- Cython

  • 39
slide-40
SLIDE 40

Complex Results & Errors - Python

  • 40
slide-41
SLIDE 41

Wrapping Callbacks

  • 41
slide-42
SLIDE 42

Wrapping Callbacks - Go Code

  • 42

How to have some callbacks? Callback: Int (age) to bool

slide-43
SLIDE 43

Wrapping Callbacks - Thougths

  • 43

C callpacks are function pointers… Cython can build C-functions that call Python Code So, we can have a C function pointer with Python Code but CGO cannot call function pointers CGO can call a c function that calls a function pointer We want Go to call Python and use the result We need to express the callback in our C/CGO layer

slide-44
SLIDE 44

Callbacks - Rube Goldberg Approach

  • 44

Cython C Callback Cython Method Wrapper C Function Pointer C Function to Call Function Pointer Go Method with Call Back CGO Version of Go Method Local Go Func Python Callback

passed to passed to calls assigns Python callback to is calls calls calls passed to defines and call passed to captures in Closure

slide-45
SLIDE 45

C Helper Functions for Callbacks

  • 45
slide-46
SLIDE 46

Wrapping Callbacks - CGO

  • 46
slide-47
SLIDE 47

Wrapping Callbacks - Cython

  • 47
slide-48
SLIDE 48

Python

  • 48
slide-49
SLIDE 49

Performance, Effort, Summary

  • 49
slide-50
SLIDE 50

Effort and Gain Wrapping Golang

  • 50

Golang ~ 130 Lines CGO ~ 220 Lines C Header ~30 Lines CPP Support Code ~50 Lines Cython ~200 Lines

For this (simple) example, the amount of wrapper code is significant Code is repetitive, leads itself to automation

slide-51
SLIDE 51

A Simple (Accessor) Benchmark

  • 51
slide-52
SLIDE 52

Go Bindings vs. Python Performance

  • 52

1493ms 1360ms 3209ms

93ms

443ms 256ms 619ms

Go Python

slide-53
SLIDE 53

Performance of Method Calls

  • 53

Go

Python

slide-54
SLIDE 54

Wrapping Go - Lessons Learned

  • 54

Things work Cython works and is a pleasure to use CGO works Some boilerplate, but nothing to bad Many Go Feature can be expressed in Python Performance is a challenge Improvements possible (Errors, Interfaces, Channels?) Overhead to call into Go (runtime) Mapping from Pointers to IDs Looking into alternatives (C++ Msg Queue?)

slide-55
SLIDE 55
  • 55

The End