How to write Rust instead of C and get away with it (yes, its a - - PowerPoint PPT Presentation

▶
how to write rust instead of c and get away with it
SMART_READER_LITE
LIVE PREVIEW

How to write Rust instead of C and get away with it (yes, its a - - PowerPoint PPT Presentation

Feb 24, 2023 231 likes •1.62k views

How to write Rust instead of C and get away with it (yes, its a Python talk) Antonio Verardi Flavien Raynaud @porosVII @flavray Yelps Mission Connecting people with great local businesses. Why we are here schema = {

slide-1
SLIDE 1

How to write Rust instead of C and get away with it

(yes, it’s a Python talk)

Antonio Verardi – Flavien Raynaud @porosVII – @flavray

slide-2
SLIDE 2

Yelp’s Mission

Connecting people with great local businesses.

slide-3
SLIDE 3

Why we are here

slide-4
SLIDE 4
slide-5
SLIDE 5

schema = { "type": "record", "name": "my_record", "fields": [ { "name": "a", "type": "long", "default": 42 }, { "name": "b", "type": "string" }, ] }

slide-6
SLIDE 6

schema = { "type": "record", "name": "my_record", "fields": [ { "name": "a", "type": "long", "default": 42 }, { "name": "b", "type": "string" }, ] }

Binary Data Serialization format JSON {"a": 27, "b": "foo"} Apache Avro™ 0x 36 06 66 6f 6f

https://avro.apache.org
slide-7
SLIDE 7

2004

slide-8
SLIDE 8

2018 …

slide-9
SLIDE 9
slide-10
SLIDE 10

. . .

slide-11
SLIDE 11
slide-12
SLIDE 12

What you will bring home

  • How to use Rust code in Python packages
  • Why and When
  • Python, C and Rust facts
  • Jokes?
slide-13
SLIDE 13

The Problem

slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16

Scale up/out

slide-17
SLIDE 17

Scale up/out ¯\_(ツ)_/¯

slide-18
SLIDE 18

Change Interpreter

slide-19
SLIDE 19
slide-20
SLIDE 20

Use

slide-21
SLIDE 21

C Extensions

slide-22
SLIDE 22

ctypes / cffi

slide-23
SLIDE 23

ctypes / cffi

pyavro-rs pyavro-rs

slide-24
SLIDE 24
slide-25
SLIDE 25

fastavro fastavro

slide-26
SLIDE 26

How to CFFI

slide-27
SLIDE 27

from cffi import FFI ffi = FFI() with open("my_header.h") as header: ffi.cdef(header.read()) lib = ffi.dlopen("my_binary.so") lib.my_function()

slide-28
SLIDE 28

from cffi import FFI ffi = FFI() with open("my_header.h") as header: ffi.cdef(header.read()) lib = ffi.dlopen("my_binary.so") lib.my_function()

slide-29
SLIDE 29

from cffi import FFI ffi = FFI() with open("my_header.h") as header: ffi.cdef(header.read()) lib = ffi.dlopen("my_binary.so") lib.my_function()

slide-30
SLIDE 30

from cffi import FFI ffi = FFI() with open("my_header.h") as header: ffi.cdef(header.read()) lib = ffi.dlopen("my_binary.so") lib.my_function()

slide-31
SLIDE 31

ABI

slide-32
SLIDE 32

Application Binary Interface

slide-33
SLIDE 33

ABI

slide-34
SLIDE 34

ABI

slide-35
SLIDE 35

ABI

slide-36
SLIDE 36

ABI

slide-37
SLIDE 37
slide-38
SLIDE 38

Rust

  • guaranteed memory safety
slide-39
SLIDE 39

Rust

  • guaranteed memory safety
  • concurrency without data races
slide-40
SLIDE 40

Rust

  • guaranteed memory safety
  • concurrency without data races
  • zero-cost abstractions
slide-41
SLIDE 41

Rust

  • guaranteed memory safety
  • concurrency without data races
  • zero-cost abstractions
  • modern syntax
slide-42
SLIDE 42

Rust

  • guaranteed memory safety
  • concurrency without data races
  • zero-cost abstractions
  • modern syntax
  • awesome tooling
slide-43
SLIDE 43

Rust

  • guaranteed memory safety
  • concurrency without data races
  • zero-cost abstractions
  • modern syntax
  • awesome tooling
slide-44
SLIDE 44

How to Avro

slide-45
SLIDE 45
  • serialization and deserialization
  • serde.rs
  • implement Serializer & Deserializer
  • code generation
  • avro.apache.org/docs/current/spec.html
slide-46
SLIDE 46
slide-47
SLIDE 47
slide-48
SLIDE 48

#[derive(Debug, Deserialize, Serialize)] pub struct Test { pub a: i64, pub b: String, }

slide-49
SLIDE 49

#[derive(Debug, Deserialize, Serialize)] pub struct Test { pub a: i64, pub b: String, }

slide-50
SLIDE 50

let schema = Schema::parse_str(r#"{ "type": "record", "name": "test", "fields": [ { "name": "a", "type": "long", "default": 42 }, { "name": "b", "type": "string" }, ] }"#)?;

slide-51
SLIDE 51

let mut writer = Writer::new( &schema, Vec::new() // io::Write );

slide-52
SLIDE 52

let mut writer = Writer::new( &schema, Vec::new() // io::Write ); let record1 = Test { a: 27, b: "foo".to_owned() };

slide-53
SLIDE 53

let mut writer = Writer::new( &schema, Vec::new() // io::Write ); let record1 = Test { a: 27, b: "foo".to_owned() }; let record2 = Record::new(); record2.put("a", 27); record2.put("b", “foo”.to_owned());

slide-54
SLIDE 54

let mut writer = Writer::new( &schema, Vec::new() // io::Write ); let record1 = Test { a: 27, b: "foo".to_owned() }; let record2 = Record::new(); record2.put("a", 27); record2.put("b", “foo”.to_owned()); writer.append_ser(record1); writer.append(record2); writer.flush();

slide-55
SLIDE 55

let mut bytes = writer.into_inner();

slide-56
SLIDE 56

let mut bytes = writer.into_inner(); let mut reader = Reader::new(&bytes[..]);

slide-57
SLIDE 57

let mut bytes = writer.into_inner(); let mut reader = Reader::new(&bytes[..]); for record in reader { let test = from_value::<Test>(&record?); // … }

slide-58
SLIDE 58

How to FFI

slide-59
SLIDE 59

Structs

slide-60
SLIDE 60

#[repr(C)] pub struct AvroStr { pub data: *mut c_char, pub len: usize, pub owned: bool, } pub struct AvroReader;

slide-61
SLIDE 61

#[repr(C)] pub struct AvroStr { pub data: *mut c_char, pub len: usize, pub owned: bool, } pub struct AvroReader;

slide-62
SLIDE 62

#[repr(C)] pub struct AvroStr { pub data: *mut c_char, pub len: usize, pub owned: bool, } pub struct AvroReader;

slide-63
SLIDE 63

#[repr(C)] pub struct AvroStr { pub data: *mut c_char, pub len: usize, pub owned: bool, } pub struct AvroReader;

slide-64
SLIDE 64

Functions

slide-65
SLIDE 65

ffi_fn! { unsafe fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema> { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) } }

slide-66
SLIDE 66

ffi_fn! { unsafe fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema> { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) } }

slide-67
SLIDE 67

ffi_fn! { unsafe fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema> { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) } }

slide-68
SLIDE 68

ffi_fn! { unsafe fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema> { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) } }

slide-69
SLIDE 69

ffi_fn! { unsafe fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema> { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) } }

slide-70
SLIDE 70

Inside the macro

slide-71
SLIDE 71

#[no_mangle] pub unsafe extern "C" fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema, Error> + panic::UnwindSafe { utils::safe_unwind(|| { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) }) }

slide-72
SLIDE 72

#[no_mangle] pub unsafe extern "C" fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema, Error> + panic::UnwindSafe { utils::safe_unwind(|| { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) }) }

slide-73
SLIDE 73

#[no_mangle] pub unsafe extern "C" fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema, Error> + panic::UnwindSafe { utils::safe_unwind(|| { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) }) }

slide-74
SLIDE 74

#[no_mangle] pub unsafe extern "C" fn avro_schema_from_json( json: *const AvroStr ) -> Result<*mut AvroSchema, Error> + panic::UnwindSafe { utils::safe_unwind(|| { let schema = Schema::parse_str((&*json).as_str())?; Ok( Box::into_raw(Box::new(schema)) as *mut AvroSchema ) }) }

slide-75
SLIDE 75

Gotchas

slide-76
SLIDE 76
  • unwind un-safety
slide-77
SLIDE 77
  • unwind un-safety
  • error codes
slide-78
SLIDE 78
  • unwind un-safety
  • error codes
  • explicit memory management
slide-79
SLIDE 79
  • unwind un-safety
  • error codes
  • explicit memory management
  • enum duplication
slide-80
SLIDE 80
  • unwind un-safety
  • error codes
  • explicit memory management
  • enum duplication
  • complex arguments
slide-81
SLIDE 81
  • unwind un-safety
  • error codes
  • explicit memory management
  • enum duplication
  • complex arguments
  • ...
slide-82
SLIDE 82
slide-83
SLIDE 83
slide-84
SLIDE 84

avro-rs-ffi

slide-85
SLIDE 85

Header File

slide-86
SLIDE 86

typedef struct { char *data; uintptr_t len; bool owned; } AvroStr; typedef struct AvroReader AvroReader; AvroSchema *avro_schema_from_json(const AvroStr *json);

slide-87
SLIDE 87
slide-88
SLIDE 88

cbindgen is awesome!

slide-89
SLIDE 89

include/avro.h: $(shell find src -type f -name "*.rs") RUSTUP_TOOLCHAIN=nightly \ cbindgen -v -c cbindgen.toml . -o $@

Makefile

slide-90
SLIDE 90

include/avro.h: $(shell find src -type f -name "*.rs") RUSTUP_TOOLCHAIN=nightly \ cbindgen -v -c cbindgen.toml . -o $@

Makefile

slide-91
SLIDE 91

cbindgen is awesome!

slide-92
SLIDE 92

How to Python wrapper

slide-93
SLIDE 93

from pyavro_rs._lowlevel import ffi, lib def avro_int(n): return lib.avro_value_int_new(n) def avro_list(items): array = lib.avro_value_array_new(len(items)) for item in items: value = Value(item) lib.avro_array_append(array, value.value) return array

slide-94
SLIDE 94

class Value(RustObject): __dealloc_func__ = lib.avro_value_free __TYPE_TO_AVRO = { NoneType: avro_null, bool: avro_bool, int: avro_int, float: avro_float, str: avro_str, bytes: avro_bytes, list: avro_list, tuple: avro_list, dict: avro_dict, } def __new__(cls, datum): fn = cls.__TYPE_TO_AVRO.get(type(datum)) if fn is None: raise Exception('Unable to encode type {}'.format(type(datum))) return cls._from_objptr(fn(datum)) @property def value(self): return self._objptr

slide-95
SLIDE 95

schema = Schema('''{.....}''') writer = Writer(schema) writer.append({'a': 27, 'b': 'foo'}) writer.append({'a': 42, 'b': 'bar'}) writer.flush()

  • utput = writer.into()

reader = Reader(output) for record in reader: print(record)

slide-96
SLIDE 96

init: git submodule add https://github.com/flavray/avro-rs-ffi build-rust: cd avro-rs-ffi && cargo build --release build: python setup.py build wheel: python setup.py sdist bdist_wheel

Makefile

slide-97
SLIDE 97

setup.py

setup( name='pyavro_rs', packages=find_packages(), include_package_data=True, package_data={ 'avro-rs-ffi': { 'include/avro_rs.h', ‘target/release/libavro_rs_ffi.so' }, }, zip_safe=False, setup_requires=['cffi'], install_requires=['cffi'], cmdclass={ 'bdist_wheel': bdist_wheel, } )

slide-98
SLIDE 98

setup.py

setup( name='pyavro_rs', packages=find_packages(), include_package_data=True, package_data={ 'avro-rs-ffi': { 'include/avro_rs.h', ‘target/release/libavro_rs_ffi.so' }, }, zip_safe=False, setup_requires=['cffi'], install_requires=['cffi'], cmdclass={ 'bdist_wheel': bdist_wheel, } )

slide-99
SLIDE 99

setup.py

setup( name='pyavro_rs', packages=find_packages(), include_package_data=True, package_data={ 'avro-rs-ffi': { 'include/avro_rs.h', ‘target/release/libavro_rs_ffi.so' }, }, zip_safe=False, setup_requires=['cffi'], install_requires=['cffi'], cmdclass={ 'bdist_wheel': bdist_wheel, } )

slide-100
SLIDE 100

setup( name='pyavro_rs', packages=find_packages(), include_package_data=True, package_data={ 'avro-rs-ffi': { 'include/avro_rs.h', ‘target/release/libavro_rs_ffi.so' }, }, zip_safe=False, setup_requires=['cffi'], install_requires=['cffi'], cmdclass={ 'bdist_wheel': bdist_wheel, } )

setup.py

slide-101
SLIDE 101
slide-102
SLIDE 102

milksnake is awesome!

slide-103
SLIDE 103

def build_native(spec): build = spec.add_external_build( cmd=['cargo', 'build', '--release'], path='./avro-rs-ffi' ) spec.add_cffi_module( module_path='pyavro_rs._lowlevel', dylib=lambda: build.find_dylib( 'avro_rs_ffi', in_path='target/release' ), header_filename=lambda: build.find_header( 'avro_rs.h', in_path=‘include' ), )

setup.py

slide-104
SLIDE 104

def build_native(spec): build = spec.add_external_build( cmd=['cargo', 'build', '--release'], path='./avro-rs-ffi' ) spec.add_cffi_module( module_path='pyavro_rs._lowlevel', dylib=lambda: build.find_dylib( 'avro_rs_ffi', in_path='target/release' ), header_filename=lambda: build.find_header( 'avro_rs.h', in_path=‘include' ), )

setup.py

slide-105
SLIDE 105

def build_native(spec): build = spec.add_external_build( cmd=['cargo', 'build', '--release'], path='./avro-rs-ffi' ) spec.add_cffi_module( module_path='pyavro_rs._lowlevel', dylib=lambda: build.find_dylib( 'avro_rs_ffi', in_path='target/release' ), header_filename=lambda: build.find_header( 'avro_rs.h', in_path=‘include' ), )

setup.py

slide-106
SLIDE 106

def build_native(spec): build = spec.add_external_build( cmd=['cargo', 'build', '--release'], path='./avro-rs-ffi' ) spec.add_cffi_module( module_path='pyavro_rs._lowlevel', dylib=lambda: build.find_dylib( 'avro_rs_ffi', in_path='target/release' ), header_filename=lambda: build.find_header( 'avro_rs.h', in_path='include' ), )

setup.py

slide-107
SLIDE 107

setup( name='pyavro_rs', packages=find_packages(), include_package_data=True, setup_requires=['milksnake'], install_requires=['milksnake'], milksnake_tasks=[build_native], )

setup.py

slide-108
SLIDE 108

setup( name='pyavro_rs', packages=find_packages(), include_package_data=True, setup_requires=['milksnake'], install_requires=['milksnake'], milksnake_tasks=[build_native], )

setup.py

slide-109
SLIDE 109

>> python setup.py build running build running build_py … Compiling avro-rs v0.4.1 Compiling avro-rs-ffi v0.0.1 Finished release [optimized] target(s) in 115.66 secs …

slide-110
SLIDE 110

>> python setup.py build running build running build_py … Compiling avro-rs v0.4.1 Compiling avro-rs-ffi v0.0.1 Finished release [optimized] target(s) in 115.66 secs … >> tree build/ build └── lib └── pyavro_rs ├── __init__.py ├── _lowlevel.py ├── _lowlevel__ffi.py └── _lowlevel__lib.so

slide-111
SLIDE 111

milksnake is awesome!

slide-112
SLIDE 112

Was it faster?

slide-113
SLIDE 113

time [s] 0.0 2.3 4.5 6.8 9.0 CPython - Write CPython - Read Pypy - Write Pypy - Read pyavro-rs

slide-114
SLIDE 114

time [s] 0.0 2.3 4.5 6.8 9.0 CPython - Write CPython - Read Pypy - Write Pypy - Read pyavro-rs fastavro

slide-115
SLIDE 115

time [s] 0.0 2.3 4.5 6.8 9.0 CPython - Write CPython - Read Pypy - Write Pypy - Read pyavro-rs fastavro avro

slide-116
SLIDE 116

time [s] 0.0 17.5 35.0 52.5 70.0 CPython - Write CPython - Read Pypy - Write Pypy - Read pyavro-rs fastavro avro

slide-117
SLIDE 117

time [s] 0.0 17.5 35.0 52.5 70.0 CPython - Write CPython - Read Pypy - Write Pypy - Read pyavro-rs fastavro avro avro-rs

slide-118
SLIDE 118
slide-119
SLIDE 119

time [s] 0.0 2.3 4.5 6.8 9.0 CPython - Write CPython - Read Pypy - Write Pypy - Read pyavro-rs avro-rs

slide-120
SLIDE 120

How to get away with it

slide-121
SLIDE 121

How to convince my colleagues?

slide-122
SLIDE 122

How to convince my colleagues?

  • Most Loved Language (78.9%)


StackOverflow Survey 2017

slide-123
SLIDE 123

How to convince my colleagues?

  • Most Loved Language (78.9%)


StackOverflow Survey 2017

  • compilation just works (cargo)
slide-124
SLIDE 124

How to convince my colleagues?

  • Most Loved Language (78.9%)


StackOverflow Survey 2017

  • compilation just works (cargo)
  • fast release cycle
slide-125
SLIDE 125

How to convince my colleagues?

  • Most Loved Language (78.9%)


StackOverflow Survey 2017

  • compilation just works (cargo)
  • fast release cycle
  • it's a ton of fun!
slide-126
SLIDE 126

How to convince my company?

slide-127
SLIDE 127

How to convince my company?

  • wheels = compile once
slide-128
SLIDE 128

How to convince my company?

  • wheels = compile once
  • FFI interoperability (Java, Objective-C, C++, etc)
slide-129
SLIDE 129

How to convince my company?

  • wheels = compile once
  • FFI interoperability (Java, Objective-C, C++, etc)
  • as fast as C = cheap to run
slide-130
SLIDE 130

How to convince my company?

  • wheels = compile once
  • FFI interoperability (Java, Objective-C, C++, etc)
  • as fast as C = cheap to run
  • safer than C = cheap to maintain
slide-131
SLIDE 131

How to convince my company?

  • wheels = compile once
  • FFI interoperability (Java, Objective-C, C++, etc)
  • as fast as C = cheap to run
  • safer than C = cheap to maintain
  • used in production (Yelp, Mozilla, Dropbox, etc)
slide-132
SLIDE 132

What to bring home

slide-133
SLIDE 133

Write instead of

slide-134
SLIDE 134

cbindgen is awesome!

slide-135
SLIDE 135

milksnake is awesome!

slide-136
SLIDE 136

Links

  • avro-rs
  • avro-rs-ffi
  • pyavro-rs
  • cbindgen
  • milksnake
slide-137
SLIDE 137
slide-138
SLIDE 138