Boosting Python with Rust The case of Mercurial FOSDEM 2020 Raphal - - PowerPoint PPT Presentation

▶
boosting python with rust
SMART_READER_LITE
LIVE PREVIEW

Boosting Python with Rust The case of Mercurial FOSDEM 2020 Raphal - - PowerPoint PPT Presentation

Feb 01, 2024 318 likes •592 views

Boosting Python with Rust The case of Mercurial FOSDEM 2020 Raphal Goms @ Mercurial Same generation as Git Written in Python (200k lines) Boosted by C extensions (45k lines) Handles huge repos (millions of files and/or

slide-1
SLIDE 1

Boosting Python with Rust

The case of Mercurial

Raphaël Gomès @

FOSDEM 2020

slide-2
SLIDE 2

1 Feb 2020 Boosting Python with Rust 2

Mercurial

  • Same generation as Git
  • Written in Python (200k lines)
  • Boosted by C extensions (45k

lines)

  • Handles huge repos (millions of

files and/or revisions)

  • Very powerful extension system
slide-3
SLIDE 3

1 Feb 2020 Boosting Python with Rust

Why Rust?

slide-4
SLIDE 4

1 Feb 2020 Boosting Python with Rust

Rust

  • Low-level language
  • Powerful type system
  • No garbage collector
  • Compile-time memory safety
  • Simple(r) parallelism
slide-5
SLIDE 5

1 Feb 2020 Boosting Python with Rust

Maintainability

  • Better signal/noise ratio
  • Better compile-time guarantees
  • Standardized and modern

tooling

  • "Safe" by default (unsafe

blocks)

Compared to C

slide-6
SLIDE 6

1 Feb 2020 Boosting Python with Rust

Performance

  • Comparable to C for

sequential code

  • Parallel code is much

simpler to write and maintain

  • Allows for optimizations

impossible for C compilers

slide-7
SLIDE 7

1 Feb 2020 Boosting Python with Rust

Performance

“hg status” experiment by Valentin Gatien-Baron hg Rust hg status 2.4s 50ms status -u 2.4s 39ms status -mard 400ms 14ms

slide-8
SLIDE 8

1 Feb 2020 Boosting Python with Rust

rust-cpython

  • A low-level crate for the CPython

ABI

  • A high-level crate to interact

with Python:

– Expose a Rust module to Python – Create Python function and

classes

– Execute Python from Rust

slide-9
SLIDE 9

1 Feb 2020 Boosting Python with Rust

Structure

CPython

Pure Python

C Extensions

hg-cpython

hg-core

slide-10
SLIDE 10

1 Feb 2020 Boosting Python with Rust

<Alphare> So, I finished rewriting this function in Rust <Alphare> The bad news is: it’s twice as slow

A slow start

slide-11
SLIDE 11

1 Feb 2020 Boosting Python with Rust

Friction with Python

  • Complex interface code
  • Exchanging data is costly
slide-12
SLIDE 12

1 Feb 2020 Boosting Python with Rust

  • stat of 100k files in Rust: 30ms
  • Giving the results to Python: 300ms

Friction with Python

slide-13
SLIDE 13

1 Feb 2020 Boosting Python with Rust

Possible solutions

  • Exchange less data
  • Do more in Rust
  • Communicate with C directly
slide-14
SLIDE 14

1 Feb 2020 Boosting Python with Rust

Talk to C directly

CPython

Pure Python

C Extensions

hg-cpython

hg-core

slide-15
SLIDE 15

1 Feb 2020 Boosting Python with Rust

???

CPython

Pure Python

C Extensions

hg-cpython

hg-core

Talk to C directly

slide-16
SLIDE 16

1 Feb 2020 Boosting Python with Rust

Capsules

  • PyCapsule: Python object that

encapsulates function pointers

  • Can be defined in a module,

used in another

  • Exactly made to share a C API

between extensions

slide-17
SLIDE 17

1 Feb 2020 Boosting Python with Rust

Capsules

slide-18
SLIDE 18

1 Feb 2020 Boosting Python with Rust

Missing features

  • PySet
  • Simple support of PyCapsule
  • Inheritence for classes

written in Rust

  • Properties and __setattr__
  • Iterators on Rust collections
slide-19
SLIDE 19

1 Feb 2020 Boosting Python with Rust

A Python iterator in Rust

  • Should behave exactly as a

Python iterator

  • Tell the Rust compiler that it

really has to let go

  • Handle sharing references

between the two languages

slide-20
SLIDE 20

1 Feb 2020 Boosting Python with Rust

Upstream work

  • PySet
  • Simple PyCapsule support
  • Properties
  • Iterators on Rust

collections

slide-21
SLIDE 21

1 Feb 2020 Boosting Python with Rust

Performance

“hg status” experiment by Valentin Gatien-Baron hg Rust hg status 2.4s 50ms status -u 2.4s 39ms status -mard 400ms 14ms

slide-22
SLIDE 22

1 Feb 2020 Boosting Python with Rust

Current performance (pathological case, 100k files)

Python + C Python + Rust

status 6.23s 1.59s status -mard 1.46s 840ms diff 1.5s 880ms

slide-23
SLIDE 23

1 Feb 2020 Boosting Python with Rust

Current performance (more realistic case, 260k files) Python + C Python + Rust

status 2.9s 2.0s status -mard 1.7s 1.0s diff 1.9s 1.2s

slide-24
SLIDE 24

1 Feb 2020 Boosting Python with Rust

#TODO

  • Do more things in parallel
  • Better conditional execution
  • Rethink the order of execution
  • Fewer exchanges between Python

and Rust

  • Fewer allocations, etc.

Contourner Python ???

slide-25
SLIDE 25

1 Feb 2020 Boosting Python with Rust

#TODO

  • Do more things in parallel
  • Better conditional execution
  • Rethink the order of execution
  • Fewer exchanges between Python

and Rust

  • Fewer allocations, etc.
  • ...not start Python ??hon ???
slide-26
SLIDE 26

1 Feb 2020 Boosting Python with Rust

A renewed appreciation for Python

  • Code is very easy to

understand

  • You get something that works

very quickly

  • Allows for experimentation
  • It is a lot faster than Rust

code you are not done writing

slide-27
SLIDE 27

1 Feb 2020 Boosting Python with Rust

Thank you!

FOSDEM 2020

Raphaël Gomès @