T h e B l a c k M a g i c o f P y t h o n Wh e - - PowerPoint PPT Presentation

SLIDE 1

T h e B l a c k M a g i c

• f

P y t h

• n

Wh e e l s

E l a n a H a s h ma n P y t h

• n

P a c k a g i n g A u t h

• r

i t y P y C

• n

U S 2 1 9 – C l e v e l a n d , O H @ e h a s h d n

SLIDE 2

@ehashdn :: #pycon2019

Wh e e l s / B l a c k M a g i c F A Q

Q : B u t I ’ m n

• t

a w i t c h ? ! A : S

• m

e t i m e s t h e g r e a t e r g

• d

r e q u i r e s a l i t t l e s a c r i f i c e .

SLIDE 3

@ehashdn :: #pycon2019

SLIDE 4

@ehashdn :: #pycon2019

T

• p

i c s

 P

y t h

• n

p a c k a g i n g a n d d i s t r i b u t i

• n

 E

L F ( E x e c u t a b l e a n d L i n k a b l e F

• r

ma t ) f i l e s

 D

y n a mi c l i n k i n g

 A

B I s ( A p p l i c a t i

• n

B i n a r y I n t e r f a c e s ) a n d s y mb

• l

v e r s i

• n

i n g

SLIDE 5

@ehashdn :: #pycon2019

O u t l i n e

 A

b r i e f h i s t

• r

y

• f

P y t h

• n

p a c k a g i n g a n d d i s t r i b u t i

• n

 A

n

• v

e r v i e w

• f

t h e w h e e l

 Wh

y w e n e e d n a t i v e e x t e n s i

• n

s

 H

• w

d

• n

a t i v e e x t e n s i

• n

s e v e n w

• r

k , r e a l l y ?

– Wh

a t a r e manylinux a n d auditwheel f

• r

?

 H

• w

y

• u

c a n g e t i n v

• l

v e d

SLIDE 6

@ehashdn :: #pycon2019

A B r i e f H i s t

• r

y

• f

P y t h

• n

P a c k a g i n g : E g g s

 O

r g a n i c a l l y a d

• p

t e d ( n

• g

u i d i n g P E P )

 N

• s

t a n d a r d ma n y i n c

• mp

a t i b l e i mp l e me n t a t i

• n

s

•  D

e s i g n e d t

• b

e d i r e c t l y i mp

• r

t a b l e , c

• u

l d i n c l u d e c

• mp

i l e d P y t h

• n

( .pyc f i l e s )

SLIDE 7

@ehashdn :: #pycon2019

P y t h

• n

P a c k a g i n g R e i n v e n t e d : T h e Wh e e l

 A

d

• p

t e d v i a P E P 4 2 7

 F

• l

l

• w

s t h e P E P 3 7 6 s t a n d a r d f

• r

d i s t r i b u t i

• n

s a n d P E P 4 2 6 s t a n d a r d f

• r

p a c k a g e me t a d a t a

 D

e s i g n e d f

• r

d i s t r i b u t i

• n

, c a n n

• t

i n c l u d e .pyc f i l e s ( b u t ma y i n c l u d e

• t

h e r p r e

• c
• mp

i l e d r e s

• u

r c e s )

SLIDE 8

@ehashdn :: #pycon2019

Wh e e l s “ ma k e i t e a s i e r t

• r
• l

l

• u

t ” P y t h

• n

 P

u r e w h e e l s

– O

n l y c

• n

t a i n P y t h

• n

c

• d

e

– M

a y t a r g e t a s p e c i f i c v e r s i

• n
• f

P y t h

• n

 U

n i v e r s a l w h e e l s

– P

y t h

• n

2 / 3 c

• mp

a t i b l e p u r e w h e e l s

E x t e n s i

• n

w h e e l s

? ? ? pip install wheel python setup.py bdist_wheel pip install wheel python setup.py bdist_wheel

SLIDE 9

@ehashdn :: #pycon2019

Wh e e l s “ ma k e i t e a s i e r t

• r
• l

l

• u

t ” P y t h

• n

 P

u r e w h e e l s

– O

n l y c

• n

t a i n P y t h

• n

c

• d

e

– M

a y t a r g e t a s p e c i f i c v e r s i

• n
• f

P y t h

• n

 U

n i v e r s a l w h e e l s

– P

y t h

• n

2 / 3 c

• mp

a t i b l e p u r e w h e e l s

 E

x t e n s i

• n

w h e e l s

– ?

? ?

SLIDE 10

@ehashdn :: #pycon2019

E x t e n s i

• n

s w i t h

• u

t b i n a r y d i s t r i b u t i

• n

s

$ pip install cryptography # source-only download ... c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory #include <Python.h> ^~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 $ sudo apt install python-dev # get Python.h

SLIDE 11

@ehashdn :: #pycon2019

E x t e n s i

• n

s w i t h

• u

t b i n a r y d i s t r i b u t i

• n

s

$ pip install cryptography ... c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory #include <ffi.h> ^~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 $ sudo apt install libffi-dev # get ffi.h

SLIDE 12

@ehashdn :: #pycon2019

E x t e n s i

• n

s w i t h

• u

t b i n a r y d i s t r i b u t i

• n

s

$ pip install cryptography ... build/temp.linux-x86_64-2.7/_openssl.c:498:10: fatal error: openssl/opensslv.h: No such file or directory #include <openssl/opensslv.h> ^~~~~~~~~~~~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 $ sudo apt install libssl-dev # get opensslv.h

SLIDE 13

@ehashdn :: #pycon2019

E x t e n s i

• n

s w i t h

• u

t b i n a r y d i s t r i b u t i

• n

s

$ time pip install cryptography Successfully installed asn1crypto-0.24.0 cffi- 1.11.5 cryptography-2.3.1 enum34-1.1.6 idna-2.7 ipaddress-1.0.22 pycparser-2.19 six-1.11.0 real 0m16.369s user 0m15.823s sys 0m0.627s

SLIDE 14

@ehashdn :: #pycon2019

E x t e n s i

• n

s w i t h b i n a r y d i s t r i b u t i

• n

s

$ time pip install cryptography # prebuilt binary Successfully installed asn1crypto-0.24.0 cffi- 1.11.5 cryptography-2.3.1 enum34-1.1.6 idna-2.7 ipaddress-1.0.22 pycparser-2.19 six-1.11.0 real 0m1.088s user 0m0.980s sys 0m0.108s

SLIDE 15

@ehashdn :: #pycon2019

Wh a t s

• r

t

• f

b l a c k ma g i c i s t h i s ? ✨🔯

SLIDE 16

@ehashdn :: #pycon2019

E x t e n s i

• n

Wh e e l s a r e s a f e t

• pip install!

 I

n s t a l l i n g P y t h

• n

n a t i v e e x t e n s i

• n

s w i t h

• u

t w h e e l s i s p a i n f u l

 C

• n

d a w a s d e v e l

• p

e d t

• a

d d r e s s t h i s g a p : w h y n

• t

u s e t h a t ?

– L

i k e e g g s , C

• n

d a w a s n

• t

a d

• p

t e d b y a P E P

– C

• n

d a p a c k a g e s a r e n

• t

P y t h

• n
• s

p e c i f i c , n

• t

s u p p

• r

t e d b y P y P I

– C

• n

d a p a c k a g e s a r e n

• t

c

• mp

a t i b l e w i t h n

• n
• C
• n

d a e n v i r

• n

me n t s

 Wh

e e l s a r e c

• mp

a t i b l e w i t h t h e e n t i r e P y t h

• n

e c

• s

y s t e m

SLIDE 17

@ehashdn :: #pycon2019

Wh a t i s a P y t h

• n

( N a t i v e ) E x t e n s i

• n

?

 N

a t i v e : t h e c

• d

e w a s c

• mp

i l e d s p e c i f i c a l l y f

• r

my

• p

e r a t i n g s y s t e m

 E

x t e n s i

• n

: t h i s l i b r a r y e x t e n d s P y t h

• n

’ s f u n c t i

• n

a l i t y w i t h n

• n
• P

y t h

• n

c

• d

e

 E

x a mp l e

: cryptography

– I

t u s e s C F F I : t h e “ C F

• r

e i g n F u n c t i

• n

I n t e r f a c e ” f

• r

P y t h

• n

SLIDE 18

@ehashdn :: #pycon2019

P y t h

• n

c

• d

e i s n

• t

j u s t P y t h

• n

. F

• r

P y t h

• n

t

• h

a r n e s s i t s f u l l p

• t

e n t i a l , i t mu s t b e a b l e t

• d

e p e n d

• n

C l i b r a r i e s .

SLIDE 19

@ehashdn :: #pycon2019

C i s a c

• mp

i l e d l a n g u a g e

// hello.c #include<stdio.h> int main(void) { puts("hello world"); }

# a.out (hexadecimal) 0000000 7f45 4c46 0201 0100 0000008 0000 0000 0000 0000 0000010 0300 3e00 0100 0000 0000018 5005 0000 0000 0000 0000020 4000 0000 0000 0000 ... gcc ( c

• mp

i l e r ) gcc hello.c hexdump a.out

E L F F i l e

SLIDE 20

@ehashdn :: #pycon2019

H e x e s a n d E L F s

$ readelf -a a.out ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: Advanced Micro Devices X86-64 ...

SLIDE 21

@ehashdn :: #pycon2019

H e x e s a n d E L F s

$ readelf -a a.out

... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align INTERP 0x0000000000000238 0x0000000000000238 0x0000000000000238 0x000000000000001c 0x000000000000001c R 0x1 [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

E L F i n t e r p r e t e r

SLIDE 22

@ehashdn :: #pycon2019

H e x e s a n d E L F s

$ readelf -a a.out ... Relocation section '.rela.plt' at offset 0x4d0 contains 1 entry: Offset Info Type 000000200fd0 000200000007 R_X86_64_JUMP_SLO

• Sym. Value Sym. Name + Addend

0000000000000000 puts@GLIBC_2.2.5 + 0

S y mb

• l

V e r s i

• n

SLIDE 23

@ehashdn :: #pycon2019

H e x e s a n d E L F s

$ readelf -a a.out ...

Version needs section '.gnu.version_r' contains 1 entry: Addr: 0x00000000000003f0 Offset: 0x0003f0 Link: 6 (.dynstr) 000000: Version: 1 File: libc.so.6 Cnt: 1 0x0010: Name: GLIBC_2.2.5 Flags: none Version: 2

SLIDE 24

@ehashdn :: #pycon2019

S y mb

• l

V e r s i

• n

s i n A c t i

• n

hello.c #include<stdio.h> ... puts("hello world"); a.out .rela.plt Symbol Name puts@GLIBC_2.2.5 .gnu.version_r File Symbol Version libc.so.6 GLIBC_2.2.5 libc.so.6 .gnu.version_d Symbol Versions Available: GLIBC_2.2.5 GLIBC_2.2.6 GLIBC_2.3 ... GLIBC_2.27 .dynsym Type Name FUNC puts@GLIBC_2.2.5

SLIDE 25

@ehashdn :: #pycon2019

Wh a t h a p p e n s w h e n w e r u n t h i s ?

 O

S p a r s e s “ ma g i c E L F ” t e x t

 O

S i n v

• k

e s t h e E L F i n t e r p r e t e r s p e c i f i e d b y t h e b i n a r y

 E

L F i n t e r p r e t e r l

• a

d s a n y r e q u i r e d f i l e s w i t h v a l i d v e r s i

• n

s

 E

L F i n t e r p r e t e r r e l

• c

a t e s t h e p r

• g

r a m c

• d

e a n d d e p e n d e n c i e s i n me mo r y s

• t

h a t i t c a n r u n

 T

h i s i s c a l l e d d y n a m i c l i n k i n g

SLIDE 26

@ehashdn :: #pycon2019

H

• w

t

• g

e t C i n t

• 🐎️

? ?

 O

l d w a y : ma k e u s e r s c

• mp

i l e f r

• m

s

• u

r c e

– O

b t a i n i n g d e p e n d e n c i e s i s t h e u s e r ’ s p r

• b

l e m

– C

• mp

i l e a g a i n s t s y s t e m l i b r a r y v e r s i

• n

a t P y t h

• n

i n s t a l l t i me

 N

e w w a y : u s e r s i n s t a l l p r e

• b

u i l t P y t h

• n

w h e e l s

– B

u n d l e p r e

• c
• mp

i l e d b i n a r y d e p e n d e n c i e s i n s i d e a P y t h

• n

w h e e l ✨

SLIDE 27

@ehashdn :: #pycon2019

H

• w

t

• g

e t C i n t

• 🐎️

? ?

 T

h e

• l

d w a y s h a v e ma n y p r

• b

l e ms

– S

l

• w

( c

• mp

i l i n g f r

• m

s

• u

r c e )

– V

e r s i

• n

mi s ma t c h e s

– R

e q u i r e s k n

• w

l e d g e

• f

s y s t e m p a c k a g e ma n a g e me n t

 P

y t h

• n

w h e e l s s

• l

v e t h i s !

– D

e p e n d e n c i e s p r

• v

i d e d a r e t h e r i g h t v e r s i

• n

s a n d p r e c

• mp

i l e d

– Wh

e e l s a r e P y t h

• n
• n

a t i v e : j u s t pip install t h e m

SLIDE 28

@ehashdn :: #pycon2019

B u t h

• w

c a n w e e n s u r e t h e p r e

• c
• mp

i l e d b i n a r i e s a r e c

• mp

a t i b l e w i t h m y s y s t e m ?

SLIDE 29

@ehashdn :: #pycon2019

Q : H

• w

c a n w e s h i p c

• mp

i l e d P y t h

• n

e x t e n s i

• n

s c

• mp

a t i b l e w i t h a s ma n y s y s t e ms a s p

• s

s i b l e ? A : S y mb

• l

v e r s i

• n

i n g ( manylinux) a n d d e p e n d e n c y b u n d l i n g ( auditwheel) .

SLIDE 30

@ehashdn :: #pycon2019

Wh a t a r e manylinux a n d auditwheel?

 P

E P s 5 1 3 a n d 5 7 1 d e f i n e a s e t

• f

p e r mi t t e d l i b r a r i e s a n d t h e i r s y mb

• l

v e r s i

• n

s f

• r

L i n u x s y s t e ms

– “

M a n y ” L i n u x s y s t e ms a r e c

• mp

a t i b l e w i t h t h i s s t a n d a r d

 manylinux

i s b

• t

h t h e n a me

• f

t h e p

• l

i c y a n d a D

• c

k e r i ma g e

– manylinux1

( P E P 5 1 3 ) : C e n t O S 5 , i 3 8 6 / a md 6 4 a r c h i t e c t u r e s

– manylinux2010

( P E P 5 7 1 ) : C e n t O S 6 , i 3 8 6 / a md 6 4 a r c h i t e c t u r e s

 auditwheel

i s a t

• l

t

• e

n f

• r

c e t h e s y mb

• l

p

• l

i c i e s

SLIDE 31

@ehashdn :: #pycon2019

Wh a t i s auditwheel?

 auditwheel

u s e s d a r k ma g i c t

• v

e n d

• r

e x t e r n a l b i n a r y d e p e n d e n c i e s i n t

• y
• u

r w h e e l ️👼️

 E

mp

• w

e r s d e v e l

• p

e r s t

• b

u i l d manylinux w h e e l s w i t h

• u

t h a v i n g t

• c

h a n g e t h e i r b u i l d p r

• c

e s s e s

– J

u s t b u i l d a n d i n s t a l l d e p e n d e n c i e s i n t h e manylinux i ma g e

– auditwheel repair

w i l l b u n d l e d e p e n d e n c i e s i n t

• a

c

• mp

l i a n t w h e e l

SLIDE 32

@ehashdn :: #pycon2019

Wh e e l B u i l d e r ’ s P i p e l i n e f

• r

L i n u x

❷ ❸ ❶

❶

A d d y

• u

r c

• d

e , d e p e n d e n c i e s t

• t

h e manylinux D

• c

k e r i ma g e a n d b u i l d a g a i n s t y

• u

r s u p p

• r

t e d P y t h

• n

v e r s i

• n

s / a r c h i t e c t u r e s

❷

R e p a i r a n d i n s p e c t t h e b u i l t w h e e l w i t h auditwheel f

• r

c

• mp

l i a n c e

❸

U p l

• a

d t

• P

y P I !

P y t h

• n

E x t e n s i

• n

manylinux c

• n

t a i n e r auditwheel r e p a i r + i n s p e c t i

• n

P y P I

SLIDE 33

@ehashdn :: #pycon2019

Wa n t i n

• n

t h e ma g i c ?

 H

e l p b u i l d w h e e l s !

– F

e e d b a c k e n t h u s i a s t i c a l l y w e l c

• me

d

✨

 p

y t h

• n

w h e e l s . c

• m

– S

e e w h a t p a c k a g e s a l r e a d y b u i l d w h e e l s

– F

i n d e x a mp l e s f

• r

h

• w

t

• b

u i l d y

• u

r s ( i n c l u d i n g Wi n d

• w

s , O S X )

 g

i t h u b . c

• m/

p y p a / p y t h

• n
• ma

n y l i n u x

• d

e mo

– S

i mp l e d e mo t

• l

e a r n L i n u x w h e e l b u i l d i n g

SLIDE 34

@ehashdn :: #pycon2019

auditwheel n e e d s a n e w ma i n t a i n e r

 I

’ m s t e p p i n g d

• w

n a f t e r t h r e e y e a r s !

– h

a s h ma n . c a / l e a v i n g

• p

y p a

 T

h e manylinux D

• c

k e r i ma g e s a n d n e w manylinux2014 s p e c n e e d s

• me

l

• v

e , t

SLIDE 35

@ehashdn :: #pycon2019

Q u e s t i

• n

s ?

Q u e s t i

• n

s ?

SLIDE 36

@ehashdn :: #pycon2019

T h a n k s t

• :

R e d H a t N e l s

• n

E l h a g e , P a u l K e h r e r , D

• n

a l d S t u f f t

T a l k r e s

• u

r c e s : h t t p s : / / h a s h ma n . c a / p y c

• n
• 2

1 9

SLIDE 37

@ehashdn :: #pycon2019

I ma g e L i c e n s e I n f

• r

ma t i

• n

 T

r e e C a t S i l h

• u

e t t e S u n s e t : P u b l i c D

• ma

i n ( C C ) @ b e s s h a mi t i h t t p s : / / p l i x s . c

• m/

p h

• t
• /

3 2 9 7 / t r e e

• c

a t

• s

i l h

• u

e t t e

• s

u n s e t

 H

a p p y H a l l

• w

e e n ! ( C

• s

t u me D

• g

) : P u b l i c D

• ma

i n ( C C ) @ mi l k y f a c t

• r

y h t t p s : / / f l i c . k r / p / A r W1 N 9