T h e B l a c k M a g i c o f P y t h o n Wh e - PowerPoint PPT Presentation

T h e B l a c k M a g i c o f P y t h o n Wh e e l s E l a n a H a s h ma n P y t h o n P a c k a g i n g A u t h o r i t y P y C o n U S 2 0 1 9 – C l e v e l a n d , O H @ e h a s h d n

Wh e e l s / B l a c k M a g i c F A Q Q : B u t I ’ m n o t a w i t c h ? ! A : S o m e t i m e s t h e g r e a t e r g o o d r e q u i r e s a l i t t l e s a c r i f i c e . @ehashdn :: #pycon2019

@ehashdn :: #pycon2019

T o p i c s  P y t h o n p a c k a g i n g a n d d i s t r i b u t i o n  E L F ( E x e c u t a b l e a n d L i n k a b l e F o r ma t ) f i l e s  D y n a mi c l i n k i n g  A B I s ( A p p l i c a t i o n B i n a r y I n t e r f a c e s ) a n d s y mb o l v e r s i o n i n g @ehashdn :: #pycon2019

O u t l i n e  A b r i e f h i s t o r y o f P y t h o n p a c k a g i n g a n d d i s t r i b u t i o n  A n o v e r v i e w o f t h e w h e e l  Wh y w e n e e d n a t i v e e x t e n s i o n s  H o w d o n a t i v e e x t e n s i o n s e v e n w o r k , r e a l l y ? – Wh manylinux auditwheel a t a r e a n d f o r ?  H o w y o u c a n g e t i n v o l v e d @ehashdn :: #pycon2019

A B r i e f H i s t o r y o f P y t h o n P a c k a g i n g : E g g s  O r g a n i c a l l y a d o p t e d ( n o g u i d i n g P E P )  N � o s t a n d a r d ma n y i n c o mp a t i b l e i mp l e me n t a t i o n s  D e s i g n e d t o b e d i r e c t l y i mp o r t a b l e , c o u l d i n c l u d e .pyc c o mp i l e d P y t h o n ( f i l e s ) @ehashdn :: #pycon2019

P y t h o n P a c k a g i n g R e i n v e n t e d : T h e Wh e e l  A d o p t e d v i a P E P 4 2 7  F o l l o w s t h e P E P 3 7 6 s t a n d a r d f o r d i s t r i b u t i o n s a n d P E P 4 2 6 s t a n d a r d f o r p a c k a g e me t a d a t a  D .pyc e s i g n e d f o r d i s t r i b u t i o n , c a n n o t i n c l u d e f i l e s ( b u t ma y i n c l u d e o t h e r p r e - c o mp i l e d r e s o u r c e s ) @ehashdn :: #pycon2019

Wh e e l s “ ma k e i t e a s i e r t o r o l l o u t ” P y t h o n  P u r e w h e e l s – O n l y c o n t a i n P y t h o n c o d e – M a y t a r g e t a s p e c i f i c v e r s i o n o f P y t h o n  U n i v e r s a l w h e e l s – P y t h o n 2 / 3 c o mp a t i b l e p u r e w h e e l s E x t e n s i o n w h e e l s pip install wheel pip install wheel ? ? ? python setup.py bdist_wheel python setup.py bdist_wheel @ehashdn :: #pycon2019

Wh e e l s “ ma k e i t e a s i e r t o r o l l o u t ” P y t h o n  P u r e w h e e l s – O n l y c o n t a i n P y t h o n c o d e – M a y t a r g e t a s p e c i f i c v e r s i o n o f P y t h o n  U n i v e r s a l w h e e l s – P y t h o n 2 / 3 c o mp a t i b l e p u r e w h e e l s  E x t e n s i o n w h e e l s – ? ? ? @ehashdn :: #pycon2019

E x t e n s i o n s w i t h o u t b i n a r y d i s t r i b u t i o n s $ pip install cryptography # source-only download ... c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory #include <Python.h> ^~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 $ sudo apt install python-dev # get Python.h @ehashdn :: #pycon2019

E x t e n s i o n s w i t h o u t b i n a r y d i s t r i b u t i o n s $ pip install cryptography ... c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory #include <ffi.h> ^~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 $ sudo apt install libffi-dev # get ffi.h @ehashdn :: #pycon2019

E x t e n s i o n s w i t h o u t b i n a r y d i s t r i b u t i o n s $ pip install cryptography ... build/temp.linux-x86_64-2.7/_openssl.c:498:10: fatal error: openssl/opensslv.h: No such file or directory #include <openssl/opensslv.h> ^~~~~~~~~~~~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 $ sudo apt install libssl-dev # get opensslv.h @ehashdn :: #pycon2019

E x t e n s i o n s w i t h o u t b i n a r y d i s t r i b u t i o n s $ time pip install cryptography Successfully installed asn1crypto-0.24.0 cffi- 1.11.5 cryptography-2.3.1 enum34-1.1.6 idna-2.7 ipaddress-1.0.22 pycparser-2.19 six-1.11.0 real 0m16.369s user 0m15.823s sys 0m0.627s @ehashdn :: #pycon2019

E x t e n s i o n s w i t h b i n a r y d i s t r i b u t i o n s $ time pip install cryptography # prebuilt binary Successfully installed asn1crypto-0.24.0 cffi- 1.11.5 cryptography-2.3.1 enum34-1.1.6 idna-2.7 ipaddress-1.0.22 pycparser-2.19 six-1.11.0 real 0m1.088s user 0m0.980s sys 0m0.108s @ehashdn :: #pycon2019

✨🔯 Wh a t s o r t o f b l a c k ma g i c i s t h i s ? @ehashdn :: #pycon2019

pip install ! E x t e n s i o n Wh e e l s a r e s a f e t o  I n s t a l l i n g P y t h o n n a t i v e e x t e n s i o n s w i t h o u t w h e e l s i s p a i n f u l  C o n d a w a s d e v e l o p e d t o a d d r e s s t h i s g a p : w h y n o t u s e t h a t ? – L i k e e g g s , C o n d a w a s n o t a d o p t e d b y a P E P – C o n d a p a c k a g e s a r e n o t P y t h o n - s p e c i f i c , n o t s u p p o r t e d b y P y P I – C o n d a p a c k a g e s a r e n o t c o mp a t i b l e w i t h n o n - C o n d a e n v i r o n me n t s  Wh e e l s a r e c o mp a t i b l e w i t h t h e e n t i r e P y t h o n e c o s y s t e m @ehashdn :: #pycon2019

Wh a t i s a P y t h o n ( N a t i v e ) E x t e n s i o n ?  N a t i v e : t h e c o d e w a s c o mp i l e d s p e c i f i c a l l y f o r my o p e r a t i n g s y s t e m  E x t e n s i o n : t h i s l i b r a r y e x t e n d s P y t h o n ’ s f u n c t i o n a l i t y w i t h n o n - P y t h o n c o d e  E x a mp l e cryptography : – I t u s e s C F F I : t h e “ C F o r e i g n F u n c t i o n I n t e r f a c e ” f o r P y t h o n @ehashdn :: #pycon2019

P y t h o n c o d e i s n o t j u s t P y t h o n . F o r P y t h o n t o h a r n e s s i t s f u l l p o t e n t i a l , i t mu s t b e a b l e t o d e p e n d o n C l i b r a r i e s . @ehashdn :: #pycon2019

C i s a c o mp i l e d l a n g u a g e // hello.c # a.out (hexadecimal) 0000000 7f45 4c46 0201 0100 #include<stdio.h> 0000008 0000 0000 0000 0000 0000010 0300 3e00 0100 0000 int main( void ) { 0000018 5005 0000 0000 0000 puts( "hello 0000020 4000 0000 0000 0000 ... world" ); E L F F i l e } gcc hexdump a.out gcc hello.c ( c o mp i l e r ) @ehashdn :: #pycon2019

H e x e s a n d E L F s $ readelf -a a.out ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: Advanced Micro Devices X86-64 ... @ehashdn :: #pycon2019

H e x e s a n d E L F s $ readelf -a a.out ... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align INTERP 0x0000000000000238 0x0000000000000238 0x0000000000000238 0x000000000000001c 0x000000000000001c R 0x1 [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2] E L F i n t e r p r e t e r @ehashdn :: #pycon2019

H e x e s a n d E L F s $ readelf -a a.out ... Relocation section '.rela.plt' at offset 0x4d0 contains 1 entry: Offset Info Type 000000200fd0 000200000007 R_X86_64_JUMP_SLO Sym. Value Sym. Name + Addend 0000000000000000 puts@GLIBC_2.2.5 + 0 S y mb o l V e r s i o n @ehashdn :: #pycon2019