P YTHON P ACKAGING Where we are Where we’re headed gg.gg/pycon-in-2019 @pradyunsg
W HO ? Pradyun Gedam @pradyunsg pradyunsg.me Member of Python Packaging Authority Maintainer of pip, virtualenv, packaging and more PSF Fellow College student!
W HERE W E A RE An Overview of Tooling in the Python Packaging ecosystem
N OT H OW WE GOT HERE https:/ /www.youtube.com/watch?v=AQsZsgJ30AE
W HAT ’ S P ACKAGING ?
Medium for Moving Packages Package Package Thing Thing
S OFTWARE D ISTRIBUTION
"Moving" Software Distribution Distribution Source Code Working Software
S OFTWARE D ISTRIBUTION A N U NSOLVED P ROBLEM https:/ /youtu.be/IVzjVqr_Bzs?t=430
"Moving" Software Distribution Distribution Source Code Working Software
Publisher User "Moving" Software Distribution Distribution Source Code Working Software
Publisher Source Code
Publisher Distribution "build" Source Code
Publisher - Build a Distribution from Source Code Distribution - Build Environment - Build Mechanisms "build" Source Code
Publisher "Moving" Software Distribution "build" Source Code
Publisher "Moving" Software "upload" Distribution "build" Source Code
Publisher "Moving" Software "upload" Distribution - Upload the Distribution "build" - Upload Mechanisms Source Code
User "Moving" Software
User "Moving" Software Distribution
User "Moving" Software "download" Distribution
User "Moving" Software "download" Distribution - Download the Distribution - Download Mechanisms - Choosing what to download
User "Moving" Software "download" Distribution Working Software
User "Moving" Software "download" Distribution "install" Working Software
User "Moving" Software "download" Distribution - Install from the Distribution - Install Environment "install" - Install Mechanisms Working Software
Publisher User "Moving" Software "upload" "download" Distribution Distribution "build" "install" Source Code Working Software
Publisher User "Moving" Software "upload" "download" Distribution Distribution
Publisher User Distribution Distribution "build" "install" Source Code Working Software
D EALING WITH E NVIRONMENTS • The Packaging Gradient https:/ /sedimental.org/the_packaging_gradient.html https:/ /www.youtube.com/watch?v=iLVNWfPWAC8 • Python Packaging Overview https:/ /packaging.python.org/overview/
P Y PA T OOLING Tools built by volunteers
Built on standards (mostly) https:/ /packaging.python.org/speci fi cations/
Package Index Interfaces Publisher User Python Package Index "upload" "download" Distribution Distribution
Package Distribution Metadata Publisher User Distribution Distribution "build" "install" Source Code Working Software
W E WENT TOO FAR ONCE . • PEP 426 - Metadata 2.0 • Was trying to solve all the problems with Python’s metadata. Never became a reality. • Incremental Improvements • Our current, softer approach — workable approach with volunteers.
Publisher - Usually setuptools - PEP 517 and PEP 518 Distribution - "modern" source distributions build backend - pyproject.toml Source Code
Publisher PyPI twine Distribution
User PyPI pip Distribution virtualenv pip venv Working Software
It’s not that straightforward.
Publisher User PyPI twine pip Distribution Distribution virtualenv setuptools pip venv Source Code Working Software
User pip via PyPI pipenv Distribution pipenv Working Software
NON -P Y PA T OOLING Tools built by companies or different volunteers.
Environments are dif fi cult.
Publisher - Simpli fi es the "build" experience Distribution - Metadata in pyproject.toml fl it Source Code
User Distribution pex Working Software
User Distribution pipx Working Software
Publisher User PyPI poetry poetry Distribution Distribution poetry poetry Source Code Working Software
Publisher User Conda Channels conda conda Conda Packages Conda Packages conda conda Source Code Working Software
"… all of this stuff is really hard and there’s a lot of knowledge scattered around in people’s heads that could really do with being captured somewhere" — P AUL M OORE
W HERE W E ’ RE H EADED Future Improvements in Python Packaging
I DON ’ T HAVE A TIME MACHINE .
C ORE F UNCTIONALITY I MPROVEMENTS ongoing • securing PyPI — The Update Framework (TUF) • newer manylinux standards — manylinux2014 • yanking packages from PyPI • better licensing metadata
C ORE F UNCTIONALITY I MPROVEMENTS soon? • structured lock fi les in pip • package preview functionality from PyPI • security noti fi cations for vulnerable packages • pip dependency resolver
B ETTER U SER E XPERIENCE • better defaults in pip • feature fl ags on PyPI • interoperability testing • professional UX review
R EDUCING T ECHNICAL D EBT • virtualenv rewrite • making the transition from distutils -> setuptools • pip build logic refactor • pip "working schemes" cleanup • change the right environments
M OAR S TANDARDISATION • editable installations • better, more powerful, extras • less ambiguous licensing • interoperability with other package managers
H OW DO WE GET THERE ? • Volunteer! • Tell us how you deal with "interesting" issues. • Fund one of the targeted funded projects.
Thank you! \o/
Recommend
More recommend
