R ETHINKING P YTHON P ACKAGING A thought experiment @pradyunsg - - PowerPoint PPT Presentation

▶
r ethinking
SMART_READER_LITE
LIVE PREVIEW

R ETHINKING P YTHON P ACKAGING A thought experiment @pradyunsg - - PowerPoint PPT Presentation

Jan 16, 2024 452 likes •807 views

R ETHINKING P YTHON P ACKAGING A thought experiment @pradyunsg pradyunsg.me/talks W HO ? Pradyun Gedam @pradyunsg pradyunsg.me Member of Python Packaging Authority Maintainer of pip, virtualenv, packaging and more PSF

slide-1
SLIDE 1

pradyunsg.me/talks

A thought experiment

RETHINKING
 PYTHON PACKAGING

@pradyunsg

slide-2
SLIDE 2

WHO?

Pradyun Gedam


@pradyunsg
 pradyunsg.me Member of


Python Packaging Authority

Maintainer of


pip, virtualenv, packaging and more PSF Fellow College student!

slide-3
SLIDE 3

RETHINKING PYTHON PACKAGING

slide-4
SLIDE 4

Distribution

PyPI

Working? Software pip

User

Distribution Source Code setuptools twine

Publisher

pip virtualenv virtualenv

slide-5
SLIDE 5
slide-6
SLIDE 6

RETHINKING PYTHON PACKAGING

slide-7
SLIDE 7

Free to ignore existing tooling Best practices as defaults Try to not break everything Easier to understand, use and maintain

slide-8
SLIDE 8

CONSTRAINTS

  • No removal of functionality for:
  • infrastructure — PyPI
  • published packages
  • existing PyPA standards
slide-9
SLIDE 9

DISCLAIMERS

complete speculation
 (sort of)

  • ther folks’ ideas + my opinions

very little UX experience

slide-10
SLIDE 10

"THINGS"

slide-11
SLIDE 11

Source Tree

slide-12
SLIDE 12

Source Distribution Wheel Distribution

slide-13
SLIDE 13

Environments

Installed Packages Platform Details

slide-14
SLIDE 14

PyPI

slide-15
SLIDE 15

PEP 517’S MODEL

slide-16
SLIDE 16

PEP 517’S MODEL

slide-17
SLIDE 17

MODERN PACKAGING TOOLING’S MODEL

slide-18
SLIDE 18

MODERN PACKAGING TOOLING’S MODEL

  • Build Backend — like setuptools, flit etc
  • Produce distributions from source trees
  • Build Frontend — like pip
  • is user-facing — user workflows
  • manages environment
  • handles distributions
  • orchestrates build backends
slide-19
SLIDE 19

BACKEND OPERATIONS

  • Produce distributions from source trees
  • Handling of Python code
  • Handling of non-Python code / artefacts
  • Metadata generation
  • Properly place files into a distribution
slide-20
SLIDE 20

METADATA

  • setuptools allows for arbitrary logic, which can be used

for generating metadata

  • What we’d want
  • Static metadata, by default
  • Dynamic metadata, by opt-in
  • Keep metadata in pyproject.toml
  • Allow specifying a script for enhancing it
slide-21
SLIDE 21

HANDLING OF NON-PYTHON CODE / ARTEFACTS

  • setuptools invokes the compilers directly
  • complicated build process → complicated setup.py
  • What we’d want
  • Generate build instructions for *other* build tools
  • Invoke the build tools
slide-22
SLIDE 22

PROPERLY PLACE FILES INTO A DISTRIBUTION

  • Idempotent source distributions
  • unpack sdist + build sdist → same as initial
  • Build non-Python "stuff" only for wheels
slide-23
SLIDE 23

Build sdist Build wheel Source Tree Source Distribution Unpack Unpack Installed Package Wheel Distributions

pradyunsg.me/talks @pradyunsg

slide-24
SLIDE 24

BUILD SDIST

  • Load information from pyproject.toml
  • Generate metadata
  • Package relevant files into a .tar.gz file
slide-25
SLIDE 25

BUILD WHEEL

  • Load information from pyproject.toml
  • Build non-Python code into binaries
  • Generate metadata
  • Package relevant files into a .whl file
slide-26
SLIDE 26

FRONTEND OPERATIONS

  • Environment management
  • Dependency resolution (hah!)
  • Installation / Uninstallation / Upgrade
  • Handling distributions
  • Discovering and fetching distributions
  • Unpacking distributions correctly
  • Orchestrates build backends
  • Manage source tree → distribution
slide-27
SLIDE 27

HANDLING DISTRIBUTIONS

  • Discovering and fetching distributions
  • Index interaction
  • Networking — TLS, proxies, caching and more
  • Unpacking distributions
  • Placing the contents of a wheel correctly
slide-28
SLIDE 28

ENVIRONMENT MANAGEMENT

  • Allow installation and uninstallation of projects into an

environment.

  • Binary Compatibility — wheel compatibility tags
  • Reproducibility — lockfile
  • Ensure all requirements are satisfied for all installed

packages

  • Dependency Resolution
slide-29
SLIDE 29

FRONT END USABILITY

  • Produce a local directory of artifacts for offline usage
  • downloading from PyPI
  • producing wheels
  • Introspect the current state of installed packages, and

get more information about them

slide-30
SLIDE 30

UPLOADING DISTRIBUTIONS

  • Two Phase uploads — enable new workflows
  • https:/

/github.com/pypa/warehouse/issues/726

  • Easier communication for deprecations
  • Stricter metadata requirements for new uploads
  • Better licensing information
  • Yanking releases
  • Yank due to security bugs
slide-31
SLIDE 31

OK, GREAT. NOW WHAT?

slide-32
SLIDE 32

WE CHART A COURSE TO GET THERE.

slide-33
SLIDE 33

FURTHER READING

  • https:/

/discuss.python.org/t/876

  • https:/

/discuss.python.org/t/2062

  • https:/

/discuss.python.org/t/2570

  • https:/

/discuss.python.org/t/2579

  • https:/

/discuss.python.org/t/2584

  • https:/

/discuss.python.org/t/2587

  • https:/

/discuss.python.org/t/2591

  • https:/

/www.curiousefficiency.org/posts/2016/09/ python-packaging-ecosystem.html

slide-34
SLIDE 34

THANKS!

pradyunsg.me/talks @pradyunsg