Package software for any distribution with upt Cyril Roelandt - - PowerPoint PPT Presentation

▶
package software for any distribution with upt
SMART_READER_LITE
LIVE PREVIEW

Package software for any distribution with upt Cyril Roelandt - - PowerPoint PPT Presentation

Sep 28, 2022 166 likes •810 views

Package software for any distribution with upt Cyril Roelandt Package software for any distribution with upt 1/28 Introduction Cyril Roelandt OpenStack (Red Hat) GNU Guix (Do not miss Ludovics talk at 4:30PM!) Package software for any

slide-1
SLIDE 1

Package software for any distribution with upt

Cyril Roelandt

Package software for any distribution with upt 1/28

slide-2
SLIDE 2

Introduction

Cyril Roelandt OpenStack (Red Hat) GNU Guix (Do not miss Ludovic’s talk at 4:30PM!)

Package software for any distribution with upt 2/28

slide-3
SLIDE 3

Outline

1

Packaging

2

Automation

3

The Universal Packaging Tool

4

fpm

5

Conclusion

Package software for any distribution with upt 3/28

slide-4
SLIDE 4

Outline

1

Packaging

2

Automation

3

The Universal Packaging Tool

4

fpm

5

Conclusion

Package software for any distribution with upt 4/28

slide-5
SLIDE 5

What is packaging?

Turn an “upstream” package into a “downstream” package

Package software for any distribution with upt 5/28

slide-6
SLIDE 6

What is packaging?

Turn an “upstream” package into a “downstream” package Debugging

Package software for any distribution with upt 5/28

slide-7
SLIDE 7

What is packaging?

Turn an “upstream” package into a “downstream” package Debugging Send/maintain patches

Package software for any distribution with upt 5/28

slide-8
SLIDE 8

Is packaging hard?

Many sources (PyPI, RubyGems. . . )

Package software for any distribution with upt 6/28

slide-9
SLIDE 9

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . )

Package software for any distribution with upt 6/28

slide-10
SLIDE 10

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:

Package software for any distribution with upt 6/28

slide-11
SLIDE 11

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:

Dependency hell

Package software for any distribution with upt 6/28

slide-12
SLIDE 12

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:

Dependency hell Debugging

Package software for any distribution with upt 6/28

slide-13
SLIDE 13

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:

Dependency hell Debugging

Some tasks can be automated:

Package software for any distribution with upt 6/28

slide-14
SLIDE 14

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:

Dependency hell Debugging

Some tasks can be automated:

Package layout

Package software for any distribution with upt 6/28

slide-15
SLIDE 15

Is packaging hard?

Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:

Dependency hell Debugging

Some tasks can be automated:

Package layout Metadata

Package software for any distribution with upt 6/28

slide-16
SLIDE 16

Outline

1

Packaging

2

Automation

3

The Universal Packaging Tool

4

fpm

5

Conclusion

Package software for any distribution with upt 7/28

slide-17
SLIDE 17

Existing tools

Debian Fedora Guix FreeBSD OpenBSD CPAN dh-make-perl cpan2rpm guix import ? PortGen NPM npm2deb npm2rpm N/A ? ? PyPI pypi2deb pyp2rpm guix import pytoport PortGen Ruby gem2deb gem2rpm guix import ? PortGen

Package software for any distribution with upt 8/28

slide-18
SLIDE 18

Issues

One CLI per tool

Package software for any distribution with upt 9/28

slide-19
SLIDE 19

Issues

One CLI per tool Different behaviours

Package software for any distribution with upt 9/28

slide-20
SLIDE 20

Issues

One CLI per tool Different behaviours Code duplication (read upstream archive)

Package software for any distribution with upt 9/28

slide-21
SLIDE 21

Issues

One CLI per tool Different behaviours Code duplication (read upstream archive) Code duplication (write downstream package)

Package software for any distribution with upt 9/28

slide-22
SLIDE 22

Example: licenses (PyPI)

SPDX identifiers?

Package software for any distribution with upt 10/28

slide-23
SLIDE 23

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

Package software for any distribution with upt 10/28

slide-24
SLIDE 24

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

License :: OSI Approved :: BSD License

Package software for any distribution with upt 10/28

slide-25
SLIDE 25

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ?

Package software for any distribution with upt 10/28

slide-26
SLIDE 26

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996

Package software for any distribution with upt 10/28

slide-27
SLIDE 27

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996

“license” field (setup.{cfg,py})

Package software for any distribution with upt 10/28

slide-28
SLIDE 28

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996

“license” field (setup.{cfg,py})

For values not available in the standard classifiers

Package software for any distribution with upt 10/28

slide-29
SLIDE 29

Example: licenses (PyPI)

SPDX identifiers? Classifiers:

License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996

“license” field (setup.{cfg,py})

For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .

Package software for any distribution with upt 10/28

slide-30
SLIDE 30

Example: dependencies (PyPI)

pipenv

$ curl -s https://pypi.org/pypi/pipenv/json \ | jq .info.requires_dist [ "ordereddict; python_version < \"2.7\"", "requests[security]; python_version < \"2.7\"", "virtualenv", "virtualenv-clone (>=0.2.5)", "setuptools (>=36.2.1)", "certifi", "pip (>=9.0.1)" ]

Package software for any distribution with upt 11/28

slide-31
SLIDE 31

Example: dependencies (PyPI)

botocore

$ curl -s https://pypi.org/pypi/botocore/json \ | jq .info.requires_dist null

Package software for any distribution with upt 12/28

slide-32
SLIDE 32

Example: dependencies (PyPI)

botocore

$ curl -s https://pypi.org/pypi/botocore/json \ | jq .info.requires_dist null $ wget https://.../botocore-1.12.16-py2.py3-none-any.whl $ unzip botocore-1.12.16-py2.py3-none-any.whl $ cd botocore-1.12.16.dist-info/ $ jq .run_requires metadata.json [ { "requires": [ "jmespath>=0.7.1,<1.0.0", "docutils>=0.10", "urllib3>=1.20,<1.24" ] }, ... ]

Package software for any distribution with upt 12/28

slide-33
SLIDE 33

Example: dependencies (PyPI)

MarkupSafe

$ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null

Package software for any distribution with upt 13/28

slide-34
SLIDE 34

Example: dependencies (PyPI)

MarkupSafe

$ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe

Package software for any distribution with upt 13/28

slide-35
SLIDE 35

Example: dependencies (PyPI)

MarkupSafe

$ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe $ cat setup.py ... extras_require={ 'dev': ... 'docs': ... }

Package software for any distribution with upt 13/28

slide-36
SLIDE 36

Example: dependencies (PyPI)

Read the JSON file provided by PyPI Read the wheel Read the source code Install the package, run “pip list” What about test dependencies? There might be even more ideas!

Package software for any distribution with upt 14/28

slide-37
SLIDE 37

What we really need

Unified CLI

Package software for any distribution with upt 15/28

slide-38
SLIDE 38

What we really need

Unified CLI Unified behaviour

Package software for any distribution with upt 15/28

slide-39
SLIDE 39

What we really need

Unified CLI Unified behaviour Modular

Package software for any distribution with upt 15/28

slide-40
SLIDE 40

What we really need

Unified CLI Unified behaviour Modular Simple

Package software for any distribution with upt 15/28

slide-41
SLIDE 41

Outline

1

Packaging

2

Automation

3

The Universal Packaging Tool

4

fpm

5

Conclusion

Package software for any distribution with upt 16/28

slide-42
SLIDE 42

Architecture

Upstream | upt | upt | upt | Downstream Repos | Frontends | | Backends | Packages

  • +------+ |

+----------+ | +-----+ | +-------------+ | +----------+ | | | | | | | | | | | | | | | PyPI |-|->| upt-pypi |-|->| |-|->| upt-guix |-|->| Guix pkg | | | | | | | | | | | | | | | +------+ | +----------+ | | | | +-------------+ | +----------+ | | | upt | | | +------+ | +----------+ | | | | +-------------+ | +----------+ | | | | | | | | | | | | | | | CPAN |-|->| upt-cpan |-|->| |-|->| upt-openbsd |-|->| Makefile | | | | | | | | | | | | | | | +------+ | +----------+ | +-----+ | +-------------+ | +----------+ | | | |

Package software for any distribution with upt 17/28

slide-43
SLIDE 43

Usage

Install (with modules)

$ pip install upt[frontends,backends]

Package software for any distribution with upt 18/28

slide-44
SLIDE 44

Usage

Install (with modules)

$ pip install upt[frontends,backends]

Available frontends

$ upt list-frontends cpan pypi rubygems

Package software for any distribution with upt 18/28

slide-45
SLIDE 45

Usage

Install (with modules)

$ pip install upt[frontends,backends]

Available frontends

$ upt list-frontends cpan pypi rubygems

Available backends

$ upt list-backends fedora freebsd guix nix

  • penbsd

Package software for any distribution with upt 18/28

slide-46
SLIDE 46

Usage

Package upt for OpenBSD

$ upt package -f pypi -b openbsd -o /tmp/py-upt upt [INFO ] [Backend] Creating the directory \ structure in /tmp/py-upt [INFO ] [Backend] Creating the Makefile [INFO ] [Backend] Creating pkg/DESCR [INFO ] [Backend] You still need to create \ /tmp/py-upt/distinfo [INFO ] [Backend] You still need to create \ /tmp/py-upt/pkg/PLIST

Package software for any distribution with upt 19/28

slide-47
SLIDE 47

Usage

$ less /tmp/py-upt/Makefile

# $OpenBSD$ COMMENT= package software from any package manager MODPY_EGG_VERSION= 0.4.1 DISTNAME= upt-${MODPY_EGG_VERSION} PKGNAME= py-upt-${MODPY_EGG_VERSION} CATEGORIES= XXX HOMEPAGE= https://framagit.org/upt/upt MAINTAINER= XXX <xxx@xxx.xxx> # BSD-3-Clause PERMIT_PACKAGE_CDROM = Yes MODULES= lang/python MODPY_SETUPTOOLS= Yes MODPY_PI= Yes RUN_DEPENDS= xxx/py-spdx-lookup${MODPY_FLAVOR} .include <bsd.port.mk>

Package software for any distribution with upt 20/28

slide-48
SLIDE 48

Usage

$ upt package -f pypi -b nix upt

{ stdenv, buildPythonPackage, fetchPypi , spdx-lookup }: buildPythonPackage rec { name = "${pname}-${version}"; version = "0.4.1"; pname = "upt"; src = fetchPypi { inherit pname version; sha256 = "TODO"; }; propagatedBuildInputs = [ spdx-lookup ]; meta = with stdenv.lib; { description = ... homepage = https://framagit.org/upt/upt; license = [ licenses.bsd3 ]; } }

Package software for any distribution with upt 21/28

slide-49
SLIDE 49

Modularity

One module (backend or frontend) -> one Git repo

Package software for any distribution with upt 22/28

slide-50
SLIDE 50

Modularity

One module (backend or frontend) -> one Git repo Easier for “small” projects

Package software for any distribution with upt 22/28

slide-51
SLIDE 51

Modularity

One module (backend or frontend) -> one Git repo Easier for “small” projects More independence

Package software for any distribution with upt 22/28

slide-52
SLIDE 52

Add a backend or frontend

$ cookiecutter https://framagit.org/upt/cookiecutter-upt

project_name []: upt-archlinux project_slug [upt_archlinux]: author []: John Doe email []: john@doe.org short_description []: Arch Linux backend for upt. url [https://framagit.org/upt/upt-archlinux]: Select license: 1 - BSD-3 2 - other Choose from 1, 2 (1, 2) [1]: 1 Select kind: 1 - backend 2 - frontend Choose from 1, 2 (1, 2) [1]: 1 Initialized empty Git repository in /tmp/upt-archlinux/.git/

Package software for any distribution with upt 23/28

slide-53
SLIDE 53

Outline

1

Packaging

2

Automation

3

The Universal Packaging Tool

4

fpm

5

Conclusion

Package software for any distribution with upt 24/28

slide-54
SLIDE 54

A different approcah

Ruby

Package software for any distribution with upt 25/28

slide-55
SLIDE 55

A different approcah

Ruby Modules are distributed along with the core

Package software for any distribution with upt 25/28

slide-56
SLIDE 56

A different approcah

Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder)

Package software for any distribution with upt 25/28

slide-57
SLIDE 57

A different approcah

Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager

Package software for any distribution with upt 25/28

slide-58
SLIDE 58

A different approcah

Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers?

Package software for any distribution with upt 25/28

slide-59
SLIDE 59

Outline

1

Packaging

2

Automation

3

The Universal Packaging Tool

4

fpm

5

Conclusion

Package software for any distribution with upt 26/28

slide-60
SLIDE 60

Future work

More modules (written by you?)

Package software for any distribution with upt 27/28

slide-61
SLIDE 61

Future work

More modules (written by you?) Type hints

Package software for any distribution with upt 27/28

slide-62
SLIDE 62

Future work

More modules (written by you?) Type hints Need a standard format for all upstream package archives

Package software for any distribution with upt 27/28

slide-63
SLIDE 63

Join me!

Code: https://framagit.org/upt Emails: cyril.roelandt@aquilenet.fr, cyril@redhat.com Freenode: #upt-packaging, Steap Questions?

Package software for any distribution with upt 28/28