Outsourcing Source Code Distribution Requirements Alexios Zavras, - - PowerPoint PPT Presentation

▶
outsourcing source code distribution requirements
SMART_READER_LITE
LIVE PREVIEW

Outsourcing Source Code Distribution Requirements Alexios Zavras, - - PowerPoint PPT Presentation

Jan 30, 2023 124 likes •425 views

Outsourcing Source Code Distribution Requirements Alexios Zavras, Stefano Zacchiroli Intel, alexios.zavras@intel.com Sofware Heritage, zack@upsilon.cc 4 February 2018 FOSDEM Brussels, Belgium Alexios Zavras, Stefano Zacchiroli Outsourcing

slide-1
SLIDE 1

Outsourcing Source Code Distribution Requirements

Alexios Zavras, Stefano Zacchiroli

Intel, alexios.zavras@intel.com Sofware Heritage, zack@upsilon.cc

4 February 2018 FOSDEM Brussels, Belgium

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 1 / 19

slide-2
SLIDE 2

The setup

Intel delivers a lot of sofware Sofware is a combination of own and FOSS components Many components have a legal source code distribution requirement

we also might deliver source in other cases

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 2 / 19

slide-3
SLIDE 3

The legal requirement

For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. — GPLv2

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 3 / 19

slide-4
SLIDE 4

Complete Corresponding Source (CCS)

Different terms used GPLv2: “complete corresponding machine-readable source code” / “accompany” GPLv3: “Corresponding Source” / “convey” MPLv2: “Source Code Form” / “made available” EPLv2: “Source Code” / “made available”

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 4 / 19

slide-5
SLIDE 5

The problem

In an ideal world Fool-proof processes in place Set it up once, always working Practical considerations People change roles or leave Re-organizations happen Things get forgoten

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 5 / 19

slide-6
SLIDE 6

Use cases

Trying to build an internal service: Our delivery contains our own FOSS sw.tar.gz Our delivery contains gcc-7.3 Our delivery contains gcc snapshot of revision 257214 Our delivery contains gcc-7.3 patched with patches.tar.gz

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 6 / 19

slide-7
SLIDE 7

Functional requirements

We need to be able to: provide our own sofware package refer to a “well-known” FOSS component

with release version or unique revision

combine the two

well-known component with own patches

Great Idea Can we outsource the fulfilment of these requirements?

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 7 / 19

slide-8
SLIDE 8

The idea

Is it compliant? GPL FAQ: Can I put the binaries on my Internet server and put the source on a different Internet site? [v3] Yes. Section 6(d) allows this. However, you must provide clear instructions people can follow to obtain the source, and you must take care to make sure that the source remains available for as long as you distribute the object code. [v2] The GPL says you must offer access to copy the source code “from the same place”; that is, next to the binaries. However, if you make arrangements with another site to keep the necessary source code available, and put a link

  • r cross-reference to the source code next to the binaries, we think that

qualifies as “from the same place”.

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 8 / 19

slide-9
SLIDE 9

The idea

Is it compliant? GPL FAQ: Can I put the binaries on my Internet server and put the source on a different Internet site? [v3] Yes. Section 6(d) allows this. However, you must provide clear instructions people can follow to obtain the source, and you must take care to make sure that the source remains available for as long as you distribute the object code. [v2] The GPL says you must offer access to copy the source code “from the same place”; that is, next to the binaries. However, if you make arrangements with another site to keep the necessary source code available, and put a link

  • r cross-reference to the source code next to the binaries, we think that

qualifies as “from the same place”. Wouldn’t it be great if someone could fulfill our requirements?

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 8 / 19

slide-10
SLIDE 10

The Sofware Heritage Project

THE GREAT LIBRARY OF SOURCE CODE

Our mission Collect, preserve and share the source code of all the sofware that is publicly available. Past, present and future Preserving the past, enhancing the present, preparing the future.

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 9 / 19

slide-11
SLIDE 11

Our principles

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 10 / 19

slide-12
SLIDE 12

Our principles

Open approach

  • pen source

transparency In for the long haul non profit replication

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 10 / 19

slide-13
SLIDE 13

Data flow

dsc dsc hg hg hg git git git git svn svn svn tar zip

software

  • rigins

Package repos Software Heritage Archive Forges

GitHub lister GitLab lister Debian lister Git loader Mercurial loader Debian source package loader PyPi lister tar loader Merkle DAG + blob storage

. . . . . . Distros ... Scheduling Listing (full/incremental) Loading & deduplication

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 11 / 19

slide-14
SLIDE 14

Archive coverage

Current sources live: GitHub, Debian

  • ne-off: Gitorious, Google Code

WIP: Bitbucket

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 12 / 19

slide-15
SLIDE 15

Archive coverage

Current sources live: GitHub, Debian

  • ne-off: Gitorious, Google Code

WIP: Bitbucket 150 TB blobs, 5 TB database (as a graph: 7 B nodes + 60 B edges)

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 12 / 19

slide-16
SLIDE 16

Archive coverage

Current sources live: GitHub, Debian

  • ne-off: Gitorious, Google Code

WIP: Bitbucket 150 TB blobs, 5 TB database (as a graph: 7 B nodes + 60 B edges) The richest public source code archive, ... and growing daily!

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 12 / 19

slide-17
SLIDE 17

Pushing source code to Sofware Heritage

Deposit service complement regular (pull) crawling of forges and distributions restricted access (i.e., not a warez dumpster!) deposit.softwareheritage.org Tech bits SWORD 2.0 compliant server, for digital repositories interoperability RESTful API for deposit and monitoring, with CLI wrapper

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 13 / 19

slide-18
SLIDE 18

Prepare a deposit

Prepare source code tarball $ tar caf software.tar.gz /path/to/software/

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 14 / 19

slide-19
SLIDE 19

Prepare a deposit

Prepare source code tarball $ tar caf software.tar.gz /path/to/software/ Associate metadata

$ cat > software.tar.gz.metadata.xml <?xml version="1.0"?> <entry xmlns="http://www.w3.org/2005/Atom" xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0"> <title>Je suis GPL</title> <codemeta:url>https://forge.softwareheritage.org/source/jesuisgpl/</codemeta:url> <codemeta:author> <codemeta:name>Stefano Zacchiroli</codemeta:name> <codemeta:jobTitle>Maintainer</codemeta:jobTitle> </codemeta:author> </entry> ^D

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 14 / 19

slide-20
SLIDE 20

Send a deposit

$ swh-deposit --username ’name’ --password ’pass’ \

  • -archive software.tar.gz

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 15 / 19

slide-21
SLIDE 21

Send a deposit

$ swh-deposit --username ’name’ --password ’pass’ \

  • -archive software.tar.gz

{ ’deposit_id’: ’11’, ’deposit_status’: ’deposited’, ’deposit_date’: ’Jan. 30, 2018, 9:37 a.m.’ }

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 15 / 19

slide-22
SLIDE 22

Ingestion status

partial deposited verified rejected done failed

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 16 / 19

slide-23
SLIDE 23

Ingestion status

partial deposited verified rejected done failed

$ swh-deposit --username ’name’ --pass ’secret’ \

  • -deposit-id ’11’ --status

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 16 / 19

slide-24
SLIDE 24

Ingestion status

partial deposited verified rejected done failed

$ swh-deposit --username ’name’ --pass ’secret’ \

  • -deposit-id ’11’ --status

{ ’deposit_id’: 11, ’deposit_status’: ’done’, ’deposit_status_detail’: The deposit has been successfully loaded into the Software Heritage archive’, ’deposit_swh_id’: ’swh:1:rev:a86747d201ab8f8657d145df4376676d5e47cf9f’ }

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 16 / 19

slide-25
SLIDE 25

Access a deposit

Afer ingestion a deposit becomes an integral, permanent part of the Sofware Heritage archive. it has a persistent identifier

e.g., swh:1:rev:a86747d201ab8f8657d145df4376676d5e47cf9f

it can be browsed online at archive.softwareheritage.org

e.g., https://archive.softwareheritage.org/browse/swh:1: rev:a86747d201ab8f8657d145df4376676d5e47cf9f

it can be bulk downloaded using the Sofware Heritage Vault

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 17 / 19

slide-26
SLIDE 26

Bulk download

source code is thoroughly deduplicated within the Sofware Heritage archive bulk download of large artefacts (e.g., a Linux kernel release) requires collecting millions of objects the Sofware Heritage Vault cooks and caches source code bundles for bulk download needs

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 18 / 19

slide-27
SLIDE 27

Bulk download

source code is thoroughly deduplicated within the Sofware Heritage archive bulk download of large artefacts (e.g., a Linux kernel release) requires collecting millions of objects the Sofware Heritage Vault cooks and caches source code bundles for bulk download needs

$ curl -X POST /api/1/vault/revision/a86747d2.../gitfast { ’fetch_url’: ’/api/1/vault/revision/a86747d2.../gitfast/raw/’, ’progress_message’: None, ’status’: ’new’, ’id’: 4, ’obj_id’: ’a86747d201ab8f8657d145df4376676d5e47cf9f’, ’obj_type’: ’revision_gitfast’ }

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 18 / 19

slide-28
SLIDE 28

Bulk download

source code is thoroughly deduplicated within the Sofware Heritage archive bulk download of large artefacts (e.g., a Linux kernel release) requires collecting millions of objects the Sofware Heritage Vault cooks and caches source code bundles for bulk download needs

$ curl -X POST /api/1/vault/revision/a86747d2.../gitfast { ’fetch_url’: ’/api/1/vault/revision/a86747d2.../gitfast/raw/’, ’progress_message’: None, ’status’: ’new’, ’id’: 4, ’obj_id’: ’a86747d201ab8f8657d145df4376676d5e47cf9f’, ’obj_type’: ’revision_gitfast’ } $ curl -O dump.gz /api/1/vault/revision/a86747d2.../gitfast/raw/ $ git init $ zcat dump.gz | git fast-import $ git checkout HEAD

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 18 / 19

slide-29
SLIDE 29

Wrapping up

long-term hosting of CCS archives can be onerous in the real-world it is A-OK to outsource that responsibility to third parties Sofware Heritage crawls (pull) all FOSS and can now accept push deposits Intel and Sofware Heritage are working together on practical FOSS tooling to

  • utsource CCS hosting to the Sofware Heritage archive

Come and join us! alexios.zavras@intel.com , zack@upsilon.cc https://www.softwareheritage.org https://deposit.softwareheritage.org https://archive.softwareheritage.org (FOSDEM 2018 preview!)

Alexios Zavras, Stefano Zacchiroli Outsourcing Source Code Distribution Requirements FOSDEM 2018 19 / 19