reproducible installation of applications using zc
play

Reproducible installation of applications using zc.buildout Thomas - PowerPoint PPT Presentation

Reproducible installation of applications using zc.buildout Thomas Lotze Europython 2012 tl@gocept.com July 5, Florence, Italy Outline Overview: Scope 1 Sketch: Simple example of a buildout 2 Close-up: How zc.buildout installs Python code


  1. What’s needed? assume zc.buildout is not installed download bootstrap.py $ wget http://svn.zope.org/*checkout*/zc.buildout/ \ trunk/bootstrap/bootstrap.py create a buildout configuration file [buildout] 1 parts = sphinx 2 3 [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6

  2. What’s needed? assume zc.buildout is not installed download bootstrap.py $ wget http://svn.zope.org/*checkout*/zc.buildout/ \ trunk/bootstrap/bootstrap.py create a buildout configuration file [buildout] 1 parts = sphinx 2 3 [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6

  3. Getting started $ python bootstrap.py -d Downloading http://pypi..../distribute-0.6.27.tar.gz ... Creating directory ’/home/thomas/py/bin’. Creating directory ’/home/thomas/py/parts’. Creating directory ’/home/thomas/py/eggs’. Creating directory ’/home/thomas/py/develop-eggs’. Generated script ’/home/thomas/py/bin/buildout’.

  4. After bootstrapping $ ls * bootstrap.py buildout.cfg bin: buildout develop-eggs: eggs: distribute-0.6.27-py2.7.egg zc.buildout-1.5.2-py2.7.egg parts: buildout

  5. What’s next? $ bin/buildout Getting distribution for ’zc.recipe.egg’. Got zc.recipe.egg 1.3.2. Installing sphinx. Getting distribution for ’sphinx’. Got Sphinx 1.1.3. Getting distribution for ’docutils>=0.7’. warning: ... Got docutils 0.9.1. Getting distribution for ’Jinja2>=2.3’. warning: ... Got Jinja2 2.6. Getting distribution for ’Pygments>=1.2’. Got Pygments 1.5. Generated script ’/home/thomas/py/bin/sphinx-apidoc’. Generated script ’/home/thomas/py/bin/sphinx-build’. Generated script ’/home/thomas/py/bin/sphinx-quickstart’. Generated script ’/home/thomas/py/bin/sphinx-autogen’.

  6. After the buildout run $ ls * bootstrap.py buildout.cfg bin: buildout sphinx-apidoc sphinx-autogen sphinx-build sphinx-quickstart develop-eggs: eggs: distribute-0.6.27-py2.7.egg docutils-0.9.1-py2.7.egg Jinja2-2.6-py2.7.egg Pygments-1.5-py2.7.egg Sphinx-1.1.3-py2.7.egg zc.buildout-1.5.2-py2.7.egg zc.recipe.egg-1.3.2-py2.7.egg parts: buildout

  7. What happened? [buildout] 1 parts = sphinx 2 3 [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 buildout part “sphinx” is installed work is done by a recipe: plug-in point recipe comes as an egg Getting distribution for ’zc.recipe.egg’. Got zc.recipe.egg 1.3.2. Installing sphinx.

  8. What happened? [buildout] 1 parts = sphinx 2 3 [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 buildout part “sphinx” is installed work is done by a recipe: plug-in point recipe comes as an egg Getting distribution for ’zc.recipe.egg’. Got zc.recipe.egg 1.3.2. Installing sphinx.

  9. What happened? [buildout] 1 parts = sphinx 2 3 [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 buildout part “sphinx” is installed work is done by a recipe: plug-in point recipe comes as an egg Getting distribution for ’zc.recipe.egg’. Got zc.recipe.egg 1.3.2. Installing sphinx.

  10. What happened? [buildout] 1 parts = sphinx 2 3 [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 buildout part “sphinx” is installed work is done by a recipe: plug-in point recipe comes as an egg Getting distribution for ’zc.recipe.egg’. Got zc.recipe.egg 1.3.2. Installing sphinx.

  11. What happened? The sphinx part zc.recipe.egg invokes zc.buildout’s easy_install API download sphinx sources (as configured) Getting distribution for ’sphinx’. build the egg Got Sphinx 1.1.3. follow declared dependencies Getting distribution for ’docutils>=0.5’. Got docutils 0.9.1. ... detect and install scripts provided by explicitly listed eggs Generated script ’/home/thomas/py/bin/sphinx-apidoc’. ...

  12. What happened? The sphinx part zc.recipe.egg invokes zc.buildout’s easy_install API download sphinx sources (as configured) Getting distribution for ’sphinx’. build the egg Got Sphinx 1.1.3. follow declared dependencies Getting distribution for ’docutils>=0.5’. Got docutils 0.9.1. ... detect and install scripts provided by explicitly listed eggs Generated script ’/home/thomas/py/bin/sphinx-apidoc’. ...

  13. What happened? The sphinx part zc.recipe.egg invokes zc.buildout’s easy_install API download sphinx sources (as configured) Getting distribution for ’sphinx’. build the egg Got Sphinx 1.1.3. follow declared dependencies Getting distribution for ’docutils>=0.5’. Got docutils 0.9.1. ... detect and install scripts provided by explicitly listed eggs Generated script ’/home/thomas/py/bin/sphinx-apidoc’. ...

  14. What happened? The sphinx part zc.recipe.egg invokes zc.buildout’s easy_install API download sphinx sources (as configured) Getting distribution for ’sphinx’. build the egg Got Sphinx 1.1.3. follow declared dependencies Getting distribution for ’docutils>=0.5’. Got docutils 0.9.1. ... detect and install scripts provided by explicitly listed eggs Generated script ’/home/thomas/py/bin/sphinx-apidoc’. ...

  15. What happened? The sphinx part zc.recipe.egg invokes zc.buildout’s easy_install API download sphinx sources (as configured) Getting distribution for ’sphinx’. build the egg Got Sphinx 1.1.3. follow declared dependencies Getting distribution for ’docutils>=0.5’. Got docutils 0.9.1. ... detect and install scripts provided by explicitly listed eggs Generated script ’/home/thomas/py/bin/sphinx-apidoc’. ...

  16. Repeating the buildout run with configuration unchanged: $ bin/buildout Updating sphinx. already installed, not installed again unconditional update phase looks for new releases by default

  17. Repeating the buildout run with configuration unchanged: $ bin/buildout Updating sphinx. already installed, not installed again unconditional update phase looks for new releases by default

  18. Repeating the buildout run with configuration unchanged: $ bin/buildout Updating sphinx. already installed, not installed again unconditional update phase looks for new releases by default

  19. Repeating the buildout run modify configuration: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 scripts = sphinx-build sphinx-apidoc 7 $ bin/buildout Uninstalling sphinx. Installing sphinx. Generated script ’/home/thomas/py/sphinx-apidoc’. Generated script ’/home/thomas/py/sphinx-build’. part with modified configuration is re-installed from scratch previously created files (e.g. scripts) are removed parts with unchanged configuration are updated

  20. Repeating the buildout run modify configuration: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 scripts = sphinx-build sphinx-apidoc 7 $ bin/buildout Uninstalling sphinx. Installing sphinx. Generated script ’/home/thomas/py/sphinx-apidoc’. Generated script ’/home/thomas/py/sphinx-build’. part with modified configuration is re-installed from scratch previously created files (e.g. scripts) are removed parts with unchanged configuration are updated

  21. Repeating the buildout run modify configuration: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 scripts = sphinx-build sphinx-apidoc 7 $ bin/buildout Uninstalling sphinx. Installing sphinx. Generated script ’/home/thomas/py/sphinx-apidoc’. Generated script ’/home/thomas/py/sphinx-build’. part with modified configuration is re-installed from scratch previously created files (e.g. scripts) are removed parts with unchanged configuration are updated

  22. Repeating the buildout run modify configuration: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 scripts = sphinx-build sphinx-apidoc 7 $ bin/buildout Uninstalling sphinx. Installing sphinx. Generated script ’/home/thomas/py/sphinx-apidoc’. Generated script ’/home/thomas/py/sphinx-build’. part with modified configuration is re-installed from scratch previously created files (e.g. scripts) are removed parts with unchanged configuration are updated

  23. Repeating the buildout run modify configuration: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 scripts = sphinx-build sphinx-apidoc 7 $ bin/buildout Uninstalling sphinx. Installing sphinx. Generated script ’/home/thomas/py/sphinx-apidoc’. Generated script ’/home/thomas/py/sphinx-build’. part with modified configuration is re-installed from scratch previously created files (e.g. scripts) are removed parts with unchanged configuration are updated

  24. Contents Overview: Scope 1 Sketch: Simple example of a buildout 2 Close-up: How zc.buildout installs Python code 3 Our perspective: Reproducible builds 4 Bigger picture: More complex applications 5 Summary and Outlook: zc.buildout’s future 6

  25. How Python code is installed scripts and their environment a Python interpreter compare to virtualenv + pip

  26. How Python code is installed scripts and their environment a Python interpreter compare to virtualenv + pip

  27. How Python code is installed scripts and their environment a Python interpreter compare to virtualenv + pip

  28. How egg installation works $ cat bin/sphinx-quickstart #!/usr/bin/python 1 2 import sys 3 sys.path[0:0] = [ 4 ’/home/thomas/py/eggs/Sphinx-1.1.3-py2.7.egg’, 5 ’/home/thomas/py/eggs/docutils-0.9.1-py2.7.egg’, 6 ’/home/thomas/py/eggs/Jinja2-2.6-py2.7.egg’, 7 ’/home/thomas/py/eggs/Pygments-1.5-py2.7.egg’, 8 ] 9 10 import sphinx.quickstart 11 12 if __name__ == ’__main__’: 13 sphinx.quickstart.main() 14

  29. How egg installation works each script calls one of the egg’s entry points if __name__ == ’__main__’: 13 sphinx.quickstart.main() 14 each script sets up its own Python path sys.path[0:0] = [ 4 ’/home/thomas/py/eggs/Sphinx-1.1.3-py2.7.egg’, 5 ... 6 use a Python installation without modifying it

  30. How egg installation works each script calls one of the egg’s entry points if __name__ == ’__main__’: 13 sphinx.quickstart.main() 14 each script sets up its own Python path sys.path[0:0] = [ 4 ’/home/thomas/py/eggs/Sphinx-1.1.3-py2.7.egg’, 5 ... 6 use a Python installation without modifying it

  31. How egg installation works each script calls one of the egg’s entry points if __name__ == ’__main__’: 13 sphinx.quickstart.main() 14 each script sets up its own Python path sys.path[0:0] = [ 4 ’/home/thomas/py/eggs/Sphinx-1.1.3-py2.7.egg’, 5 ... 6 use a Python installation without modifying it

  32. Using eggs with an interpreter configure the eggs’ part to create an interpreter: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 interpreter = py 7 the egg recipe creates an executable script: $ bin/buildout ... Generated interpreter ’/home/thomas/py/bin/py’.

  33. Using eggs with an interpreter configure the eggs’ part to create an interpreter: [sphinx] 4 recipe = zc.recipe.egg 5 eggs = sphinx 6 interpreter = py 7 the egg recipe creates an executable script: $ bin/buildout ... Generated interpreter ’/home/thomas/py/bin/py’.

  34. Using eggs with an interpreter #!/usr/bin/python 1 2 import sys 3 sys.path[0:0] = [ 4 ’/var/lib/python-eggs/Sphinx-1.1.3-py2.7.egg’, 5 ’/var/lib/python-eggs/docutils-0.9.1-py2.7.egg’, 6 ’/var/lib/python-eggs/Jinja2-2.6-py2.7.egg’, 7 ’/var/lib/python-eggs/Pygments-1.5-py2.7.egg’, 8 ] 9 10 ... 11 exec _val 12 __import__("runpy").run_module(...) 13 execfile(...) 14 __import__("code").interact(...) 15 just another script that sets up its path invokes Python interpreter according to options

  35. Using eggs with an interpreter #!/usr/bin/python 1 2 import sys 3 sys.path[0:0] = [ 4 ’/var/lib/python-eggs/Sphinx-1.1.3-py2.7.egg’, 5 ’/var/lib/python-eggs/docutils-0.9.1-py2.7.egg’, 6 ’/var/lib/python-eggs/Jinja2-2.6-py2.7.egg’, 7 ’/var/lib/python-eggs/Pygments-1.5-py2.7.egg’, 8 ] 9 10 ... 11 exec _val 12 __import__("runpy").run_module(...) 13 execfile(...) 14 __import__("code").interact(...) 15 just another script that sets up its path invokes Python interpreter according to options

  36. Using eggs with an interpreter #!/usr/bin/python 1 2 import sys 3 sys.path[0:0] = [ 4 ’/var/lib/python-eggs/Sphinx-1.1.3-py2.7.egg’, 5 ’/var/lib/python-eggs/docutils-0.9.1-py2.7.egg’, 6 ’/var/lib/python-eggs/Jinja2-2.6-py2.7.egg’, 7 ’/var/lib/python-eggs/Pygments-1.5-py2.7.egg’, 8 ] 9 10 ... 11 exec _val 12 __import__("runpy").run_module(...) 13 execfile(...) 14 __import__("code").interact(...) 15 just another script that sets up its path invokes Python interpreter according to options

  37. Contrast: virtualenv + pip creates a Python installation meant to be modified pip requirements file: minimal set of packages defines the Python path as a well-known directory Python path implicitly set up by using the local interpreter Python path may be exported: “activate” the environment

  38. Contrast: virtualenv + pip creates a Python installation meant to be modified pip requirements file: minimal set of packages defines the Python path as a well-known directory Python path implicitly set up by using the local interpreter Python path may be exported: “activate” the environment

  39. Contrast: virtualenv + pip creates a Python installation meant to be modified pip requirements file: minimal set of packages defines the Python path as a well-known directory Python path implicitly set up by using the local interpreter Python path may be exported: “activate” the environment

  40. Contrast: virtualenv + pip creates a Python installation meant to be modified pip requirements file: minimal set of packages defines the Python path as a well-known directory Python path implicitly set up by using the local interpreter Python path may be exported: “activate” the environment

  41. Contrast: virtualenv + pip creates a Python installation meant to be modified pip requirements file: minimal set of packages defines the Python path as a well-known directory Python path implicitly set up by using the local interpreter Python path may be exported: “activate” the environment

  42. Contents Overview: Scope 1 Sketch: Simple example of a buildout 2 Close-up: How zc.buildout installs Python code 3 Our perspective: Reproducible builds 4 Bigger picture: More complex applications 5 Summary and Outlook: zc.buildout’s future 6

  43. Reproducibility specifying what to install: pinning versions enforcing a complete specification known-good sets of software packages

  44. Reproducibility specifying what to install: pinning versions enforcing a complete specification known-good sets of software packages

  45. Reproducibility specifying what to install: pinning versions enforcing a complete specification known-good sets of software packages

  46. Which eggs are installed? determined by buildout configuration and dependencies full paths baked into scripts: no random additions eggs are looked up at the package index eggs may also come from files or an on-line list of links

  47. Which eggs are installed? determined by buildout configuration and dependencies full paths baked into scripts: no random additions eggs are looked up at the package index eggs may also come from files or an on-line list of links

  48. Which eggs are installed? determined by buildout configuration and dependencies full paths baked into scripts: no random additions eggs are looked up at the package index eggs may also come from files or an on-line list of links

  49. Which eggs are installed? determined by buildout configuration and dependencies full paths baked into scripts: no random additions eggs are looked up at the package index eggs may also come from files or an on-line list of links

  50. Which egg versions are installed? declared dependencies on versions are always fulfilled newest matching versions are used search for newer versions may be suppressed versions still depend on first installation

  51. Which egg versions are installed? declared dependencies on versions are always fulfilled newest matching versions are used search for newer versions may be suppressed versions still depend on first installation

  52. Which egg versions are installed? declared dependencies on versions are always fulfilled newest matching versions are used search for newer versions may be suppressed versions still depend on first installation

  53. Which egg versions are installed? declared dependencies on versions are always fulfilled newest matching versions are used search for newer versions may be suppressed versions still depend on first installation

  54. How to pin versions with buildout global option: [buildout] 1 versions = versions 2 3 [versions] 4 sphinx = 1.1.2 5 version pinnings are always honoured versions of other packages are still unpredictable

  55. How to pin versions with buildout global option: [buildout] 1 versions = versions 2 3 [versions] 4 sphinx = 1.1.2 5 version pinnings are always honoured versions of other packages are still unpredictable

  56. How to pin versions with buildout global option: [buildout] 1 versions = versions 2 3 [versions] 4 sphinx = 1.1.2 5 version pinnings are always honoured versions of other packages are still unpredictable

  57. Forcing all versions to be pinned [buildout] 1 parts = sphinx 2 versions = versions 3 allow-picked-versions = false 4 5 [versions] 6 Jinja2 = 2.6 7 Pygments = 1.5 8 distribute = 0.6.27 9 docutils = 0.9.1 10 sphinx = 1.1.2 11 zc.buildout = 1.5.2 12 zc.recipe.egg = 1.3.2 13 14 [sphinx] 15 recipe = zc.recipe.egg 16 eggs = sphinx 17

  58. Forcing all versions to be pinned recipes and even zc.buildout itself are pinned known-good build in addition to known-good code: be sure that pieces of the build system match predictable configuration (e.g. paths, generated scripts) still not pinned: Python itself

  59. Forcing all versions to be pinned recipes and even zc.buildout itself are pinned known-good build in addition to known-good code: be sure that pieces of the build system match predictable configuration (e.g. paths, generated scripts) still not pinned: Python itself

  60. Forcing all versions to be pinned recipes and even zc.buildout itself are pinned known-good build in addition to known-good code: be sure that pieces of the build system match predictable configuration (e.g. paths, generated scripts) still not pinned: Python itself

  61. Forcing all versions to be pinned recipes and even zc.buildout itself are pinned known-good build in addition to known-good code: be sure that pieces of the build system match predictable configuration (e.g. paths, generated scripts) still not pinned: Python itself

  62. Forcing all versions to be pinned recipes and even zc.buildout itself are pinned known-good build in addition to known-good code: be sure that pieces of the build system match predictable configuration (e.g. paths, generated scripts) still not pinned: Python itself

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend