Migrate Python from 2.X to 3.X WHO AM I? C++ & PYTHON DEVELOPER - PowerPoint PPT Presentation

Migrate Python from 2.X to 3.X

WHO AM I?

C++ & PYTHON DEVELOPER 6,5 years in 20,5 years in CAD&Numeric C++ simulations PHILIPPE BOULANGER 3 years in 19,5 years in embedded Python programming 11,5 years in finance @Pythonicien

CONCERNS

▪ 2017 ➢ Instagram migrated major part of SOME its code to Python 3 NEWS ▪ September 2018 ➢ Dropbox announced the end of its migration to Python 3 (they began in 2015)!

TECHNICAL ASPECTS (1/2) ▪ Support for Pyt ython 2.7 will ill stop soon ➢ January the 1 st 2020 ▪ Some li libraries are no no more compliant ➢ Django, numpy (2019), etc. ▪ Pyt ython 4 ➢ Will arrive in the next few years (2023 ?)

TECHNICAL ASPECTS (2/2) ▪ Asynchronous programming (a (asyncio) ▪ Consistency ➢ Return generator instead of containers ➢ Functional programming

DIFFERENCES (1/3) Python 2 Python 3 print print ‘blabla' print ( ‘blabla’ )

DIFFERENCES (1/3) Python 2 Python 3 print print ‘blabla' print ( ‘blabla’ ) raise raise IOError, ‘file error' raise IOError( ‘file error’ )

DIFFERENCES (1/3) Python 2 Python 3 print print ‘blabla' print ( ‘blabla’ ) raise raise IOError, ‘file error' raise IOError( ‘file error’ ) long (myvar) int (myvar) long 5/2 = 2 5/2 = 2.5 5//2 = 2

DIFFERENCES (2/3) Python 2 Python 3 unicode str string str bytes

DIFFERENCES (2/3) Python 2 Python 3 unicode str string str bytes dict.items(): list dict.items(): dict_items dict, map, zip dict.keys()[0] list(dict.keys())[0] dict.iteritems() dict.items()

DIFFERENCES (3/3) u’toto’ UNICODE b’titi’ instruction unicode() MANAGEMENT method __unicode__() StringIO/BytesIO

FINANCIAL ASPECT ▪ Mig igration costs? ➢ Heavy costs at short term ➢ Few costs at long term ▪ Costs to to keep Pyt ython 2? ➢ No immediat costs ➢ Heavy cost at middle/long term

GOAL…

NON-REGRESSION TESTS

▪ Is the migration a success? ▪ Are the performances as good as the HOW TO 2.X version? VALIDATE THE MIGRATION? ▪ What is the coverage of the tests? ▪ We need in indicators!

UNIT TESTS ▪ Use a unit test for a small part of code testing (As a function). ▪ Utopic goal: have unit tests for all API.

FUNCTIONAL TESTS (1/2) ▪ Functional tests are more complex because several API are linked but cover a real service or functionality. ▪ Objective: cover most of the functionalities as possible. Having a tool to mesure code coverage will be useful.

FUNCTIONAL TESTS (2/2) ▪ The need to automate tests is increasing with program size ▪ A functional test could be: ➢ A chain of API calls ➢ GUI actions (use of UFT/QTP)

PERFORMANCES TESTS ▪ You need to validate that migration keep the application performances : algorithms used in libraries could be replaced between versions, some conflicts between libraries could appears … ➢ Load tests?

COVERAGE TESTS (1/2) ▪ Knowing the number of lines of codes tested when all the tests are using (unit, functional or performance) https://coverage.readthedocs.io/en/coverage-4.5.1a

COVERAGE TESTS (2/2) ▪ According to my experience, with less than 60% of covered code, the chance of having hidden bugs is very important ▪ A good target is 80% of covered code Target 80% of covered code

GUI TESTS ▪ Test the GUI - Either manual - Or use tool like UFT (previous name: QTP) - … ▪ Allow to automate test as if it was done by a user

HUMAN TESTS ▪ A developer tests the code from a way which corresponds to the implementation he done, a real user tests according to its habits ➢ Update GUI controls , click… raise events and code execution and the order of calls can change the behavior ➢ human add random part inside tests

PERIMETER

PERIMETER? ?

ENVIRONMENT OS DB PROCESSOR PYTHON TOOLING DISTRIBUTION

WHAT MODULES ARE LOADED? (1/2) ▪ Standard modules in Python? ▪ Modules which were developed in intern? ▪ What are the external modules?

WHAT MODULES ARE LOADED? (2/2) ▪ How to determine the list of dynamically loaded modules … Available since Python 2.3.

AND THEN: PROBLEMS?… ▪ Is there module: ➢ with ended support or unmigrated? ➢ with modified API? ➢ licencing changed? ➢ library name changed?

NON-PYTHON MODULES ▪ Problems with C/C++ written modules ➢ C++ compiler migration ➢ porting C++ libraries ➢ tools problems (swig…) ➢ licences, etc…

MIGRATION METHODOLOGY

▪ « Divide to reign »: it will be better to migrate small groups of files to minimize interactions. SPLIT IN BUNDLES ▪ Create bundles in using module dependencies (have a graph should be useful), internal or external module …

PORTING EXTERNAL CODE (1/2) ▪ External code has no dependency with house-made code: start with them will be a good idea. ▪ Take count of tools: ➢ Compiler ➢ Integration tools in Python: swig, boost.python, etc. ➢ External libraries

PORTING EXTERNAL CODE (2/2) ▪ Library was ported or not? ▪ API changed? ▪ Licensing changed? ▪ Is there constraints according to the versions of different libraries? ▪ Is source code available ?

ADD PYTHON 3.X CHANGES INSIDE PYTHON 2.X CODE (1/3) ▪ from __future__ import division - PEP 238: Changing the Division Operator ▪ from __future__ import print_function - PEP 3105: Make print a function

ADD PYTHON 3.X CHANGES INSIDE PYTHON 2.X CODE (2/3) ▪ from __future__ import absolute_import - PEP 328: Imports: Multi-Line and Absolute/Relative ▪ from __future__ import unicode_literals - PEP 3112: Bytes literals in Python 3000

ADD PYTHON 3.X CHANGES INSIDE PYTHON 2.X CODE (3/3) ▪ Six : six.readthedoc.io Python 3 Python 2 Six to add 3.X features in 2.X code

TOOLS (1/2) ▪ 2to3 try: mathilde@pc-moi~ 2to3 example.py x = urlopen[my_url].read() RefactoringTool: Refactored example.py - print x --- example.py (original) +++ example.py (refactored) -except Exception, e: @@ -1,9 +1,9 @@ - raise IOError, "Error 404" + print(x) - from urllib2 import urlopen +except Exception as e: + from urllib.request import urlopen + raise IOError("Error 404") my_url = “http://pythonprogramming.net” RefactoringTool: Files that need to be modified: RefactoringTool: example.py

TOOLS (2/2) ▪ 2to6 ➢ Based on 2to3 ➢ For compilancy between 2 and 3 ➢ Add __future__, six

REFACTORING

REFACTORING (1/6) ▪ listdir vs scandir

REFACTORING (2/6) ▪ Use generators

REFACTORING (3/6) ▪ Comprehension containers

REFACTORING (4/6) ▪ String format name = ‘‘Toto’’ ‘‘My name is %s’’ % ( name ) # since 1.x ‘‘My name is {}’’ .format( name ) # since 2.0 f ‘‘My name is {name}’’ # since 3.6

REFACTORING (5/6) ▪ JIT compiler: numba

REFACTORING (6/6) ▪ Cache strategy

TO CONCLUDE

AND NOW… ▪ Migrations are like children: each of them is different ▪ Split in steps … ➢ After each step, TEST!!!! ▪ You will have difficulties but keep hope.

ENABLER CONT CONTACT CT Philippe BOULANGER Python Expertise Manager Philippe.boulanger@invivoo.com PARIS BORDEAUX LONDRES 13, Rue de Rue Lucien Landsdowne House / City Forum www.invivoo.com l’abreuvoir Faure 250 City Road – London EC1V 92400 Courbevoie 33000 2PU www.blog.invivoo.com Bordeaux www.xcomponent.com