Don't repeat yourself CREATIN G ROBUS T P YTH ON W ORK F LOW S - PowerPoint PPT Presentation

Don't repeat yourself CREATIN G ROBUS T P YTH ON W ORK F LOW S Martin Skarzynski Co-Chair, Foundation for Advanced Education in the Sciences (FAES)

What will you learn? Learning Objectives: Develop your own personal work�ow Steps you take Follow best practices T ools you use Use helpful technologies Can evolve over time Write Python code that is easy to read use maintain share CREATING ROBUST PYTHON WORKFLOWS

The DRY principle DRY (Don't Repeat Yourself) # Read in dataset info from text files WET (Waste Everyone's Time) with open('diabetes.txt', 'r') as file: diabetes = file.read() with open('boston.txt', 'r') as file: boston = file.read() with open('iris.txt', 'r') as file: iris = file.read() CREATING ROBUST PYTHON WORKFLOWS

Functions One of the repetitive code blocks: filename parameter represents any possible �lename # Read in diabetes.txt "diabetes.txt" argument with open('diabetes.txt', 'r') as file: a speci�c �lename diabetes = file.read() A function de�nition: A function call: # Define a function to read text files # Use read() to read in diabetes.txt def read(filename): diabetes = read("diabetes.txt") with open(filename, 'r') as file: return file.read() CREATING ROBUST PYTHON WORKFLOWS

Repetitive function calls De�ne a function # Define a function to read text files One with statements instead of three def read(filename): with open(filename, 'r') as file: Three repetitive function calls return file.read() # Use read() to read text files diabetes = read("diabetes.txt") boston = read("boston.txt") iris = read("iris.txt") CREATING ROBUST PYTHON WORKFLOWS

List comprehensions Avoid writing out each function call # Create a list of filenames Use a list comprehension filenames = ["diabetes.txt", "boston.txt", Similar to a for loop: "iris.txt"] # View file contents for f in filenames: # Read files with a list comprehension read(f) file_list = [read(f) for f in filenames] CREATING ROBUST PYTHON WORKFLOWS

Multiple assignment Use multiple assignment # Create a list of filenames Unpack the list filenames = ["diabetes.txt", "boston.txt", Into multiple variables "iris.txt"] # Read files with a list comprehension file_list = [read(f) for f in filenames] diabetes, boston, iris = file_list CREATING ROBUST PYTHON WORKFLOWS

Multiple assignment Use multiple assignment # Create a list of filenames Unpack the list comprehension filenames = ["diabetes.txt", "boston.txt", Into multiple variables "iris.txt"] DRY code! # Read files with a list comprehension diabetes, boston, iris = [read(f) for f in filenames] CREATING ROBUST PYTHON WORKFLOWS

Standard library read_text() method from pathlib import Path Path class # Create a list of filenames Opens and closes �le automatically filenames = ["diabetes.txt", No need for with statements "boston.txt", Not a built-in object "iris.txt"] Must be imported before use # Use pathlib in a list comprehension pathlib module diabetes, boston, iris = [ Standard library Path(f).read_text() Included with Python for f in filenames ] CREATING ROBUST PYTHON WORKFLOWS

Generator expressions T o turn a list comprehension from pathlib import Path Into a generator expression # Create a list of filenames Replace square brackets: [] filenames = ["diabetes.txt", With parentheses: () "boston.txt", "iris.txt"] Generator expression produce generators Generators # Use pathlib in a generator expression Keep track of generated values diabetes, boston, iris = ( Can run out of values Path(f).read_text() for f in filenames ) CREATING ROBUST PYTHON WORKFLOWS

Summary Don't Repeat Yourself (DRY) T ools in our DRY toolbox: Functions (e.g. read() ) Methods (e.g. read_text() ) for loops List comprehensions Generator expressions Python standard library, e.g. pathlib CREATING ROBUST PYTHON WORKFLOWS

Let's practice writing DRY code! CREATIN G ROBUS T P YTH ON W ORK F LOW S

Modularity CREATIN G ROBUS T P YTH ON W ORK F LOW S Martin Skarzynski Co-Chair, Foundation for Advanced Education in the Sciences (FAES)

What is modularity? Independent, reusable objects Each object only has one job Separate code into modules and scripts Modules and scripts Python code �les .py extensions CREATING ROBUST PYTHON WORKFLOWS

Modules versus scripts Modules Scripts Are imported Are run Provide tools Perform actions De�ne functions Call functions The say module: A script: def hello(): import say print("Hello World!") say.hello() CREATING ROBUST PYTHON WORKFLOWS

Function de�nition and calls def hello(): import say print("Hello World!") say.hello() hello() Hello World! Hello World! Hello World! CREATING ROBUST PYTHON WORKFLOWS

Function de�nition and calls def hello(): from say import hello print("Hello World!") hello() hello() Hello World! Hello World! Hello World! CREATING ROBUST PYTHON WORKFLOWS

Module-script hybrid def hello(): from say import hello print("Hello World!") hello() if __name__ == '__main__': hello() Hello World! Hello World! CREATING ROBUST PYTHON WORKFLOWS

The __name__ variable def name(): import say print(__name__) say.name() if __name__ == '__main__': name() say When imported as a module: __main__ __name__ is the module name When run as a script: the if statement code block is skipped __name__ is '__main__' the if statement code block is run CREATING ROBUST PYTHON WORKFLOWS

One function to rule them all from pathlib import Path def do_everything(filename, match): matches = (line for line in Path(filename).open() if match in line) flat = (string for sublist in matches for string in sublist) num_gen = (int(substring) for string in flat for substring in string.split() if substring.isdigit()) return zip(num_gen, num_gen) Many responsibilities: obtain matches, extract numbers etc. CREATING ROBUST PYTHON WORKFLOWS

One job per function def generate_matches(filename, match): return (line for line in Path(filename).open() if match in line) def flatten(nested_list): return (string for sublist in nested_list for string in sublist) def generate_numbers(string_source): return (int(substring) for string in string_source for substring in string.split() if substring.isdigit()) def pair(generator): return zip(generator, generator) CREATING ROBUST PYTHON WORKFLOWS

Iterators iter() def pair(items): iterator = iter(items) turns its input (e.g. list ) return zip(iterator, iterator) into an iterator (e.g. list_iterator ) pairs = list(pair([1, 2, 3, 4])) pairs type(iter([1, 2, 3, 4])) list_iterator [(1, 2), (3, 4)] CREATING ROBUST PYTHON WORKFLOWS

Generators are iterators iter() has no effect on generators: def pair(items): iterator = iter(items) type(iter(x for x in [1, 2, 3, 4])) return zip(iterator, iterator) pairs = list(pair([1, 2, 3, 4])) generator pairs [(1, 2), (3, 4)] CREATING ROBUST PYTHON WORKFLOWS

Adaptable functions Modular functions def pair(items): Adaptable iterator = iter(items) return zip(iterator, iterator) Reusable pairs = list(pair([1, 2, 3, 4])) For example, flatten() can list(flatten(pairs)) Recreate the original list From the pairs variable [1, 2, 3, 4] CREATING ROBUST PYTHON WORKFLOWS

Let's practice writing modular code! CREATIN G ROBUS T P YTH ON W ORK F LOW S

Abstraction CREATIN G ROBUS T P YTH ON W ORK F LOW S Martin Skarzynski Co-Chair, Foundation for Advanced Education in the Sciences (FAES)

Abstraction Hide implementation details Design user interfaces Facilitate code use Car example: Engine Combustion Electric CREATING ROBUST PYTHON WORKFLOWS

Classes T emplates for creating Python objects Represent real-life objects Cat class example: User interface: feed() and rub() methods Implementation details: Feline anatomy Booch, G. et al. Object-Oriented Analysis and Design with Applications . Addison-Wesley, 2007, p. 45. CREATING ROBUST PYTHON WORKFLOWS

Class de�nition from pathlib import Path class TextFile: def __init__(self, file): self.text = Path(file).read_text() The TextFile class Represents any text �le Creates TextFile instances Represent speci�c text �les CREATING ROBUST PYTHON WORKFLOWS

Instantiation TextFile creates instances diabetes = TextFile('diabetes.txt') By passing the file argument diabetes.text[:20] T o the __init__() method def __init__(self, file): '.. _diabetes_dataset' self.text = Path(file).read_text() CREATING ROBUST PYTHON WORKFLOWS

Instance attributes from pathlib import Path class TextFile: def __init__(self, filename): self.text = Path(filename).read_text() self.words = ''.join(c if c.isalpha() else ' ' for c in self.text).split() CREATING ROBUST PYTHON WORKFLOWS

Instance methods from pathlib import Path class TextFile: def __init__(self, filename): self.text = Path(filename).read_text() self.words = ''.join(c if c.isalpha() else ' ' for c in self.text).split() def len_dict(self): return {word: len(word) for word in self.words} CREATING ROBUST PYTHON WORKFLOWS