Visual Debugger for Jupyter Notebooks: Myth or Reality? Elizaveta - - PowerPoint PPT Presentation

▶
visual debugger for jupyter notebooks myth or reality
SMART_READER_LITE
LIVE PREVIEW

Visual Debugger for Jupyter Notebooks: Myth or Reality? Elizaveta - - PowerPoint PPT Presentation

Sep 19, 2023 210 likes •893 views

Visual Debugger for Jupyter Notebooks: Myth or Reality? Elizaveta Shashkova EuroPython 2019 About Me Software Developer at JetBrains, PyCharm IDE Debugger and Data Science tools @lisa_shashkova 2 Visual Debugger 3

slide-1
SLIDE 1

Visual Debugger for Jupyter Notebooks: Myth or Reality?

Elizaveta Shashkova
 EuroPython 2019

slide-2
SLIDE 2

About Me

  • Software Developer at JetBrains, PyCharm IDE
  • Debugger and Data Science tools
  • @lisa_shashkova

2

slide-3
SLIDE 3

Visual Debugger

3

slide-4
SLIDE 4

Jupyter Notebooks

  • Popular scientific tool
  • File is a sequence of cells

4

slide-5
SLIDE 5

Jupyter Notebooks Debug

5

  • Logging with print statements
  • Command-line debugger ipdb
slide-6
SLIDE 6

Jupyter Notebooks Debug

6

slide-7
SLIDE 7

Myth or Reality?

slide-8
SLIDE 8

Myth or Reality?

slide-9
SLIDE 9

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

9

slide-10
SLIDE 10

Tracing Function

10

def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc) 1 2 3 4 5 6

slide-11
SLIDE 11

Tracing Function

11

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors() 1 2 3 4 5 6 7 8 9

slide-12
SLIDE 12

Tracing Function

12

1 2 3 4 5 6 7 8 9

1 call

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()

slide-13
SLIDE 13

Tracing Function

13

1 2 3 4 5 6 7 8 9

1 call 2 line

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()

slide-14
SLIDE 14

Tracing Function

14

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Mars!

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()

slide-15
SLIDE 15

Tracing Function

15

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus!

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()

slide-16
SLIDE 16

Tracing Function

16

1 2 3 4 5 6 7 8 9 def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()

1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus! 5 line 5 return

slide-17
SLIDE 17

Breakpoint

  • frame.f_lineno - current line number
  • frame.f_code.co_filename - current file name

17

slide-18
SLIDE 18

Breakpoint

  • frame.f_lineno - current line number
  • frame.f_code.co_filename - current file name
  • Equals to breakpoint’s file and line -> suspend

program!

18

slide-19
SLIDE 19

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

19

slide-20
SLIDE 20

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

20

slide-21
SLIDE 21

Cells Execution

21

IPython kernel Front-end IDE

slide-22
SLIDE 22

Cells Execution

22

code IPython kernel Front-end IDE

slide-23
SLIDE 23

IPython kernel

Cells Execution

23

code execution Front-end IDE

slide-24
SLIDE 24

Cells Execution

24

result IPython kernel Front-end IDE

slide-25
SLIDE 25

Cells Execution

  • Kernel generates a unique name for each cell
  • <ipython-input-5-11faed10a894>
  • File name of a generated 


code object

25

IPython kernel code execution

slide-26
SLIDE 26

Jupyter Breakpoints

  • Python files: (filename, line number) -> unique

location

26

slide-27
SLIDE 27

Jupyter Breakpoints

  • Python files: (filename, line number) -> unique

location

  • Jupyter Notebooks?

27

slide-28
SLIDE 28

Jupyter Breakpoints

  • Python files: (filename, line number) -> unique

location

  • Jupyter Notebooks:
  • generated cell name
  • line inside code object

28

slide-29
SLIDE 29

Source Mapping

29

IPython kernel IDE

cell source code MyNotebook.ipynb

generated <code object>

slide-30
SLIDE 30

Source Mapping

30

IPython kernel IDE

cell source code MyNotebook.ipynb

cell id generated <code object>

slide-31
SLIDE 31

IPython kernel IDE

Source Mapping

31

?

cell source code MyNotebook.ipynb

cell id generated <code object>

slide-32
SLIDE 32

Source Mapping

32

  • Tracking cells execution in the IDE
slide-33
SLIDE 33

Source Mapping

33

  • Tracking cells execution in the IDE
  • Silent cell execution in IPython kernel
slide-34
SLIDE 34

Debug Cell Execution

34

<cell source code>

slide-35
SLIDE 35

Debug Cell Execution

35

<cell source code> patch name generation

  • silent mode
slide-36
SLIDE 36

Debug Cell Execution

36

<cell source code> patch name generation cell id

  • silent mode
slide-37
SLIDE 37

Jupyter Tracing Function

  • frame.f_code.co_filename - generated name

37

slide-38
SLIDE 38

Jupyter Tracing Function

  • frame.f_code.co_filename - generated name
  • Map: generated name -> cell id

38

slide-39
SLIDE 39

Jupyter Tracing Function

  • frame.f_code.co_filename - generated name
  • Map: generated name -> cell id
  • Send message to the IDE

39

slide-40
SLIDE 40

Jupyter Tracing Function

  • frame.f_code.co_filename - generated name
  • Map: generated name -> cell id
  • Send message to the IDE

40

slide-41
SLIDE 41

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

41

slide-42
SLIDE 42

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

42

slide-43
SLIDE 43

Debug Communication

43

“Add breakpoint in a cell 3, line 2” IPython kernel Front-end IDE

slide-44
SLIDE 44

Debug Communication

44

“Add breakpoint in a cell 3, line 2” IPython kernel Front-end IDE

slide-45
SLIDE 45

Debug Communication

45

  • Additional connection
  • Reuse Jupyter channels
slide-46
SLIDE 46

Jupyter Messaging

46

Front-end IPython kernel Kernel 
 proxy

slide-47
SLIDE 47

Jupyter Messaging

47

Front-end IPython kernel Kernel 
 proxy

slide-48
SLIDE 48

Debug Communication

48

  • Additional connection
  • Reuse Jupyter channels
slide-49
SLIDE 49

Debug Communication

49

  • Additional connection
  • Reuse Jupyter channels
slide-50
SLIDE 50

Jupyter Messaging

50

Front-end Kernel 
 proxy IPython kernel Shell IOPub stdin

slide-51
SLIDE 51

Jupyter Architecture

51

  • Event loop in a main thread for execution events
  • Event loop for output events
slide-52
SLIDE 52

Jupyter Architecture

52

code debug x Blocked IPython kernel Front-end IDE

slide-53
SLIDE 53

Debug Communication

53

  • Additional connection
  • Reuse Jupyter channels
slide-54
SLIDE 54

Debug Communication

54

  • Additional connection
  • Reuse Jupyter channels

But ipdb works!

slide-55
SLIDE 55

But ipdb Works!

55

slide-56
SLIDE 56

But ipdb Works!

56

  • Based on input()
  • Reuses user input channel
slide-57
SLIDE 57

Debug Communication

57

  • Additional connection
  • Reuse Jupyter channels
slide-58
SLIDE 58

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

58

slide-59
SLIDE 59

Contents

  • Python files debugging
  • Jupyter breakpoints
  • Debugger communication
  • Jupyter visual debugger

59

slide-60
SLIDE 60

Jupyter Visual Debugger

  • Jupyter tracing function

60

slide-61
SLIDE 61

Jupyter Visual Debugger

  • Jupyter tracing function
  • Mapping between editor and generated code

61

slide-62
SLIDE 62

Jupyter Visual Debugger

  • Jupyter tracing function
  • Mapping between editor and generated code
  • Debugger connection

62

slide-63
SLIDE 63

Live Demo

63

slide-64
SLIDE 64

Live Demo

64

  • PyCharm doesn’t convert Jupyter Notebooks to

Python files!

  • On disk it’s still the same JSON file with .ipynb

extension

slide-65
SLIDE 65

Jupyter Visual Debugger

  • Jupyter tracing function
  • Mapping between editor and generated code
  • Debugger connection

65

slide-66
SLIDE 66

Jupyter Visual Debugger

  • Implement in your favourite IDE

66

slide-67
SLIDE 67

Jupyter Visual Debugger

  • Implement in your favourite IDE
  • Try it in PyCharm Pro!

67

slide-68
SLIDE 68

Jupyter Visual Debugger

  • Implement in your favourite IDE
  • Try it in PyCharm Pro!
  • Questions?

68

@lisa_shashkova