IN HOSTILE ENVIRONMENTS @kll #PyConBLKN 2018. WHO? Luka Kladaric - - PowerPoint PPT Presentation

▶
in hostile environments
SMART_READER_LITE
LIVE PREVIEW

IN HOSTILE ENVIRONMENTS @kll #PyConBLKN 2018. WHO? Luka Kladaric - - PowerPoint PPT Presentation

Dec 19, 2022 719 likes •1.33k views

SHIPPING QUALITY SOFTWARE IN HOSTILE ENVIRONMENTS @kll #PyConBLKN 2018. WHO? Luka Kladaric Chaos Manager @ Sekura Collective recovering web developer of 10+ years architecture, infrastructure & security consultant also a startup founder

slide-1
SLIDE 1

SHIPPING QUALITY SOFTWARE

IN HOSTILE ENVIRONMENTS

@kll #PyConBLKN 2018.
slide-2
SLIDE 2

WHO?

Luka Kladaric Chaos Manager @ Sekura Collective recovering web developer of 10+ years architecture, infrastructure & security consultant also a startup founder and remote work evangelist

2 — @kll #PyConBLKN 2018.
slide-3
SLIDE 3

HOSTILE ENVIRONMENTS?

3 — @kll #PyConBLKN 2018.
slide-4
SLIDE 4

WHAT IS TECH DEBT?

4 — @kll #PyConBLKN 2018.
slide-5
SLIDE 5

Tech debt is the implied cost of additional rework caused by choosing an easy solution over a longer and better approach.

5 — @kll #PyConBLKN 2018.
slide-6
SLIDE 6

Tech debt is: an API that returns a list of results without pagination

6 — @kll #PyConBLKN 2018.
slide-7
SLIDE 7

Tech debt is: fragile code that everything runs through

7 — @kll #PyConBLKN 2018.
slide-8
SLIDE 8

Tech debt is: entire systems that have become too complex to change or deprecate

8 — @kll #PyConBLKN 2018.
slide-9
SLIDE 9

Tech debt is: parts of the codebase nobody wants to touch

9 — @kll #PyConBLKN 2018.
slide-10
SLIDE 10

Tech debt is: broken development tools and processes lack of confidence in the build and deploy process

10 — @kll #PyConBLKN 2018.
slide-11
SLIDE 11

Tech debt is: everything the team wishes they could change, but can't afford to

11 — @kll #PyConBLKN 2018.
slide-12
SLIDE 12

WHERE DOES IT COME FROM?

12 — @kll #PyConBLKN 2018.
slide-13
SLIDE 13

Insufficient up-front definition Tight coupling of components Lack of attention to the foundations Evolution over time

13 — @kll #PyConBLKN 2018.
slide-14
SLIDE 14

WHAT'S THE HARM?

14 — @kll #PyConBLKN 2018.
slide-15
SLIDE 15

DEVELOPERS ARE JUST SPOILED...

15 — @kll #PyConBLKN 2018.
slide-16
SLIDE 16

FALSE.

16 — @kll #PyConBLKN 2018.
slide-17
SLIDE 17

Unaddressed tech debt breeds more tech debt

17 — @kll #PyConBLKN 2018.
slide-18
SLIDE 18

"We'll get back to that later" "why does X have to be clean, when Y isn't?"

18 — @kll #PyConBLKN 2018.
slide-19
SLIDE 19

Productivity over time decreases Deadlines slip

19 — @kll #PyConBLKN 2018.
slide-20
SLIDE 20

CASE STUDY

20 — @kll #PyConBLKN 2018.
slide-21
SLIDE 21

So I get a call one day.

21 — @kll #PyConBLKN 2018.
slide-22
SLIDE 22

Within a few days, my alarms start going off

22 — @kll #PyConBLKN 2018.
slide-23
SLIDE 23

Massive monolithic git repo

23 — @kll #PyConBLKN 2018.
slide-24
SLIDE 24

No concept of stable

24 — @kll #PyConBLKN 2018.
slide-25
SLIDE 25

Hand-crafted build server

25 — @kll #PyConBLKN 2018.
slide-26
SLIDE 26

No local dev environments Everyone works directly on production systems

26 — @kll #PyConBLKN 2018.
slide-27
SLIDE 27

No db schema migration system or versioning

27 — @kll #PyConBLKN 2018.
slide-28
SLIDE 28

Over 1/2 of the servers not deployable from scratch

28 — @kll #PyConBLKN 2018.
slide-29
SLIDE 29

Code review tool is self-hosted abandonware

29 — @kll #PyConBLKN 2018.
slide-30
SLIDE 30

Outages a daily occurrence

30 — @kll #PyConBLKN 2018.
slide-31
SLIDE 31

Everyone focused on shipping features

31 — @kll #PyConBLKN 2018.
slide-32
SLIDE 32

How do you even begin to fix this?

32 — @kll #PyConBLKN 2018.
slide-33
SLIDE 33

It took over a year and a half.

33 — @kll #PyConBLKN 2018.
slide-34
SLIDE 34

HOW?

34 — @kll #PyConBLKN 2018.
slide-35
SLIDE 35

build server rebuilt from scratch

35 — @kll #PyConBLKN 2018.
slide-36
SLIDE 36

build and deploy jobs defined 100% in code

36 — @kll #PyConBLKN 2018.
slide-37
SLIDE 37

monolithic git repository split up into 40 smaller repositories

37 — @kll #PyConBLKN 2018.
slide-38
SLIDE 38

all servers rebuilt and redeployed with Ansible

38 — @kll #PyConBLKN 2018.
slide-39
SLIDE 39

better code review tool

39 — @kll #PyConBLKN 2018.
slide-40
SLIDE 40

most dev work doesn't require VPN any more

40 — @kll #PyConBLKN 2018.
slide-41
SLIDE 41

etc.

41 — @kll #PyConBLKN 2018.
slide-42
SLIDE 42

JOB WELL DONE!

42 — @kll #PyConBLKN 2018.
slide-43
SLIDE 43

The moral of this story is: don't wait for permission to do your job right.

43 — @kll #PyConBLKN 2018.
slide-44
SLIDE 44
  • 1. If you see something broken, fix it
  • 2. If you don't have time to fix it - write it down
  • 3. But do come back to it when you can steal a minute
  • 4. Even if it takes months to make progress
44 — @kll #PyConBLKN 2018.
slide-45
SLIDE 45

The team was well aware of how broken things were. If we pushed for it to be a single massive project, it would've never happened.

45 — @kll #PyConBLKN 2018.
slide-46
SLIDE 46

EXCEPT...

46 — @kll #PyConBLKN 2018.
slide-47
SLIDE 47

That's not how things should be.

47 — @kll #PyConBLKN 2018.
slide-48
SLIDE 48

How do we do better?

48 — @kll #PyConBLKN 2018.
slide-49
SLIDE 49

"Tech debt" work is difficult to sell

49 — @kll #PyConBLKN 2018.
slide-50
SLIDE 50

It's not like paying off your credit card

50 — @kll #PyConBLKN 2018.
slide-51
SLIDE 51

It's incredibly difficult to schedule work to address tech debt

51 — @kll #PyConBLKN 2018.
slide-52
SLIDE 52

If You Don’t Schedule Time for Maintenance, Your Equipment Will Schedule It for You

52 — @kll #PyConBLKN 2018.
slide-53
SLIDE 53

I recently came across an article that changed the way I think about this

53 — @kll #PyConBLKN 2018.
slide-54
SLIDE 54

Sprints, marathons and root canals by Gojko Adzic

HTTPS://GOJKO.NET/2018/08/30/SPRINTS-MARATHONS-ROOT-CANALS.HTML

54 — @kll #PyConBLKN 2018.
slide-55
SLIDE 55

New name: sustainability work

55 — @kll #PyConBLKN 2018.
slide-56
SLIDE 56

Budget vs planning

56 — @kll #PyConBLKN 2018.
slide-57
SLIDE 57

Helps with morale

57 — @kll #PyConBLKN 2018.
slide-58
SLIDE 58

QUESTIONS?

@kll #PyConBLKN 2018.
slide-59
SLIDE 59

THANK YOU!

Luka Kladaric twitter: @kll luka@sekura.io www.sekura.io

@kll #PyConBLKN 2018.