Re-structuring a giant, ancient code-base or: Making LibreOffice - - PowerPoint PPT Presentation

▶
re structuring a giant ancient code base
SMART_READER_LITE
LIVE PREVIEW

Re-structuring a giant, ancient code-base or: Making LibreOffice - - PowerPoint PPT Presentation

Feb 14, 2023 180 likes •634 views

Re-structuring a giant, ancient code-base or: Making LibreOffice work well everywhere Michael Meeks General Manager at Collabora Productivity michael.meeks@collabora.com Skype - mmeeks, G+ - mejmeeks@gmail.com Stand at the crossroads and

slide-1
SLIDE 1

1 / 45 FOSDEM 2018 | Michael Meeks

Michael Meeks

General Manager at Collabora Productivity michael.meeks@collabora.com

Skype - mmeeks, G+ - mejmeeks@gmail.com

Re-structuring a giant, ancient code-base

  • r: Making LibreOffice work well everywhere

“Stand at the crossroads and look; ask for the ancient paths, ask where the good way is, and walk in it, and you will find rest for your souls...” - Jeremiah 6:16

www.collaboraoffice.com

slide-2
SLIDE 2

2

2 / 45 FOSDEM 2018 | Michael Meeks

Re-structuring an overview

  • Re-structuring for new platforms
  • Making code readable
  • Parallelising / threading: OCL / Threading
  • Drawing & Widgets
  • primitives, gtk3/cairo, OpenGL, harfbuzz
  • Security work: coverity, etc.
  • Unit tests: protecting the backside ...
  • Online
  • memory sharing
  • dialog tunnelling
  • LibreOffice 6.0 bits
  • Get involved
slide-3
SLIDE 3

3

3 / 45 FOSDEM 2018 | Michael Meeks

Why re-structure at all ?

  • A 30+ year old code-base
  • Object Orientation mania

– The silver-bullet

  • OO toolkit – less interesting
  • Demo / Office apps

very interesting. →

  • What languages should we use ?
  • What platforms should we target ?
  • “Technology – the only industry that is more

fashion driven than women’s fashion”

slide-4
SLIDE 4

4

4 / 45 FOSDEM 2018 | Michael Meeks

Language choices:

Tip#1 - don’t buy a silver bullet: "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity." Fred Brooks

slide-5
SLIDE 5

5

5 / 45 FOSDEM 2018 | Michael Meeks

Sample 10x productivity wins

  • Object Oriented programming !
  • Java

develop 10x faster →

  • Garbage collection ! It rocks.
  • C#

develop 10x faster →

  • Syntactic sugar! It rocks.
  • "As such, whilst Vala is a modern language

with all of the features you would expect, it gains its power from …"

  • Language augmenting pre-processors are great
  • And on … and on.
slide-6
SLIDE 6

6

6 / 45 FOSDEM 2018 | Michael Meeks

A personal favourite

“We encourage the OpenOffice group to quickly build their version of a spread sheet or a word app using JavaFX," Ellison (according to theregister) - June 2009

  • Fatal mistake: using a unique name
  • Tip#2 – your vanity language / platform project should

have a non-google-trend-able name eg. ‘ruby’, ‘rust’, ‘go’ - not eg. ‘zsquat’

  • JavaFX’s Classic Hype-Cycle trajectory / google trend

June 2009 Slope of suffering Peak of Inflated Expectations Plateau of questing for newer peaks ...

slide-7
SLIDE 7

7

7 / 45 FOSDEM 2018 | Michael Meeks

Lucking out on language ?

"The point I'm trying to make is that the only reason why all of this was possible, where we build ... [ Microsoft Office ] ... for 30 years, not only ship those applications but evolve them - picked C/C++ - tremendously lucked out. All the other languages - two categories:

  • 1. language would have been re-invented 3 times

and had to re-write the thing 3 times.

  • 2. authors of language & industry lost interest and

it would have stagnated"

– Igor Zaika (Microsoft, Distinguished Eng.) CppCon 2014

slide-8
SLIDE 8

8

8 / 45 FOSDEM 2018 | Michael Meeks

Language bits.

  • Language bigotry:
  • Almost totally pointless
  • The leading driver of pointless duplication

– Over even license choice … ?

  • Lots of it out there.
  • We’re still slowly writing out Java
  • can’t guarentee it is on the platform

in the same way that binaries are.

  • The embedded Rhino JS-in-Java impl ? …
  • Default HSQLDB format

in databases ... →

slide-9
SLIDE 9

9

9 / 45 FOSDEM 2018 | Michael Meeks

Cross Platform-ness: Churn

  • Windows rendering APIs:
  • GDI: basic version with

windows 1.0~1985

– Made the 16bit

64bit → transition well.

  • GDI+ in XP - 2001
  • DirectWrite Win7/2007
  • finally a physical font API.
  • Direct2D: 2012
  • Easy to churn faster than

big apps can handle

  • Linux toolkits
  • gtk+1, Qt2 – 1999
  • gtk+2, Qt3 – 2000
  • Qt4 2005
  • gtk+3 2011
  • Qt5 2012
  • Apple: churn to

match

  • Bold: in-use ...
slide-10
SLIDE 10

10

10 / 45 FOSDEM 2018 | Michael Meeks

Form factor changes

  • Which changes to sit-out ?
  • PC

mobile netbook tablet → → → → web/browser watch VR audio → → → assistant: nice ...

“I’m afraid I can’t save your document Dave”

  • More trivially:
  • Threads – the world is filling

with them thanks to AMD:

slide-11
SLIDE 11

11 / 45 FOSDEM 2018 | Michael Meeks

2017 Crash reporting stats

  • Frustratingly ‘cores’ not threads.

2 1 7

  • 1
  • 1

2 1 7

  • 2
  • 1

2 1 7

  • 3
  • 1

2 1 7

  • 4
  • 1

2 1 7

  • 5
  • 1

2 1 7

  • 6
  • 1

2 1 7

  • 7
  • 1

2 1 7

  • 8
  • 1

2 1 7

  • 9
  • 1

2 1 7

  • 1
  • 1

2 1 7

  • 1

1

  • 1

2 1 7

  • 1

2

  • 1

2 1 8

  • 1
  • 1

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00%

Crash report % by CPU core count over time.

48 36 32 24 16 12 10 8 6 4 2 1 200 400 600 800 1000 1200 1400 1600 1800 2000

Reports from large core count machines.

48 40 36 32 24 16 12 10

Thanks to Markus Mohrhard

slide-12
SLIDE 12

Threading

slide-13
SLIDE 13

13 / 45 FOSDEM 2018 | Michael Meeks

Newish Threading ...

  • CPU / Image scaling
  • Rendering / Rasterizing primitives
  • Thanks to Armin Le Grand (CIB)
  • Calc core
  • Thanks to Tor Lillqvist & Dennis Francis

(Collabora)

  • XML parsing: expansion to ODF filters
  • Constant time XML parsing (with free threads,

and ~slower consumer)

  • Thanks to Mohammed Abdul Azeem
slide-14
SLIDE 14

14 / 45 FOSDEM 2018 | Michael Meeks

Threaded XML parsing ...

  • Parsing XML is a significantly costly pain
  • Format is needlessly complex; cf. JSON.
  • Namespace handling & tokenization is a pain.
  • Double checking for duplicate attributes bad too …
  • SAX API really poor for CPU cache usage:

– a heavily fragmented workload.

XML XML parser Tokenizer Swing Buffers SAX event emission Core

slide-15
SLIDE 15

15 / 45 FOSDEM 2018 | Michael Meeks

Calc: extreme coupling & threads.

ScDocument ScTable ScFormulaCell block Broadcasters ScBroadcastAreaSlotMachine ScColumn

Dependencies Dependencies

ScInterpreter

ScTokenArray ScFormulaCellGroup … Tokens … RPN Mutates: INDEX, OFFSET etc. Cloud Web fn’s Macros Ext’ns Mutates! Vlookup Cache

Number format, Link mgmt etc.

slide-16
SLIDE 16

16 / 45 FOSDEM 2018 | Michael Meeks

single1 2 4 8 16

0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 re-calculating 100k formulae on 1m doubles

Meeks/Linux Ryzen/Win10 Thread count Seconds to calculate

Calc parallelism results ...

  • Extreme care & choice
  • f threading granularity
  • Mostly lock-less, but lots
  • f assertions.
  • Would love more

language support to help; C++ weak here.

  • Some good

improvements

  • Plenty more to do to

expand the scope & reliability

slide-17
SLIDE 17

Cleanups …

slide-18
SLIDE 18

18 / 45 FOSDEM 2018 | Michael Meeks

Tastier code – easier to chew

  • Finally finished quality

translation in 5.4 of all German comments.

  • Thanks to:

Jens Carl, Johnny_M, Michael Stahl (RedHat), Katerina Behrens (CIB), Thomas Beck (CIB), Lukas Röllin, Gabor Kelemen, Stephan Bergmann (RedHat), dennisroczek, and many others.

10,000 20,000 30,000 40,000 50,000 60,000

Detected lines of German comment

slide-19
SLIDE 19

19 / 45 FOSDEM 2018 | Michael Meeks

Less eye-strain horror

  • Ongoing enum cleanup & strings ...
  • case BUTTON_ABORT: aText = rtl::OUString(
  • RTL_CONSTASCII_USTRINGPARAM( "Abort" ) );

+case StandardButtonType::Abort: aText = "Abort";

  • Pretty iterators & auto
  • for( std::unordered_map< Atom, Selection*

>::iterator it = This->m_aSelections.begin(); it != This->m_aSelections.end(); ++it ) + for (auto const& selection : This→m_aSelections)

  • And lots more.
slide-20
SLIDE 20

20 / 45 FOSDEM 2018 | Michael Meeks

Clang plugins ...

  • Expanding C++ checking:
  • unexpected bool conversion hunting
  • flatten: look for large if statement at end of function
  • cast cleanups / re-writing
  • Wider scope of understanding
  • Unused-fields checking
  • count all users of default params
  • un-necessary-virtual detection
  • Use-unique-ptr
  • And tons more: thanks to
  • Noel Grandin (Collabora) & Stephan Bergman (RedHat)
slide-21
SLIDE 21

21 / 45 FOSDEM 2018 | Michael Meeks

Drawing & Widgets ...

  • Harfbuzz
  • Kills Uniscribe (Win), and CoreText (Apple)
  • Single, cross-platform, FLOSS shaping

– Thanks to Khaled Hosny & TDF donors

  • Share glyph rendering too next ? Print...
  • Drawinglayer Primitives for borders:
  • Thanks to Armin Le Grand (CIB)
slide-22
SLIDE 22

22 / 45 FOSDEM 2018 | Michael Meeks

Gtk3: Native Widgets ...

  • Native Widget theming
  • Used for years on Win/Mac/Linux
  • Torture widgets into rendering bits of themselves
  • Missing: theming details, animations
  • Plus Native Menu-bar, File-selector, Tool-tips.
  • Finished converting all dialogs to glade / XML

years back: 977 .ui files ...

  • Now starting to load native gtk+ dialogs,

widgets & wrapping API – WxWidgets style

  • Thanks to Caolan McNamara (RedHat)
slide-23
SLIDE 23

23 / 45 FOSDEM 2018 | Michael Meeks

Video

slide-24
SLIDE 24

24 / 45 FOSDEM 2018 | Michael Meeks

Core Infrastructure Initiative

  • OSS-Fuzz
  • Huge Testing infrastructure provided by

Google

– Used for Chrome & many other OSS projects. – ~1000 core cluster to hugely accelerate testing.

  • Manual auditing – killed by AI ?

– Finds newly introduced problems rapidly

before they can escape (CVEs)

  • Coverity – still a score of ~0.00
  • Thanks to Caolan McNamara (RedHat)
slide-25
SLIDE 25

Quality Impact ?

slide-26
SLIDE 26

26 / 45 FOSDEM 2018 | Michael Meeks

Avoiding deja-vu in bug-fixing

  • Fix each bug just once;

Thanks to: Miklos Vajna (Collabora), Stephan Bergmann (RedHat), Noel Grandin (Collabora), Zden k ě Crhonek, Caolán McNamara (RedHat), Ashod Nakashian (Collabora), Justin Luth (SIL), Tamás Zolnai (Collabora), Andrea Gelmini, Jan Holesovsky (Collabora), Markus Mohrhard, Eike Rathke (RedHat), Mike Kaganski (Collabora), Jens Carl, Michael Stahl (RedHat), Szymon Kłos (Collabora), Tor Lillqvist (Collabora), Chris Sherlock, Pranav Kant (Collabora), Winfried Donkers, David Tardon (RedHat)

with >20 unit test commits in 5.4/6.0

3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2 5.3 5.4 6.0 5,000 10,000 15,000 20,000 25,000 30,000

Growth in unit tests over time

count of various CPPUNIT macros

Asserts Tests

slide-27
SLIDE 27

27 / 45 FOSDEM 2018 | Michael Meeks

Re-factor to cause regressions ?

Commits: $ git log --oneline –since ‘2016-01-01' | wc -l 33,040 Open regressions: +142 (0.4% of commits) Not for free: lots of work +2000 closed regressions in that time Many of which didn’t escape – thanks to the QA team

2016-01-05 2016-01-27 2016-02-16 2016-03-10 2016-03-29 2016-04-18 2016-05-17 2016-06-07 2016-06-28 2016-07-19 2016-08-11 2016-09-15 2016-10-04 2016-10-26 2016-11-14 2016-12-06 2017-01-03 2017-01-24 2017-02-21 2017-03-16 2017-04-04 2017-04-26 2017-05-16 2017-06-06 2017-06-27 2017-07-18 2017-08-08 2017-08-15 2017-09-05 2017-09-26 2017-10-17 2017-10-31 2017-11-21 2017-12-12 2018-01-02 2018-01-23 100 200 300 400 500 600 700 800 900 1000

Open regression bugs 2016 - today ...

sdk graphics stack framework filters and storage UI Printing and PDF export Linguistic Installation Impress Remote Formula Editor Extensions Chart Migration Writer Writer/RTF Basic Crashes Borders LibreOffice Drawing Database Presentation Spreadsheet

slide-28
SLIDE 28

28 / 45 FOSDEM 2018 | Michael Meeks

2016-11-02 2016-11-22 2016-12-13 2017-01-11 2017-01-31 2017-02-21 2017-03-16 2017-04-04 2017-04-26 2017-05-16 2017-06-06 2017-06-27 2017-07-18 2017-08-08 2017-08-15 2017-09-05 2017-09-26 2017-10-17 2017-10-31 2017-11-21 2017-12-12 2018-01-02 2018-01-23 2 4 6 8 10 12 14 16 18

Open High Priority Regressions

sdk graphics stack framework filters and storage UI Printing and PDF export Linguistic Installation Impress Remote Formula Editor Extensions Chart Migration Writer Writer/RTF Basic Crashes Borders LibreOffice Drawing Database Presentation Spreadsheet

  • Under 10 high priority ones
  • Not escaping.
  • Some bug fixes: rob Peter to pay Paul.
  • guarenteed: change a CPU/memory trade-off: bingo.

And how serious are they ?

slide-29
SLIDE 29

Online – a rather different platform

slide-30
SLIDE 30
slide-31
SLIDE 31

31

31 / 45 FOSDEM 2018 | Michael Meeks

Different optimizations

  • Online
  • dlopens ~world (RTLD_BIND_NOW)
  • Loads ~all dictionaries & hyphenation
  • then forks children

– Copy-on-write-ness everywhere.

  • waste memory in pre-init to save in children
  • Waste CPU time on startup to save it later.
  • Catching bad behaviour:
  • Touching / un-sharing pages:
slide-32
SLIDE 32

32

32 / 45 FOSDEM 2018 | Michael Meeks

Compare memory: loolmap

  • Useful tool for eg. kdeinit, webservers?

heap page: 0x011e9000 (310/967) - touched: 17 - was shared - from [heap] ... 0x0130 03 00 00 00 0a 00 00 00 4d 00 61 00 74 00 68 00 | ........M.a.t.h.

  • par't- 1 |

0x0140 4d 00 4c 00 20 00 32 00 2e 00 30 00 00 00 65 00 | M.L. .2...0...e. 0x0150 00 00 00 00 00 00 00 00 61 00 00 00 00 00 00 00 | ........a.......

Ref-count in a Unicode string incremented

  • Also dumps all strings it can find with –strings
  • Totals for heap
  • shared 23040 (92160kB)

unshared 982 (3928kB) same but unshared 191 (764kB) dirtied bytes touched 30718 per page 31.28

Drove pre-init string staticization

slide-33
SLIDE 33

33

33 / 45 FOSDEM 2018 | Michael Meeks

Memory corner cases:

pixman/pixman-glyph.c (pixman_glyph_insert) - 40Mb /* XXX: These numbers are arbitrary---we've never done any measurements. #define N_GLYPHS_HIGH_WATER (16384) #define N_GLYPHS_LOW_WATER (8192)

  • How many glyph bitmaps to keep around.

cairo/src/cairo-scaled-font.c - likewise /* XXX: This number is arbitrary---we've never done any measurement of this. */ #define MAX_GLYPH_PAGES_CACHED 512

  • Need control: ideally pre-render common glyphs

patching pixman / cairo ? ...

slide-34
SLIDE 34

34

34 / 45 FOSDEM 2018 | Michael Meeks

Tunnelling dialogs ...

  • Encouraging progress here for 6.1
  • Rich dialogs
  • Deep function.
  • Collaborative:
  • A. Edit border
  • B. Edit color
  • C. Edit size
  • Applies right.
  • Modal dialogs
slide-35
SLIDE 35

Small selection of 6.0 new features

slide-36
SLIDE 36

36

36 / 45 FOSDEM 2018 | Michael Meeks

Documents

  • Mail merge from writer tables … (Miklos Vajna –

Collabora)

  • Image rotation (Armin Le Grand – CIB)
  • EPUB3 export filter (Miklos Vajna)
  • OpenPGP document sign & encrypt (Katarína

Behrens, Samuel Mehrbrodt, Thorsten Behrens - CIB)

slide-37
SLIDE 37

37

37 / 45 FOSDEM 2018 | Michael Meeks

Filters

  • QuarkXPress v3 & 4 import via libqxp from DLP

(Alex Pantechovskis)

  • Improved SmartArt import filter (Grzegorz

Araminowicz)

  • ActiveX import/export to/from

DOCX (Tamas Zolnai – Collabora)

  • PPTX – export embedded videos,

and round-trip macros (Kendy – Collabora)

  • Many EMF+ filter fixes (Bartosz Kosiorek¸Patrick

Jaap)

slide-38
SLIDE 38

38

38 / 45 FOSDEM 2018 | Michael Meeks

Quick GUI selection ...

  • Faster Insert Special Characters

(Akshay Deep)

  • Improved Notebookbar

(Andreas Kainz)

slide-39
SLIDE 39

39

39 / 45 FOSDEM 2018 | Michael Meeks

Other bits

  • We dropped support for Windows XP
  • And Windows Vista … at last.
  • Online (from Collaborans)
  • Seccomp-bpf, Avatars, Spell-checking
  • Interactive horizontal ruler
  • Calc rows

500k. →

  • Calc: active cell reference
  • Basic chart editing ...
slide-40
SLIDE 40

40

40 / 45 FOSDEM 2018 | Michael Meeks

Android version (Ximeng Zu)

  • Insert pictures from camera:
  • Configure rows:
  • Insert, delete, hide
  • Presentation mode

IOS progress: thanks to Jan Iversen & Jon Nermut

slide-41
SLIDE 41

Getting involved.

slide-42
SLIDE 42

42

42 / 45 FOSDEM 2018 | Michael Meeks

Getting involved

  • Something for everyone:
  • Testing – just run the latest builds / triage
  • Documentation – help mend the manuals
  • Ask – see if you can answer a question.
  • Developing – poke at an easy hack

– Python, Javascript, C++ tasks … – From the very simplest to something ‘fun’

  • Design / Art – join a team call

https://www.libreoffice.org/community/get-involved/

slide-43
SLIDE 43

43

Nurture scaleable development

  • Try working a problem at doesn’t fit your head
  • These are the ‘real’ engineering problems
  • nce you

can solve them you’re a star.

  • Checkout

my paper

  • n the

topic.

slide-44
SLIDE 44

44

The future ...

  • We need smart people to help us
  • Create the future:
  • An Artificial Intelligence
  • f stunning subtlety
  • f conversational brilliance
  • That can understand what

you’re doing

  • And can suggest cool new

things to do.

  • Trained on Documents (TM)
slide-45
SLIDE 45

45

45 / 45 FOSDEM 2018 | Michael Meeks

Conclusions

  • Forced platform change is tough on big code-bases
  • Re-structuring & re-factoring
  • fun – and risky: today seems hard to justify.
  • stopping re-factoring is also risky.
  • LibreOffice is surviving the volume of

change & improving

  • Online – is great, free your data.
  • New contributors are always welcome : get stuck in
  • Thanks for supporting LibreOffice !

Oh, that my words were recorded, that they were written on a scroll, that they were inscribed with an iron tool on lead, or engraved in rock for ever! I know that my Redeemer lives, and that in the end he will stand upon the earth. And though this body has been destroyed yet in my flesh I will see God, I myself will see him, with my own eyes - I and not

  • another. How my heart yearns within me. - Job 19: 23-27