Prex: Finding Guidance for Forward and Backward Porting of Linux - - PowerPoint PPT Presentation

Prex: Finding Guidance for Forward and Backward Porting of Linux Device Drivers

Julia Lawall, Derek Palinski, Lukas Gnirke, Gilles Muller (Inria/LIP6) January 12, 2017

(Unpublished work)

1

The porting problem

4.4 driver

2

The porting problem

4.5 driver

3

The porting problem

4.6 driver

4

The porting problem

4.6 forward ported driver

5

The porting problem

4.0 forward ported driver

6

The porting problem

4.0 back ported driver

7

Porting methodology

• 1. Compile the driver with the target kernel.

8

Porting methodology

• 1. Compile the driver with the target kernel.

– See what breaks.

9

Porting methodology

• 1. Compile the driver with the target kernel.

– See what breaks. – What breaks is in the interface with the kernel.

10

Porting methodology

• 1. Compile the driver with the target kernel.

– See what breaks. – What breaks is in the interface with the kernel.

• 2. Study commits in the code history that fix each breakage.

11

Porting methodology

• 1. Compile the driver with the target kernel.

– See what breaks. – What breaks is in the interface with the kernel.

• 2. Study commits in the code history that fix each breakage.

– Other drivers have faced the same issues.

12

Porting methodology

• 1. Compile the driver with the target kernel.

– See what breaks. – What breaks is in the interface with the kernel.

• 2. Study commits in the code history that fix each breakage.

– Other drivers have faced the same issues.

• 3. Apply the observed changes to port the driver code.

13

Porting example

• lms501kf03 TFT LCD panel driver
• Introduced in Linux 3.9 (commit 1be9ca2, 2013)
• Want to port it to Linux 4.6 (2016)

14

Step 1: Compilation result (gcc)

lms501kf03.c:433:2: error: unknown field ‘suspend’ specified in initializer .suspend = lms501kf03_suspend, ^ lms501kf03.c:433:2: warning: missing braces around initializer lms501kf03.c:433:2: warning: (near initialization for ‘lms501kf03_driver.driver’) lms501kf03.c:433:2: warning: initialization from incompatible pointer type lms501kf03.c:433:2: warning: (near initialization for ‘lms501kf03_driver.driver.name’) lms501kf03.c:434:2: error: unknown field ‘resume’ specified in initializer .resume = lms501kf03_resume, ^ lms501kf03.c:434:2: warning: excess elements in struct initializer lms501kf03.c:434:2: warning: (near initialization for ‘lms501kf03_driver’)

Challenge: Many messages are triggered by earlier failures and may be inconsistent.

15

Step 1: Compilation result (gcc)

lms501kf03.c:433:2: error: unknown field ‘suspend’ specified in initializer .suspend = lms501kf03_suspend, ^ lms501kf03.c:433:2: warning: missing braces around initializer lms501kf03.c:433:2: warning: (near initialization for ‘lms501kf03_driver.driver’) lms501kf03.c:433:2: warning: initialization from incompatible pointer type lms501kf03.c:433:2: warning: (near initialization for ‘lms501kf03_driver.driver.name’) lms501kf03.c:434:2: error: unknown field ‘resume’ specified in initializer .resume = lms501kf03_resume, ^ lms501kf03.c:434:2: warning: excess elements in struct initializer lms501kf03.c:434:2: warning: (near initialization for ‘lms501kf03_driver’)

Challenge: Many messages are triggered by earlier failures and may be inconsistent.

16

Step 1: Compilation result (gcc)

lms501kf03.c:433:2: error: unknown field ‘suspend’ specified in initializer .suspend = lms501kf03_suspend, ^ lms501kf03.c:433:2: warning: missing braces around initializer lms501kf03.c:433:2: warning: (near initialization for ‘lms501kf03_driver.driver’) lms501kf03.c:433:2: warning: initialization from incompatible pointer type lms501kf03.c:433:2: warning: (near initialization for ‘lms501kf03_driver.driver.name’) lms501kf03.c:434:2: error: unknown field ‘resume’ specified in initializer .resume = lms501kf03_resume, ^ lms501kf03.c:434:2: warning: excess elements in struct initializer lms501kf03.c:434:2: warning: (near initialization for ‘lms501kf03_driver’)

Challenge: Many messages are triggered by earlier failures and may be inconsistent.

17

Step 2: Study previous commits

• Commit 9d9f780, from January 2015.
• Most recent one treating relevant suspend and resume fields.

18

Step 2: Study previous commits

@@

• 273,9 +273 ,9 @@

#ifdef CONFIG_PM_SLEEP

• static

int as3935_suspend (struct spi_device *spi , pm_message_t msg) +static int as3935_suspend (struct device *dev) {

• struct

iio_dev *indio_dev = spi_get_drvdata (spi ); + struct iio_dev *indio_dev = dev_get_drvdata (dev ); @@

• 293,9 +293 ,9 @@
• static

int as3935_resume (struct spi_device *spi) +static int as3935_resume (struct device *dev) {

• struct

iio_dev *indio_dev = spi_get_drvdata (spi ); + struct iio_dev *indio_dev = dev_get_drvdata (dev ); @@

• 311,9 +311 ,12 @@

+static SIMPLE_DEV_PM_OPS (as3935_pm_ops , as3935_suspend , as3935_resume ); +#define AS3935_PM_OPS (& as3935_pm_ops ) #else

• #define

as3935_suspend NULL

• #define

as3935_resume NULL +#define AS3935_PM_OPS NULL #endif @@

• 441,12 +444 ,11 @@

.driver = { ... + .pm = AS3935_PM_OPS , }, ...

• .suspend

= as3935_suspend ,

• .resume

= as3935_resume , 19

Studying the commit in more detail

@@

• 441,12 +444 ,11 @@

.driver = { .name = "as3935", .owner = THIS_MODULE , + .pm = AS3935_PM_OPS , }, .probe = as3935_probe , .remove = as3935_remove , .id_table = as3935_id ,

• .suspend

= as3935_suspend ,

• .resume

= as3935_resume , };

• Removes suspend and resume as required.
• The commit does other things, so some compensation seems

needed for the removal.

20

Studying the commit in more detail

@@

• 273,9 +273 ,9 @@

#ifdef CONFIG_PM_SLEEP static int

• as3935_suspend (struct

spi_device *spi , pm_message_t msg) + as3935_suspend (struct device *dev) {

• struct

iio_dev *indio_dev = spi_get_drvdata (spi ); + struct iio_dev *indio_dev = dev_get_drvdata (dev );

• as3935 suspend was stored in the removed suspend field.

– Change parameter list. – Change function body to compensate.

• Similar changes for as3935 resume.

21

Studying the commit in more detail

@@

• 311,9 +311 ,12 @@

+static SIMPLE_DEV_PM_OPS (as3935_pm_ops , as3935_suspend , + as3935_resume ); +#define AS3935_PM_OPS (& as3935_pm_ops ) #else

• #define

as3935_suspend NULL

• #define

as3935_resume NULL +#define AS3935_PM_OPS NULL #endif @@

• 441,12 +444 ,11 @@

.driver = { ... + .pm = AS3935_PM_OPS , }, ...

• .suspend

= as3935_suspend ,

• .resume

= as3935_resume ,

• Constructing replacement information.

22

Summary of observed changes

• 1. Drop suspend and resume fields.
• 2. Update parameter lists of the associated functions.
• 3. Update bodies of the associated functions.
• 4. Construct new pm field value.
• 5. Add pm field to a substructure.

Do these changes apply to our driver?

23

Step 3: Updating the lms501kf03 TFT LCD panel driver

static struct spi_driver lms501kf03_driver = { .driver = { .name = " lms501kf03 ", .owner = THIS_MODULE , }, .probe = lms501kf03_probe , .remove = lms501kf03_remove , .shutdown = lms501kf03_shutdown , .suspend = lms501kf03_suspend , .resume = lms501kf03_resume , };

24

Step 3: Updating the lms501kf03 TFT LCD panel driver

static struct spi_driver lms501kf03_driver = { .driver = { .name = " lms501kf03 ", .owner = THIS_MODULE , }, .probe = lms501kf03_probe , .remove = lms501kf03_remove , .shutdown = lms501kf03_shutdown ,

• .suspend

= lms501kf03_suspend ,

• .resume

= lms501kf03_resume , };

25

Step 3: Updating the lms501kf03 TFT LCD panel driver

static struct spi_driver lms501kf03_driver = { .driver = { .name = " lms501kf03 ", .owner = THIS_MODULE , + .pm = ..., }, .probe = lms501kf03_probe , .remove = lms501kf03_remove , .shutdown = lms501kf03_shutdown ,

• .suspend

= lms501kf03_suspend ,

• .resume

= lms501kf03_resume , };

26

Step 3: Updating the lms501kf03 TFT LCD panel driver

static int lms501kf03_suspend (struct spi_device *spi , pm_message_t mesg) { struct lms501kf03 *lcd = dev_get_drvdata (&spi->dev ); dev_dbg (&spi->dev , "lcd ->power = %d\n", lcd ->power ); return lms501kf03_power (lcd , FB_BLANK_POWERDOWN ); }

Change parameter list.

• Likewise for resume function. example commit only uses spi

in a call to spi get drvdata. This driver uses spi in calls to dev get drvdata and dev dbg.

27

Step 3: Updating the lms501kf03 TFT LCD panel driver

static int

• lms501kf03_suspend (struct

spi_device *spi , pm_message_t mesg) + lms501kf03_suspend (struct device *dev) { struct lms501kf03 *lcd = dev_get_drvdata (&spi->dev ); dev_dbg (&spi->dev , "lcd ->power = %d\n", lcd ->power ); return lms501kf03_power (lcd , FB_BLANK_POWERDOWN ); }

Change parameter list.

• Likewise for resume function. example commit only uses spi

in a call to spi get drvdata. This driver uses spi in calls to dev get drvdata and dev dbg.

28

Step 3: Updating the lms501kf03 TFT LCD panel driver

static int

• lms501kf03_suspend (struct

spi_device *spi , pm_message_t mesg) + lms501kf03_suspend (struct device *dev) { struct lms501kf03 *lcd = dev_get_drvdata (&spi->dev ); dev_dbg (&spi->dev , "lcd ->power = %d\n", lcd ->power ); return lms501kf03_power (lcd , FB_BLANK_POWERDOWN ); }

Change function body. Problem: the driver in the example commit only uses spi in a call to spi get drvdata. This driver uses spi in calls to dev get drvdata and dev dbg.

29

Step 3: Updating the lms501kf03 TFT LCD panel driver

static int

• lms501kf03_suspend (struct

spi_device *spi , pm_message_t mesg) + lms501kf03_suspend (struct device *dev) { struct lms501kf03 *lcd = dev_get_drvdata (&spi->dev ); dev_dbg (&spi->dev , "lcd ->power = %d\n", lcd ->power ); return lms501kf03_power (lcd , FB_BLANK_POWERDOWN ); }

Change function body.

• Problem: the driver in the example commit only uses spi in a

call to spi get drvdata. This driver uses spi in calls to dev get drvdata and dev dbg.

30

Returning to Step 2

Commit 9d9f780 was the most recent related to this issue. Other possibilities:

• Commit 01f9326 from March 2013 contains:
• struct

snd_card *card = dev_get_drvdata (&spi ->dev ); + struct snd_card *card = dev_get_drvdata (dev );

• Commit eba3bfb from April 2013 contains:
• dev_dbg (&spi ->dev , "lcd ->power = %d\n", lcd ->power );

+ dev_dbg(dev , "lcd ->power = %d\n", lcd ->power );

These examples plus commit 9d9f780 give enough information to port the driver.

31

Returning to Step 2

Commit 9d9f780 was the most recent related to this issue. Other possibilities:

• Commit 01f9326 from March 2013 contains:
• struct

snd_card *card = dev_get_drvdata (&spi ->dev ); + struct snd_card *card = dev_get_drvdata (dev );

• Commit eba3bfb from April 2013 contains:
• dev_dbg (&spi ->dev , "lcd ->power = %d\n", lcd ->power );

+ dev_dbg(dev , "lcd ->power = %d\n", lcd ->power );

These examples plus commit 9d9f780 give enough information to port the driver.

32

Challenges for steps 2 and 3

• A few compiler errors can require many changes.

33

Challenges for steps 2 and 3

• A few compiler errors can require many changes.
• Multiple example commits may be needed.

34

Challenges for steps 2 and 3

• A few compiler errors can require many changes.
• Multiple example commits may be needed.
• Essential to choose the right ones:

35

Challenges for steps 2 and 3

• A few compiler errors can require many changes.
• Multiple example commits may be needed.
• Essential to choose the right ones:

– Over 80 Linux kernel structure types have suspend and resume fields. – For our driver, need commits affecting spi driver structures.

36

Proposed solution

gcc-reduce:

• Eliminate redundant gcc error messages.
• Focus on information needed to find porting examples.

Prequel:

• Find commits that provide example solutions to porting issues.

ANR ITrans (TODO!)

• Generalize examples to porting rules.
• PhD and postdoc positions available.

37

Assumptions

• Examples are available.
• Examples are pertinent.
• Changes are not too complex.

– Enforced by the domain and the development process.

• Compilation is success.

38

Prequel: motivation

How to find commits that illustrate how to fix our reduced warning and error messages?

• git log -G suspend? 3799 results
• git log -S suspend? 3059 results
• google “unknown field ‘suspend’ specified in initializer”?

– First result for the early suspend structure type. – Two later results from 2011 and 2010 (too early).

• google “unknown field ‘suspend’ specified in initializer

spi driver”?

– Additional build regression message - no solution.

39

Requirements for commit search

• Search for particular code constructions:

– suspend as a field initializer, not e.g. as a function call.

• Search for particular transformations:

– suspend should be removed, not moved or added.

• Be sensitive to context information:

– Only want suspend for spi driver structures.

40

Requirements, contd.

• Rank results based on similarity to the request:

– Commit 9d9f780 does many other things, but less than e.g., a commit that introduces a new driver

• Be reasonably efficient, easy to use:

– Target a standard Linux kernel developer with a high end laptop.

41

Prequel

Pattern-matching based commit search

• Builds on previous work on Coccinelle.
• Notation familiar to Linux developers.

Example: Find commits that remove a suspend field

@@ identifier i; expression e; @@ struct spi_driver i = {

• .suspend

= e, };

42

Refined patch query

@bad depends

• n after@

identifier i; expression e; @@ struct spi_driver i = { .suspend = e, }; @depends

• n !bad@

identifier i; expression e; @@ struct spi_driver i = {

• .suspend

= e, };

43

Semantics

Rules applied to one file at a time

44

Semantics

Rules applied to one file at a time Rules can match old code, new code or both

• Rule with depends on after only matches new code.
• Rule with only - annotations only matches old code.

45

Semantics

Rules applied to one file at a time Rules can match old code, new code or both

• Rule with depends on after only matches new code.
• Rule with only - annotations only matches old code.

Matching based on line-by-line diff:

• Tokens on - lines must be on removed lines.
• Tokens on + lines must be on added lines.
• Other tokens can be on added, removed, or context lines.

– Approximate matching.

46

Semantics

Rules applied to one file at a time Rules can match old code, new code or both

• Rule with depends on after only matches new code.
• Rule with only - annotations only matches old code.

Matching based on line-by-line diff:

• Tokens on - lines must be on removed lines.
• Tokens on + lines must be on added lines.
• Other tokens can be on added, removed, or context lines.

– Approximate matching.

Scoring based on percentage of changed lines that are matched.

47

Efficiency

70K Linux kernel commits per year

• Impractical to extract, parse, and match all of them.

Indexing and abstraction:

• Indices:

– words on changed lines. – words within 3 lines of changed lines.

• Abstract patch query to words that must be present on or

near changed lines.

– Process only the relevant commits. – Some risk of false negatives

• For more distant words, find relevant files in a reference

version, and only take commits affecting those files.

48

Ease of use

Typically, no need to write patch queries for porting: Reduced compiler error messages of few types:

• Undefined variable/function/field/type.
• Wrong number of arguments to function or macro.
• Incompatible types in assignment/initialization/argument.
• Derived from the fact that the driver compiled with its original

target kernel. Patch query templates mostly sufficient.

49

Evaluation

Driver porting:

• Dataset: 56 Linux drivers introduced in 2013 or 2015
• Compiled original versions in Linux v4.6 (May 2016)
• 103 patch queries to find commits illustrating fixes

for the compiler errors. Performance, per patch query:

100 101 102 103 101 102 103 selected commits execution time (s) 2013 2015

50

Comparison with git

prior commits prior lines mn avg mx mn avg max 2013 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 1.3 1.3 7 8 163.2 38.1 1421 239 Unknown type 0.0 0.0 0.0 0.0 Unknown field 3 0 852.7 0.7 2195 10 184 0 146K.1 63.1 411K 756 Unknown variable 2.6 0.0 6 202.6 0.0 452 Arg error 8.8 1.2 ∞ 10 0 1189.2 117.9 ∞ 943 Value type change 2 46.6 0.0 73 0 178 0 5691.9 0.0 8582 Context type change ∞ ∞ 0.1 ∞ 1 ∞ ∞ 3.6 ∞ 29 2015 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 0.5 0.0 2 87.8 0.0 351 Unknown field 217 0 996.0 0.6 2243 1 31K 0 143K 15.6 318K 29 Arg error 1 1.0 0.0 1 30 30.0 0.0 30

51

Comparison with git

prior commits prior lines mn avg mx mn avg max 2013 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 1.3 1.3 7 8 163.2 38.1 1421 239 Unknown type 0.0 0.0 0.0 0.0 Unknown field 3 0 852.7 0.7 2195 10 184 0 146K.1 63.1 411K 756 Unknown variable 2.6 0.0 6 202.6 0.0 452 Arg error 8.8 1.2 ∞ 10 0 1189.2 117.9 ∞ 943 Value type change 2 46.6 0.0 73 0 178 0 5691.9 0.0 8582 Context type change ∞ ∞ 0.1 ∞ 1 ∞ ∞ 3.6 ∞ 29 2015 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 0.5 0.0 2 87.8 0.0 351 Unknown field 217 0 996.0 0.6 2243 1 31K 0 143K 15.6 318K 29 Arg error 1 1.0 0.0 1 30 30.0 0.0 30

52

Comparison with git

prior commits prior lines mn avg mx mn avg max 2013 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 1.3 1.3 7 8 163.2 38.1 1421 239 Unknown type 0.0 0.0 0.0 0.0 Unknown field 3 0 852.7 0.7 2195 10 184 0 146K.1 63.1 411K 756 Unknown variable 2.6 0.0 6 202.6 0.0 452 Arg error 8.8 1.2 ∞ 10 0 1189.2 117.9 ∞ 943 Value type change 2 46.6 0.0 73 0 178 0 5691.9 0.0 8582 Context type change ∞ ∞ 0.1 ∞ 1 ∞ ∞ 3.6 ∞ 29 2015 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 0.5 0.0 2 87.8 0.0 351 Unknown field 217 0 996.0 0.6 2243 1 31K 0 143K 15.6 318K 29 Arg error 1 1.0 0.0 1 30 30.0 0.0 30

53

Comparison with git

prior commits prior lines mn avg mx mn avg max 2013 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 1.3 1.3 7 8 163.2 38.1 1421 239 Unknown type 0.0 0.0 0.0 0.0 Unknown field 3 0 852.7 0.7 2195 10 184 0 146K.1 63.1 411K 756 Unknown variable 2.6 0.0 6 202.6 0.0 452 Arg error 8.8 1.2 ∞ 10 0 1189.2 117.9 ∞ 943 Value type change 2 46.6 0.0 73 0 178 0 5691.9 0.0 8582 Context type change ∞ ∞ 0.1 ∞ 1 ∞ ∞ 3.6 ∞ 29 2015 git prequel git prequel git prequel git prequel git prequel git prequel Unknown function 0.5 0.0 2 87.8 0.0 351 Unknown field 217 0 996.0 0.6 2243 1 31K 0 143K 15.6 318K 29 Arg error 1 1.0 0.0 1 30 30.0 0.0 30

54

Conclusion

• Prex: Finding example commits to guide device driver porting

– Gcc-reduce for minimizing error messages – Pattern language for searching commit histories – Templates connecting error messages to patch queries – 56 drivers from 2013 or 2015 ported to 2016 Linux kernel.

• Future work:

– Word diff. – Improving scoring algorithm. – Inferring changes from identified commits (ANR ITrans).

• Other potential applications:

– Finding potentially risky constructions – Automated program repair. – Metrics

• http://prequel-pql.gforge.inria.fr/

55

Prequel performance by error type.

Total Commit # commits Cocci time time (sec.) selection time selected per commit 2013 mn avg mx mn avg mx mn avg mx mn avg mx Unknown function 6 21 66 4 5 8 2 22 118 1 11 Unknown type 6 6 6 5 5 5 1 1 1 Unknown field 11 299 736 4 86 292 6 373 951 1 2 Unknown variable 7 15 28 4 5 5 4 24 69 1 2 Arg error 6 35 316 4 6 24 2 66 693 1 Value type change 6 30 92 4 6 10 2 43 221 1 1 Context type change 49 93 141 7 10 12 92 201 288 2015 mn avg mx mn avg mx mn avg mx mn avg mx Unknown function 4 6 7 2 2 3 1 6 11 1 3 Unknown field 100 443 875 4 59 147 100 726 1569 Arg error 3 10 15 2 3 3 1 23 33 mn = min, avg = average, mx = max. Times rounded to the nearest second.

56