Increasing Automation in the Backporting of Linux Drivers Using - - PowerPoint PPT Presentation

increasing automation in the backporting of linux drivers
SMART_READER_LITE
LIVE PREVIEW

Increasing Automation in the Backporting of Linux Drivers Using - - PowerPoint PPT Presentation

Increasing Automation in the Backporting of Linux Drivers Using Coccinelle Luis R. Rodriguez, (SUSE Labs) Julia Lawall (Inria/LIP6/UPMC/Sorbonne University-Whisper) January, 2015 (Unpublished work) What is backporting? port Linux v3.18


slide-1
SLIDE 1

Increasing Automation in the Backporting of Linux Drivers Using Coccinelle

Luis R. Rodriguez, (SUSE Labs) Julia Lawall (Inria/LIP6/UPMC/Sorbonne University-Whisper) January, 2015

(Unpublished work)

slide-2
SLIDE 2

What is backporting?

Linux v3.18 driver port

  • backport

BSD Linux v3.0 Why would we want to do that?

slide-3
SLIDE 3

What is backporting?

Linux v3.18 driver port

  • backport
  • BSD

Linux v3.0 Why would we want to do that?

slide-4
SLIDE 4

What is backporting?

Linux v3.18 driver port

  • backport
  • BSD

Linux v3.0 Why would we want to do that?

slide-5
SLIDE 5

The latency of product development

Athalon wireless device

slide-6
SLIDE 6

The latency of product development

v2.6.32 v2.6.34 v2.6.36 v2.6.38 Jul’09 Dec’09 May’10 Oct’10 Mar’11 ChromeOS announced CR48 released

  • ChromeOS based on Linux v2.6.32.
  • New devices appear all the time.

– Eg, Atheros IEEE 802.11n wireless chipset.

  • Ath9k driver developed for Linux v2.6.38, not Linux v2.6.32
slide-7
SLIDE 7

Possible solutions

Make an ath9k driver for Linux v2.6.32?

  • Lots of work, error-prone.
  • Atheros may not be motivated.
  • ChromeOS may modernize to eg Linux v2.6.36.
slide-8
SLIDE 8

Possible solutions

Make an ath9k driver for Linux v2.6.32?

  • Lots of work, error-prone.
  • Atheros may not be motivated.
  • ChromeOS may modernize to eg Linux v2.6.36.

Modernize ChromeOS to Linux v2.6.38?

  • Not in the short term.
  • May prefer using a stable kernel.
slide-9
SLIDE 9

Possible solutions

Make an ath9k driver for Linux v2.6.32?

  • Lots of work, error-prone.
  • Atheros may not be motivated.
  • ChromeOS may modernize to eg Linux v2.6.36.

Modernize ChromeOS to Linux v2.6.38?

  • Not in the short term.
  • May prefer using a stable kernel.

Ensure Linux v2.6.38 drivers run out of the box on Linux v2.6.32?

  • Hinders advancement.
  • Not in the Linux philosophy.
slide-10
SLIDE 10

Backporting

Goal:

  • Slightly modify modern drivers for compatability with older

versions.

slide-11
SLIDE 11

Backporting

Goal:

  • Slightly modify modern drivers for compatability with older

versions. Issues:

  • Where to start?
  • How to express modifications?
  • Scalability.

– 10,000 or so Linux drivers. – Code arrives/modified every day.

slide-12
SLIDE 12

Where to start?

OS

  • driver

Product

  • Device manufacturer targets an OS version relevant to

potential customers.

  • OS always moves ahead

– New Linux release every 2.5-3 months.

  • Products may modernize as well.
  • A driver targeting any specific release is always left behind.
slide-13
SLIDE 13

Upstream-first development

Our problem:

  • A driver is too modern for existing clients,
  • And too old fashioned for future clients.
slide-14
SLIDE 14

Upstream-first development

Our problem:

  • A driver is too modern for existing clients,
  • And too old fashioned for future clients.

Upstream-first development:

  • Driver integrated with HEAD of Linus’s git tree.
slide-15
SLIDE 15

Upstream-first development

Our problem:

  • A driver is too modern for existing clients,
  • And too old fashioned for future clients.

Upstream-first development:

  • Driver integrated with HEAD of Linus’s git tree.
  • Advantages

– Driver developed once, modernized by kernel maintainers. – Solves our second problem.

  • Inconveniences

– Coding style constraints. – What about backporting?

slide-16
SLIDE 16

Upstream-first development

Our problem:

  • A driver is too modern for existing clients,
  • And too old fashioned for future clients.

Upstream-first development:

  • Driver integrated with HEAD of Linus’s git tree.
  • Advantages

– Driver developed once, modernized by kernel maintainers. – Solves our second problem.

  • Inconveniences

– Coding style constraints. – What about backporting?

To make upstream-first development attractive, we need an “industrial-strength” solution to backporting.

slide-17
SLIDE 17

How to express modifications?

Typical strategy: #ifdefs by kernel versions. An artificial example: #if (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29)) A_new(); #elif (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25)) A_older(); #else A_very_old(); #endif

slide-18
SLIDE 18

A real example

Linux v2.6.28 code: drivers/net/usb/usbnet.c

net->change_mtu = usbnet_change_mtu; net->get_stats = usbnet_get_stats; net->hard_start_xmit = usbnet_start_xmit; net->open = usbnet_open; net->stop = usbnet_stop; net->watchdog_timeo = TX_TIMEOUT_JIFFIES; net->tx_timeout = usbnet_tx_timeout;

Current code: (12.12.2014)

net->netdev_ops = &usbnet_netdev_ops; net->watchdog_timeo = TX_TIMEOUT_JIFFIES;

slide-19
SLIDE 19

Issues

Given net->netdev ops = &usbnet netdev ops;, must:

  • Find the definition of usbnet netdev ops:

static const struct net_device_ops usbnet_netdev_ops = { .ndo_open = usbnet_open, .ndo_stop = usbnet_stop, .ndo_start_xmit = usbnet_start_xmit, .ndo_tx_timeout = usbnet_tx_timeout, .ndo_set_rx_mode = usbnet_set_rx_mode, .ndo_change_mtu = usbnet_change_mtu, .ndo_set_mac_address = eth_mac_addr, .ndo_validate_addr = eth_validate_addr, };

  • Find the names of the corresponding fields.

– Some perhaps didn’t exist.

  • Remove the definition of usbnet netdev ops.
  • Construct the new code.
slide-20
SLIDE 20

Result, part 1

  • -- a/drivers/net/usb/usbnet.c

+++ b/drivers/net/usb/usbnet.c @@

  • 1151,6 +1151 ,7 @@

} EXPORT_SYMBOL_GPL ( usbnet_disconnect ); +#if ( LINUX_VERSION_CODE >= KERNEL_VERSION (2 ,6 ,29)) static const struct net_device_ops usbnet_netdev_ops = { .ndo_open = usbnet_open , .ndo_stop = usbnet_stop , @@

  • 1160,6 +1161 ,7 @@

. ndo_set_mac_address = eth_mac_addr , . ndo_validate_addr = eth_validate_addr , }; +#endif /*-----------------------------------------------------*/

slide-21
SLIDE 21

Result, part 2

@@

  • 1229,7 +1231 ,15 @@

net ->features |= NETIF_F_HIGHDMA ; #endif +#if ( LINUX_VERSION_CODE >= KERNEL_VERSION (2 ,6 ,29)) net -> netdev_ops = & usbnet_netdev_ops ; +#else + net -> change_mtu = usbnet_change_mtu ; + net -> hard_start_xmit = usbnet_start_xmit ; + net ->open = usbnet_open ; + net ->stop = usbnet_stop ; + net -> tx_timeout = usbnet_tx_timeout ; +#endif net -> watchdog_timeo = TX_TIMEOUT_JIFFIES ; net -> ethtool_ops = & usbnet_ethtool_ops ;

slide-22
SLIDE 22

Assessment

  • Added: 5 lines of C code, 5 lines of #ifdefs.
  • Changes maintained as patches

– Allows changes upstream.

  • 61 netdev ops fields possible.
  • All field names change.
  • ndo_set_mac_address and ndo_validate_addr removed.

– Have default values.

  • Bug? What happened to ndo_set_rx_mode?

484 netdev ops initializations in 434 files.

slide-23
SLIDE 23

Backports via a compatability library

Observations:

  • The code to modify is copious but repetitive.

– Remove a structure, because its type is not available. – Copy structure field values.

slide-24
SLIDE 24

Backports via a compatability library

Observations:

  • The code to modify is copious but repetitive.

– Remove a structure, because its type is not available. – Copy structure field values.

These changes can be encapsulated in a library:

  • Define the missing type.
  • Define a function to perform the structure copy.
slide-25
SLIDE 25

Backports via a compatability library

Observations:

  • The code to modify is copious but repetitive.

– Remove a structure, because its type is not available. – Copy structure field values.

These changes can be encapsulated in a library:

  • Define the missing type.
  • Define a function to perform the structure copy.

Proposition of the Linux backports project

  • Initiated in 2007 by Luis R. Rodriguez, to backport 802.11

wireless drivers.

slide-26
SLIDE 26

Compat library-based approach

  • -- a/drivers/net/usb/usbnet.c

+++ b/drivers/net/usb/usbnet.c @@

  • 1446,7 +1446 ,7 @@

usbnet_probe (struct usb_interface *udev net ->features |= NETIF_F_HIGHDMA ; #endif

  • net -> netdev_ops = & usbnet_netdev_ops ;

+ netdev_attach_ops (net , & usbnet_netdev_ops ); net -> watchdog_timeo = TX_TIMEOUT_JIFFIES ; net -> ethtool_ops = & usbnet_ethtool_ops ;

  • -- a/drivers/net/wireless/ath/ath6kl/main.c

+++ b/drivers/net/wireless/ath/ath6kl/main.c @@

  • 1289,7 +1289 ,7 @@

static const struct net_device_ops ath6k void init_netdev (struct net_device *dev) {

  • dev -> netdev_ops = & ath6kl_netdev_ops ;

+ netdev_attach_ops (dev , & ath6kl_netdev_ops ); dev -> destructor = free_netdev ; dev -> watchdog_timeo = ATH6KL_TX_TIMEOUT ;

Backports two drivers, in one line each.

slide-27
SLIDE 27

Scalability

Current status of the backports project:

  • 800 ethernet, wireless, bluetooth, NFC, ieee802154, media,

and regulator drivers.

  • Backported from their linux-next, release candidate, and

recent stable versions.

  • 18 earlier releases as backport targets.
  • linux-next and linux-stable evolve every day.
  • Changes maintained as patches, which become out of date.
  • 2-6 iterations of tests, refinements, compiles for all supported

versions.

– Patches are fragile.

Goal: Automate the transformation part.

slide-28
SLIDE 28

Coccinelle to the rescue

Our transformations have a lot in common:

  • net -> netdev_ops = & usbnet_netdev_ops ;

+ netdev_attach_ops (net , & usbnet_netdev_ops );

  • dev -> netdev_ops = & ath6kl_netdev_ops ;

+ netdev_attach_ops (dev , & ath6kl_netdev_ops );

Similar, but one per file. Coccinelle:

  • Semantic patches, generalizing over unimportant details.
  • Used for over 2000 Linux kernel patches.
slide-29
SLIDE 29

Backporting netdev ops with Coccinelle

  • net -> netdev_ops = & usbnet_netdev_ops ;

+ netdev_attach_ops (net , & usbnet_netdev_ops );

6 lines to backport this change for all drivers.

slide-30
SLIDE 30

Backporting netdev ops with Coccinelle

  • dev -> netdev_ops = &ops;

+ netdev_attach_ops (dev , &ops );

6 lines to backport this change for all drivers.

slide-31
SLIDE 31

Backporting netdev ops with Coccinelle

@@ struct net_device *dev; struct net_device_ops

  • ps;

@@

  • dev -> netdev_ops = &ops;

+ netdev_attach_ops (dev , &ops );

6 lines to backport this change for all drivers.

slide-32
SLIDE 32

Backporting netdev ops with Coccinelle

@@ struct net_device *dev; struct net_device_ops

  • ps;

@@

  • dev -> netdev_ops = &ops;

+ netdev_attach_ops (dev , &ops );

6 lines to backport this change for all drivers.

slide-33
SLIDE 33

Performance

Patch

  • Applies to a specific file and line number.
  • No parsing required.
slide-34
SLIDE 34

Performance

Patch

  • Applies to a specific file and line number.
  • No parsing required.

Coccinelle

  • Parses semantic patch and C code,
  • Searches for positions where the semantic patch matches,
  • Performs the tranformation.
slide-35
SLIDE 35

Performance

Patch

  • Applies to a specific file and line number.
  • No parsing required.

Coccinelle

  • Parses semantic patch and C code,
  • Searches for positions where the semantic patch matches,
  • Performs the tranformation.

Coccinelle optimizations

  • Parallelism, by file.
  • Keyword indexing.
  • Can be faster than sequential patch application.
slide-36
SLIDE 36

A more complex example

Threaded IRQs introduced in Linux v2.6.31.

  • Adds an extra handler to normal request irq call.
  • Need somewhere to store this handler.
slide-37
SLIDE 37

A more complex example

Threaded IRQs introduced in Linux v2.6.31.

  • Adds an extra handler to normal request irq call.
  • Need somewhere to store this handler.

Solution

  • Use device’s private structure.
  • Need to find structure type name, extend structure definition.
slide-38
SLIDE 38

Extending the private structure using Coccinelle

@ threaded_irq @ identifier ret; type T; T *private; expression irq , irq_handler , irq_thread_handler , flags , name; @@ +#if LINUX_VERSION_CODE >= KERNEL_VERSION (2 ,6 ,31) ret = request_threaded_irq (irq , irq_handler , irq_thread_handler , flags , name , private ); +#else +ret = compat_request_threaded_irq (& private ->irq_compat , + irq , irq_handler , irq_thread_handler , + flags , name , private ); +#endif @ modify_private_header depends

  • n

threaded_irq @ type threaded_irq .T; @@ T { +#if LINUX_VERSION_CODE < KERNEL_VERSION (2 ,6 ,31) + struct compat_threaded_irq irq_compat ; +#endif ... };

Update some IRQ oprations accordingly.

slide-39
SLIDE 39

Conclusion

Status

v3.11 v3.12 v3.13 v3.14 v3.15 v3.16.2 v3.17-rc3 600 650 700 750 800

backported drivers

backported drivers

Currently 5 semantic patches, representing 471 lines of code. Future work:

  • Make Linux code more backport friendly.
  • Infer semantic patches, or even compat library code.
  • Address correctness issues - currently, only compilation.
slide-40
SLIDE 40

Conclusion

Status

v3.11 v3.12 v3.13 v3.14 v3.15 v3.16.2 v3.17-rc3 600 650 700 750 800

backported drivers

backported drivers

Currently 5 semantic patches, representing 471 lines of code. Future work:

  • Make Linux code more backport friendly.
  • Infer semantic patches, or even compat library code.
  • Address correctness issues - currently, only compilation.

“All the patches that broke often in the early days are now using coccinelle or are removed because they were only needed for the

  • lder kernel versions.” [Hauke Mehrtens, 10.23.2014]