compressing strings of the kernel
play

Compressing Strings of the Kernel Wolfram Sang Consultant - PowerPoint PPT Presentation

Jan 14, 2023 •460 likes •909 views

. . Compressing Strings of the Kernel Wolfram Sang Consultant 21.8.2014, LinuxCon14 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 1 / 36 The origin: CEWG project 1 kernel debug messages.

  1. . . Compressing Strings of the Kernel Wolfram Sang Consultant 21.8.2014, LinuxCon14 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 1 / 36

  2. The origin: CEWG project 1 kernel debug messages. 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) 1 http://elinux.org/Compressed_printk_messages … Timothy Miller did some work on this in 2003 … savings attractive, because they still would like to see . developers do not use it, even when they find the space option to disable all printks, but many embedded messages to save kernel runtime footprint. There is an Attempts have been made in the past to compress printk . . From the proposal: 2 / 36

  3. Timothy’s approach 2 4 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) 2 http://lwn.net/Articles/28935/ Compile again 6 . There, replace strings with tokens 5 . . Create copies of the source files . . . Compress those strings using tokenization 3 . . Filter them for printk strings 2 . . Compile kernel and keep .i -files 1 . 3 / 36

  4. Further notes no code was made public, only the description, a codebook and some results not even sure depacking at printk was ever made allyesconfig was used for the tests based on 2.4.20 and 2.5.68 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 4 / 36

  5. Author asks the golden question, too . Timothy Miller: 3 . . ”So, I ask... is this a useful savings? Is there any chance anyone would bother to increase their compile time by a factor of 5 in order to shave off 4% or 100k bytes?” 3 https://lkml.org/lkml/2003/6/6/207 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 5 / 36

  6. The Graph! Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 6 / 36

  7. The Graph! Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 6 / 36 5 4 3 compiletime size 2 1 0 before after

  8. Three problems identified . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) Replace printk format strings 3 . Compress printk format strings . 2 . . Extract printk format strings 1 . 7 / 36

  9. Extracting . Problem: Find all printk-strings . . There are lots of functions/defines embedding printk/vprintk_emit They are nested in all ways you can think of Moving target, there will be more <new_subsys>_dev_err, … Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 8 / 36

  10. Extracting: Options I . Scan the source files . . needs to know all printk-emerging functions misses merging of literals handle all ways of string concatenation Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 9 / 36

  11. Extracting: Options II . printk strings to own section . . scales a bit better (only base functions need to be converted) no knowledge where strings came from needs changes to core functions Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 10 / 36

  12. Extracting: own section + 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) +} while (0) printk(fmt, ##args); \ + } else \ printk(__f, ##args); \ +#define __printk(fmt, args...) \ + char __f[] = fmt; \ + static const __attribute__((section("__printk"))) \ + if (__builtin_constant_p(fmt)) { \ + +do { \ 11 / 36

  13. Extracting: own section II + 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) BTW don’t redefine printk . Really, don’t! ... __printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__) printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__) #define pr_emerg(fmt, ...) \ - #define pr_alert(fmt, ...) \ __printk(KERN_EMERG pr_fmt(fmt), ##__VA_ARGS__) + printk(KERN_EMERG pr_fmt(fmt), ##__VA_ARGS__) - 12 / 36

  14. Extracting: own section III 5 ++++ 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) 5 files changed, 75 insertions(+), 60 deletions(-) | 27 +++++++++++++++------- include/linux/printk.h | 48 ++++++++++++++++++++++++++------------- include/linux/device.h include/asm-generic/vmlinux.lds.h | Author: Wolfram Sang <wsa@the-dreams.de> | 19 ++++++++++++---- include/asm-generic/bug.h | 36 ++++------------------------- drivers/base/core.c Signed-off-by: Wolfram Sang <wsa@the-dreams.de> Kernel doesn't fully build yet. Only pr_*, dev_*, BUG, and WARN are supported. WIP: move printk strings to a special section 13 / 36

  15. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 14 / 36

  16. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 14 / 36

  17. Compressing . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) no frequency based compression (stats3) no variable length encoding (Huffman and friends) no sliding window algos (LZ and friends) . Conclusion . . no significant overhead not somewhere in the middle of packed data should be instantly available lots of small strings . . Problem: Algorithm 15 / 36

  18. Compressing . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) no frequency based compression (stats3) no variable length encoding (Huffman and friends) no sliding window algos (LZ and friends) . Conclusion . . no significant overhead not somewhere in the middle of packed data should be instantly available lots of small strings . . Problem: Algorithm 15 / 36

  19. Compressing II tokenization is actually a good option BytePairEncoding works, too . Problem: UTF8 . . Both approaches need ’empty’ symbols which might collide with UTF8 encoding. Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 16 / 36 both achieve ≈ 50% of compression still ≈ 4% of the kernel size gained

  20. Compressing II tokenization is actually a good option BytePairEncoding works, too . Problem: UTF8 . . Both approaches need ’empty’ symbols which might collide with UTF8 encoding. Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 16 / 36 both achieve ≈ 50% of compression still ≈ 4% of the kernel size gained

  21. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 17 / 36

  22. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 17 / 36

  23. Compressing III Brainstorming 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) could save second kernel compile, too predefined codebook? . . . . what about modules? smaller kernels, smaller pool for codes allyesconfig is unrealistic for tiny systems . . Problem: Codebook 18 / 36 share codebook from kernel → tied to that build own codebook → overhead eats gain

  24. Replacing . Scanning . . run source files through filter before compiling . Own section . . Work on the section directly? Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 19 / 36

  25. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 20 / 36

  26. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 20 / 36

  27. Further issues printk strings are only a subset devicetree uses a lot of strings! which should be easier to tackle since they are accessed via of_* functions address this problem at a higher level all strings? all .rodata? Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 21 / 36

  28. Summary Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 22 / 36

  29. Summary Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 22 / 36

  30. This quote still makes sense . From: Managing Gigabytes 4 . . ”We find ourselves in the midst of a practically important and intelectually fascinating convergence between the desire for more and better compression and the need to learn about what ’structure’ there is in data.” 4 Witten/Moffat/Bell, 1st edition, p. 385 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 23 / 36

  31. Analyzing the data observations from 3.16-rc5: x86-64 allyesconfig arm-cortexa8 customer kernel maybe a bit biased for device drivers Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 24 / 36

  32. Print from central locations . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) devm_ioremap_resource() OOM error message removal . Examples . . bonus: consistent messages locations as possible strings should be emitted from as centralized . . Proposal 25 / 36

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Chapter 9 Strings 1 C-Strings vs C++ Strings T wo string types: C-strings Array

Chapter 9 Strings 1 C-Strings vs C++ Strings T wo string types: C-strings Array

Chapter 9 Strings 1 C-Strings vs C++ Strings T wo string types: C-strings Array with base type char End of string marked with null, \0 Older method inherited from C C++ strings Objects of string class 2

469 views • 18 slides
Strings Testing for equality with strings. Lexicographic ordering of strings. Other

Strings Testing for equality with strings. Lexicographic ordering of strings. Other

Strings Testing for equality with strings. Lexicographic ordering of strings. Other string methods. 1 Strings and Testing for Equality The == operator is also not appropriate for determining if two objects , such as strings, have

380 views • 13 slides
Python:Strings Strings

Python:Strings Strings

Python:Strings Strings Source:https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-0001-introduction-to-computer-science-and- programming-in-python-fall-2016/lecture-slides-code/ Strings String is a sequence of

293 views • 13 slides
Strings l Chapter 3s problem context is cryptography, but mostly it is about strings and

Strings l Chapter 3s problem context is cryptography, but mostly it is about strings and

Starting chapter 3 Strings l Chapter 3s problem context is cryptography, but mostly it is about strings and related ideas l Strings are basically sequences of characters l A string literal is enclosed in quotes ( '' or in Python):

292 views • 12 slides
ARM Assembler Strings Strings p. 1/16 Characters or Strings A string is a sequence of

ARM Assembler Strings Strings p. 1/16 Characters or Strings A string is a sequence of

Systems Architecture ARM Assembler Strings Strings p. 1/16 Characters or Strings A string is a sequence of characters ASCII 7-bit (American) character codes ( char ) One byte per character Unicode International character

901 views • 61 slides
Py Python Strings Python strings are immuatable: s = abc s[2] = d s = abd

Py Python Strings Python strings are immuatable: s = abc s[2] = d s = abd

2/25/20 CS 224 Introduction to Python Spring 2020 Class #14: Strings Py Python Strings Python strings are immuatable: s = abc s[2] = d s = abd These dont change the string abc s = s[:-1] + d they

444 views • 19 slides
s[i] Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters Strings are

s[i] Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters Strings are

Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters s[i] Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters Strings are one of Pythons primary Strings data types Strings can be used

242 views • 10 slides
Listing Bit Strings List all bit strings of length 3. 000, 001, 010, 011, 100, 101, 110, 111.

Listing Bit Strings List all bit strings of length 3. 000, 001, 010, 011, 100, 101, 110, 111.

Listing Bit Strings List all bit strings of length 3. 000, 001, 010, 011, 100, 101, 110, 111. Now do it while only flipping one bit at a time! Today: Finish graphs and talk about numbers. Forests A forest is an acyclic graph. Each connected

534 views • 26 slides
Strings A string is an array of characters s = 'abc' MATLAB Strings is equivalent to s =

Strings A string is an array of characters s = 'abc' MATLAB Strings is equivalent to s =

Strings A string is an array of characters s = 'abc' MATLAB Strings is equivalent to s = [ 'a' 'b' 'c' ] All operations that apply to vectors and Selim Aksoy arrays can be used together with Bilkent University strings as well

39 views • 3 slides
Chapter 9: Strings (To avoid confusion, C-style strings will be referred to as C-string,

Chapter 9: Strings (To avoid confusion, C-style strings will be referred to as C-string,

Chapter 9: Strings (To avoid confusion, C-style strings will be referred to as C-string, regular C++ strings from the string class will be referred to as String). C-Strings Made up of character arrays that stores strings of

690 views • 3 slides
Lempel- -Ziv Ziv- -Welch (LZW) Welch (LZW) Lempel Data Compressing Model Data Compressing

Lempel- -Ziv Ziv- -Welch (LZW) Welch (LZW) Lempel Data Compressing Model Data Compressing

Lempel- -Ziv Ziv- -Welch (LZW) Welch (LZW) Lempel Data Compressing Model Data Compressing Model Martin Chakravorti Information Information What is information? Any interaction What is information? Any interaction between objects, when

754 views • 15 slides
Strings, Languages, and Regular expressions Lecture 2 1 Strings 2 Definitions for strings

Strings, Languages, and Regular expressions Lecture 2 1 Strings 2 Definitions for strings

Strings, Languages, and Regular expressions Lecture 2 1 Strings 2 Definitions for strings e.g., = {0,1}, = { , , , } , = set of ascii characters alphabet = finite set of symbols string = finite

371 views • 36 slides
Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel Fernndez V, David P. Woodruff, Taisuke Yasuda Kernel Method Many machine learning tasks can be expressed as a function of the inner product

389 views • 22 slides
Strings Digital Medicine I Lists, strings, loops Repetition Hans-Joachim Bckenhauer Dennis

Strings Digital Medicine I Lists, strings, loops Repetition Hans-Joachim Bckenhauer Dennis

Departement Informatik Strings Digital Medicine I Lists, strings, loops Repetition Hans-Joachim Bckenhauer Dennis Komm Autumn 2020 October 15, 2020 Strings Strings Operators Addition Strings are lists of characters (there

268 views • 13 slides
1 Kernel methods &amp; optimization One example of a kernel that is frequently used in practice

1 Kernel methods &amp; optimization One example of a kernel that is frequently used in practice

Machine Learning Class Notes 9-25-12 Prof. David Sontag 1 Kernel methods &amp; optimization One example of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions is the Gaussian kernel,

240 views • 5 slides
C-Style Strings CS2253 Owen Kaser, UNBSJ Strings In C and some other low-level languages,

C-Style Strings CS2253 Owen Kaser, UNBSJ Strings In C and some other low-level languages,

C-Style Strings CS2253 Owen Kaser, UNBSJ Strings In C and some other low-level languages, strings are just consecutive memory locations that contain characters. A special null character (ASCII code 0) terminates the string.

813 views • 17 slides
THE C PROGRAMMING LANGUAGE WHY LEARN C? Compared to other high-level languages Maps almost

THE C PROGRAMMING LANGUAGE WHY LEARN C? Compared to other high-level languages Maps almost

THE C PROGRAMMING LANGUAGE WHY LEARN C? Compared to other high-level languages Maps almost directly into hardware instructions making code potentially more efficient Provides minimal set of abstractions compared to other HLLs HLLs

783 views • 50 slides
Strings 1 Sharif University of Technology Department of Computer Engineering Input and Output

Strings 1 Sharif University of Technology Department of Computer Engineering Input and Output

Strings 1 Sharif University of Technology Department of Computer Engineering Input and Output Lecture 4 Introduction Until now We have seen strings in printf Our old definition: string is a set of char between

489 views • 34 slides
Trends in Open Source Security FOSDEM 2013 Florian Weimer fweimer@redhat.com Red Hat

Trends in Open Source Security FOSDEM 2013 Florian Weimer fweimer@redhat.com Red Hat

Trends in Open Source Security FOSDEM 2013 Florian Weimer fweimer@redhat.com Red Hat Product Security Team 2013-02-02 Overview Vulnerability tracking Tool-chain hardening Distribution-wide defect analysis 2 TRENDS IN OPEN

462 views • 31 slides
From Overow to Shell From Overow to Shell An Introduction to low-level exploitation An

From Overow to Shell From Overow to Shell An Introduction to low-level exploitation An

From Overow to Shell From Overow to Shell An Introduction to low-level exploitation An Introduction to low-level exploitation Carl Svensson @ Foo Caf, February 2019 Carl Svensson @ Foo Caf, February 2019 1 / 28 1 / 28 Background

619 views • 28 slides
Building DIFT Systems for Software Security Michael Dalton Computer Systems Laboratory Stanford

Building DIFT Systems for Software Security Michael Dalton Computer Systems Laboratory Stanford

Building DIFT Systems for Software Security Michael Dalton Computer Systems Laboratory Stanford University Research Focus My focus is on softw are attacks Protect apps from malicious input Buffer overflows, XSS, SQL

718 views • 54 slides
OpenCms Days 2011 Workshop Track: Creating Plug &amp; Play Modules for OpenCms 8 Rdiger Kurz

OpenCms Days 2011 Workshop Track: Creating Plug &amp; Play Modules for OpenCms 8 Rdiger Kurz

OpenCms Days 2011 Workshop Track: Creating Plug &amp; Play Modules for OpenCms 8 Rdiger Kurz Alkacon Software GmbH Agenda Developing Web-Projects in general OpenCms Modules v7 vs. v8 The module structure The module contents

623 views • 23 slides
Mining Anomaly Detectors Paolo Tonella Software Engineering Research Unit Fondazione Bruno

Mining Anomaly Detectors Paolo Tonella Software Engineering Research Unit Fondazione Bruno

Mining Anomaly Detectors Paolo Tonella Software Engineering Research Unit Fondazione Bruno Kessler Trento, Italy http://se.fbk.eu/tonella Outline Role and classification of (mined) oracles Oracle mining techniques Empirical

408 views • 18 slides
Threads for Camelot Stephen Gilmore LFCS, University of Edinburgh Presented in Munich, 22nd

Threads for Camelot Stephen Gilmore LFCS, University of Edinburgh Presented in Munich, 22nd

Threads for Camelot Stephen Gilmore LFCS, University of Edinburgh Presented in Munich, 22nd November 2003 Updated in Edinburgh, 1st December 2003 Threads: review Threads are used to structure programs in order that two (or more) activities

290 views • 27 slides

More recommend