an evolutionary study of linux memory management for fun
play

An Evolutionary Study of Linux Memory Management for Fun and Profit - PowerPoint PPT Presentation

Jul 31, 2023 •333 likes •1.23k views

An Evolutionary Study of Linux Memory Management for Fun and Profit Jian Huang Karsten Schwan Moinuddin K. Qureshi Virtual Memory: A Long History Physical Hardware DRAM Disk 2 Virtual Memory: A Long History Virtual Memory (per process)

  1. An Evolutionary Study of Linux Memory Management for Fun and Profit Jian Huang Karsten Schwan Moinuddin K. Qureshi

  2. Virtual Memory: A Long History Physical Hardware DRAM Disk 2

  3. Virtual Memory: A Long History Virtual Memory (per process) Physical Hardware DRAM Disk 2

  4. Virtual Memory: A Long History Virtual Memory (per process) Physical Hardware DRAM Disk 2

  5. Virtual Memory: A Long History Virtual Memory (per process) Physical Hardware DRAM Disk OS Core Component + Development 2

  6. Virtual Memory: A Long History Virtual Memory (per process) Physical Hardware DRAM Disk OS Core Component + Development Pervasively Used 2

  7. Why Memory Manager Study Matters? Features & Functions 3

  8. Why Memory Manager Study Matters? Features & Functions Hardware Support 3

  9. Why Memory Manager Study Matters? Features & Functions Hardware Support System Reliability 3

  10. Why Memory Manager Study Matters? Features & Functions Hardware Support System Reliability Study on Memory Manager 3

  11. Why Memory Manager Study Matters? Features & Functions Hardware Support System Reliability Study on Memory Manager Building Better Memory Manager 3

  12. On the Study of Memory Management Understanding the Linux Virtual Memory Manager [Mel Gorman, July 9, 2007] 4

  13. On the Study of Memory Management Understanding the Linux Virtual Memory Manager [Mel Gorman, July 9, 2007] Approach: Source code analysis, Linux 2.4, 2.6 4

  14. On the Study of Memory Management Understanding the Linux Virtual Memory Manager [Mel Gorman, July 9, 2007] Approach: Source code analysis, Linux 2.4, 2.6 Milestone 4

  15. On the Study of Memory Management Understanding the Linux Virtual Memory Manager [Mel Gorman, July 9, 2007] Approach: Source code analysis, Linux 2.4, 2.6 Milestone Our Focus : Patch study, Linux 2.6 – 4.0 4

  16. On the Study of Memory Management Understanding the Linux Virtual Memory Manager [Mel Gorman, July 9, 2007] Approach: Source code analysis, Linux 2.4, 2.6 Milestone Our Focus : Patch study, Linux 2.6 – 4.0 + + + Pattern Memory Bug Semantic Optimization 4

  17. Preview of Our Findings Code changes are highly concentrated around the key functions • 80% of patches  25% of its source code • ...... • 5

  18. Preview of Our Findings Code changes are highly concentrated around the key functions • 80% of patches  25% of its source code • ...... • • Memory error – Checking – Concurrency – Logic – Programming • Memory errors: Null pointer & page alignment • Complex page states  Checking & logic bugs • ...... 5

  19. Preview of Our Findings Code changes are highly concentrated around the key functions • 80% of patches  25% of its source code • ...... • • Memory error – Checking – Concurrency – Logic – Programming • Memory errors: Null pointer & page alignment • Complex page states  Checking & logic bugs • ...... • Data structures -- Policy trade-off -- Fast path • 4 data structures, 5 design trade-offs, 8 types of fast paths • ...... 5

  20. Preview of Our Findings Code changes are highly concentrated around the key functions • 80% of patches  25% of its source code • ...... • • Memory error – Checking – Concurrency – Logic – Programming • Memory errors: Null pointer & page alignment • Complex page states  Checking & logic bugs • ...... • Data structures -- Policy trade-off -- Fast path • 4 data structures, 5 design trade-offs, 8 types of fast paths • ...... 35 key functionalities in 13 hot files • The well-developed memory allocators still have many checking & lock bugs • ...... • 5

  21. Methodology Used in Our Study Memory Allocation Resource Controller Garbage Collection Page Cache & Write-back Virtual Memory Management Swapping Exception Handling Misc (e.g., data structure) 8 components 6

  22. Methodology Used in Our Study Memory Allocation Resource Controller Garbage Collection Page Cache & Write-back Virtual Memory Management Swapping Exception Handling Misc (e.g., data structure) 8 components 4587 patches in 5 years 6

  23. Methodology Used in Our Study Patches Follow-up Description Source Code Discussions Analysis 6

  24. Methodology Used in Our Study Patches Follow-up Description Source Code Discussions Analysis Labeling & MChecker BugID Causes Type Component …… Commit Time Involved Functions MPatch 6

  25. How Is the Memory Manager Changed? 80 70 60 Lines of Code (x1000) 50 40 30 20 10 0 2.6.32 (2009) 2.6.33 (2010) 2.6.38 (2011) 3.2 (2012) 3.10 (2013) 3.14 (2014) 4.0-rc4 (2015) Linux version (released year) 7

  26. How Is the Memory Manager Changed? 80 70 60 Lines of Code (x1000) 50 40 30 The LoC has increased by 60% since Linux 2.6.32. 20 10 0 2.6.32 (2009) 2.6.33 (2010) 2.6.38 (2011) 3.2 (2012) 3.10 (2013) 3.14 (2014) 4.0-rc4 (2015) Linux version (released year) 7

  27. Where Is the Memory Manager Changing? 8

  28. Where Is the Memory Manager Changing? Memory Manager Components 8

  29. Where Is the Memory Manager Changing? Linux Version 8

  30. Where Is the Memory Manager Changing? Number of Committed Patches 8

  31. Where Is the Memory Manager Changing? 8

  32. Where Is the Memory Manager Changing? 80% of the code changes  25% of the source code 8

  33. Where Is the Memory Manager Changing? 8

  34. Where Is the Memory Manager Changing? 8

  35. Where Is the Memory Manager Changing? 8

  36. Where Is the Memory Manager Changing? 8

  37. Where Is the Memory Manager Changing? 13 hot files from 90 files  recent development focus 8

  38. Why Is the Memory Manager Changed? Bug Code Maintenance Optimization New Feature 60 50 Percentage (%) 40 30 20 10 0 2.6.33 (2010) 2.6.38 (2011) 3.2 (2012) 3.10 (2013) 3.14 (2014) 4.0-rc4 (2015) Linux version (released year) 9

  39. Why Is the Memory Manager Changed? Code Maintenance New Feature 60 50 Percentage (%) 40 30 20 10 0 2.6.33 (2010) 2.6.38 (2011) 3.2 (2012) 3.10 (2013) 3.14 (2014) 4.0-rc4 (2015) Linux version (released year) 9

  40. Why Is the Memory Manager Changed? Bug Optimization 60 50 Percentage (%) 40 30 20 70% more bugs in well-developed memory manager! 10 0 2.6.33 (2010) 2.6.38 (2011) 3.2 (2012) 3.10 (2013) 3.14 (2014) 4.0-rc4 (2015) Linux version (released year) 9

  41. On the Bugs in Memory Manager 10

  42. On the Bugs in Memory Manager Types of Memory Bugs 10

  43. On the Bugs in Memory Manager Memory Manager Component 10

  44. On the Bugs in Memory Manager Memory Allocation: 26%, Virtual Memory Management: 22%, GC: 14% 10

  45. On the Bugs in Memory Manager 10

  46. On the Bugs in Memory Manager • Page alignment Null pointer • 10

  47. On the Bugs in Memory Manager • Page alignment Null pointer • • Inappropriate check Missing check • Wrong check • 10

  48. On the Bugs in Memory Manager • Page alignment • Lock contention Null pointer Missing lock • • • Inappropriate check Missing check • Wrong check • 10

  49. On the Bugs in Memory Manager • Page alignment • Lock contention Null pointer Missing lock • • Fault handler • Inappropriate check • Missing check State update • • Wrong check Case-by-case • • 10

  50. On the Bugs in Memory Manager • Page alignment • Lock contention • API Null pointer Missing lock Misc • • • Fault handler • Inappropriate check • Missing check State update • • Wrong check Case-by-case • • 10

  51. Memory Bugs: Case Studies Page Alignment mm/nommu.c @@ -1762,6 +1765,8 @@ unsigned long do_mremap(unsigned long addr, struct vm_area_struct *vma; /* insanity checks first */ if (old_len == 0 || new_len == 0) return (unsigned long) -EINVAL; 11

  52. Memory Bugs: Case Studies Page Alignment mm/nommu.c @@ -1762,6 +1765,8 @@ unsigned long do_mremap(unsigned long addr, struct vm_area_struct *vma; /* insanity checks first */ if (old_len == 0 || new_len == 0) return (unsigned long) -EINVAL; Bug: device drivers’ mmap() failed. Cause: NOMMU does not do page_align(), which is inconsistent with MMU arch. 11

  53. Memory Bugs: Case Studies Page Alignment mm/nommu.c @@ -1762,6 +1765,8 @@ unsigned long do_mremap(unsigned long addr, struct vm_area_struct *vma; /* insanity checks first */ + old_len = PAGE_ALIGN(old_len); + new_len = PAGE_ALIGN(new_len); if (old_len == 0 || new_len == 0) return (unsigned long) -EINVAL; Bug: device drivers’ mmap() failed. Cause: NOMMU does not do page_align(), which is inconsistent with MMU arch. 11

  54. Memory Bugs: Case Studies Checking mm/bootmem.c @@ -156,21 +157,31 @@ static void __init free_bootmem_core(bootmem_data_t *bdata, unsigned long addr, 12

  55. Memory Bugs: Case Studies Checking mm/bootmem.c @@ -156,21 +157,31 @@ static void __init free_bootmem_core(bootmem_data_t *bdata, unsigned long addr, Bug: free pages wrongly. Cause: miss boundary checking. 12

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Dynamic Memory Management The Linux Perspective Allocating memory: The

Dynamic Memory Management The Linux Perspective Allocating memory: The

CPSC 410/611 : Operating Systems Dynamic Memory Management The Linux Perspective Allocating memory: The Interface Buddy System Slab Allocation Reading: Silberschatz (8 th ed.), Chapters 9.8

447 views • 6 slides
CS 377: Operating Systems Linux Case Study Outline Linux History Design Principles A

CS 377: Operating Systems Linux Case Study Outline Linux History Design Principles A

CS 377: Operating Systems Linux Case Study Outline Linux History Design Principles A review of what youve System Overview learned, and how it Process Scheduling applies to a real Memory Management operating system File

245 views • 13 slides
Dynamic Memory Management 333 Dynamic Memory Management Process Memory Layout Process Memory

Dynamic Memory Management 333 Dynamic Memory Management Process Memory Layout Process Memory

Dynamic Memory Management Dynamic Memory Management 333 Dynamic Memory Management Process Memory Layout Process Memory Layout (1) Each Linux process runs within its own virtual address space tables and the MMU (if available) security due to

1.25k views • 68 slides
1 3. Simple Memory System Cache Address Translation Example #1 16 lines, 4-byte block size

1 3. Simple Memory System Cache Address Translation Example #1 16 lines, 4-byte block size

Today Simple memory system example Case study: Core i7/Linux memory system Virtual Memory: Systems Memory mapping CSci 2021: Machine Architecture and Organization April 20th-22nd, 2020 Your instructor: Stephen McCamant Based on

587 views • 5 slides
Linux memory management at scale Chris Down Kernel, Facebook https://chrisdown.name server

Linux memory management at scale Chris Down Kernel, Facebook https://chrisdown.name server

Linux memory management at scale Chris Down Kernel, Facebook https://chrisdown.name server Image: Spc. Christopher Hernandez, US Military Public Domain Image: Simon Law on Flickr, CC-BY-SA Image: Orion J on Wikimedia Commons, CC-BY Memory

336 views • 22 slides
Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

4/12/2017 Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign physical memory to processes 5B. Simple Memory Allocation explicit requests: malloc (sbrk) implicit: program loading, stack

205 views • 8 slides
Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

4/14/2018 Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign physical memory to processes 5B. Memory Allocation explicit requests: malloc (sbrk) implicit: program loading, stack extension

555 views • 9 slides
Memory Management in Linux By: Rohan Garg 2002134 Gaurav Gupta 2002435 Architecture Independent

Memory Management in Linux By: Rohan Garg 2002134 Gaurav Gupta 2002435 Architecture Independent

Memory Management in Linux By: Rohan Garg 2002134 Gaurav Gupta 2002435 Architecture Independent Memory Model Process virtual address space divided into pages Page size given in PAGE_SIZE macro in asm/page.h (4K for x86 and 8K for

368 views • 23 slides
Virtual Memory and Linux Alan Ott Embedded Linux Conference April 4-6, 2016 About the Presenter

Virtual Memory and Linux Alan Ott Embedded Linux Conference April 4-6, 2016 About the Presenter

Virtual Memory and Linux Alan Ott Embedded Linux Conference April 4-6, 2016 About the Presenter Linux Architect at SoftIron 64-bit ARM servers and data center appliences Linux Kernel Firmware Userspace Training USB

1.04k views • 71 slides
Lecture 5: Memory Management 1 / 54 Memory Management Administrivia Assignment 1 is due on

Lecture 5: Memory Management 1 / 54 Memory Management Administrivia Assignment 1 is due on

Memory Management Lecture 5: Memory Management 1 / 54 Memory Management Administrivia Assignment 1 is due on September 7th @ 11:59pm 2 / 54 Memory Management Poll https: // www.strawpoll.me / 20863490 3 / 54 Memory Management Recap

952 views • 54 slides
From weak to weedy Effective use of memory barriers in the ARM Linux Kernel Will Deacon

From weak to weedy Effective use of memory barriers in the ARM Linux Kernel Will Deacon

Introduction ARMs memory model Linuxs memory model Finer-grained control Questions Future work From weak to weedy Effective use of memory barriers in the ARM Linux Kernel Will Deacon will.deacon@arm.com Embedded Linux Conference

539 views • 52 slides
Agenda Programming Model Linux x86 Architecture and respective adaption Memory

Agenda Programming Model Linux x86 Architecture and respective adaption Memory

Agenda Programming Model Linux x86 Architecture and respective adaption Memory Exceptions / Interrupts Multiprocessing 1 References Understanding the Linux Kernel, Bovet and Cesati Linux Device Drivers, Rubini

812 views • 49 slides
4+1 View of Architecture What does Linux have? Linux subsystems Process Scheduler (PS)

4+1 View of Architecture What does Linux have? Linux subsystems Process Scheduler (PS)

4+1 View of Architecture What does Linux have? Linux subsystems Process Scheduler (PS) responsible for supporting multitasking by deciding which user process executes. Memory Manager (MM) provides a separate memory space for

208 views • 17 slides
28.05.04 09:50 Memory Management The computer memory is a limited resource so the Memory

28.05.04 09:50 Memory Management The computer memory is a limited resource so the Memory

28.05.04 09:50 Memory Management The computer memory is a limited resource so the Memory Management memory use of programs has to be managed in some way. Memory Management The memory management is usually performed by a runtime system

782 views • 13 slides
Memory Barriers in the Linux Kernel Semantics and Practices Embedded Linux Conference April

Memory Barriers in the Linux Kernel Semantics and Practices Embedded Linux Conference April

Memory Barriers in the Linux Kernel Semantics and Practices Embedded Linux Conference April 2016. San Diego, CA. Davidlohr Bueso <dave@stgolabs.net> SUSE Labs. Agenda 1. Introduction Reordering Examples Underlying need for

773 views • 54 slides
Memory Management Memory Management Memory a linear array of bits, bytes, words, pages ... -

Memory Management Memory Management Memory a linear array of bits, bytes, words, pages ... -

CS510 Operating System Foundations Jonathan Walpole Memory Management Memory Management Memory a linear array of bits, bytes, words, pages ... - Each byte is named by a unique memory address - Holds instructions and data for OS and user

761 views • 56 slides
Solar to Low-Income Customers May 18, 2017 Housekeeping Use the red arrow to open and close

Solar to Low-Income Customers May 18, 2017 Housekeeping Use the red arrow to open and close

Bringing the Benefits of Solar to Low-Income Customers May 18, 2017 Housekeeping Use the red arrow to open and close your control panel Join audio: Choose Mic & Speakers to use VoIP Choose Telephone and dial using the

601 views • 45 slides
MA111: Contemporary mathematics Alex and Blair split a 2-bedroom apart- Master Other Alex $350

MA111: Contemporary mathematics Alex and Blair split a 2-bedroom apart- Master Other Alex $350

MA111: Contemporary mathematics Alex and Blair split a 2-bedroom apart- Master Other Alex $350 $325 ment for $550 per month so that each Blair $315 $300 Total $550 total was paying $50 less than they expected, Who gets it? Alex Blair

430 views • 6 slides
About Me Facts About Mortgages MLO with Homestar Financial Mortgage Business for 12+ years GREC

About Me Facts About Mortgages MLO with Homestar Financial Mortgage Business for 12+ years GREC

5/19/2020 Mortgage 101: About Me Facts About Mortgages MLO with Homestar Financial Mortgage Business for 12+ years GREC Course #70376 1 Beautiful Wife; 3 Handsome Sons Through Georgia Real Estate Academy GREC School #6915 GA Tech Graduate 05

430 views • 10 slides
Top 10 Mistakes Real Estate Investors Make Brandon Turner VP @ BP Twitter: @brandonatbp Get

Top 10 Mistakes Real Estate Investors Make Brandon Turner VP @ BP Twitter: @brandonatbp Get

Top 10 Mistakes Real Estate Investors Make Brandon Turner VP @ BP Twitter: @brandonatbp Get todays worksheet at www.BiggerPockets.com/1021Worksheet Thanks! (you rock!) Get todays worksheet at www.BiggerPockets.com/1021Worksheet What

1.77k views • 160 slides
Scalable Verification of Design with Multiple Properties Rohit Dureja Iowa State University

Scalable Verification of Design with Multiple Properties Rohit Dureja Iowa State University

Scalable Verification of Design with Multiple Properties Rohit Dureja Iowa State University October 23, 2019 Model Checking 2 Model Checking Usually multiple properties to be verified 3 Model Checking Report 4 Model Checking Report

446 views • 9 slides
DeepState : Bringing vulnerability detection tools into the development lifecycle Peter Goodman

DeepState : Bringing vulnerability detection tools into the development lifecycle Peter Goodman

DeepState : Bringing vulnerability detection tools into the development lifecycle Peter Goodman (Trail of Bits) Gustavo Grieco (Trail of Bits) Alex Groce (Northern Arizona University) 1 Introductions Peter Goodman Gustavo Grieco Alex

772 views • 73 slides
The CFRE Application Process: Step by Step Everything you want to know Who is CFRE?

The CFRE Application Process: Step by Step Everything you want to know Who is CFRE?

The CFRE Application Process: Step by Step Everything you want to know Who is CFRE? Certification process What is certification? CFRE examination What distinguishes How do I study? CFRE? Online resources

707 views • 39 slides
Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity

Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity

Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity Kernels in Different Applications M. Magdon-Ismail CSCI 4100/6100 recap: The Kernel Allows Us to Bypass Z -space Solve the QP 1

356 views • 12 slides

More recommend