Analyzing Memory Errors in Production
Markus Weninger Johannes Kepler University Linz, Austria Institute for System Software
in Production Markus Weninger Johannes Kepler University Linz, - - PowerPoint PPT Presentation
Analyzing Memory Errors in Production Markus Weninger Johannes Kepler University Linz, Austria Institute for System Software WHO ARE WE? Fixing Memory Problems In Production 2 WHO ARE WE? Fixing Memory Problems In Production 2 WHO ARE WE?
Markus Weninger Johannes Kepler University Linz, Austria Institute for System Software
Fixing Memory Problems In Production 2
Fixing Memory Problems In Production 2
Fixing Memory Problems In Production 2
Fixing Memory Problems In Production 2
Fixing Memory Problems In Production 2
Fixing Memory Problems In Production 2
For what? To find and fix memory anomalies!
Fixing Memory Problems In Production 2
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
3 Fixing Memory Problems In Production
4 Fixing Memory Problems In Production
* not actually Jake
4 Fixing Memory Problems In Production
* not actually Jake
4 Fixing Memory Problems In Production
* not actually Jake
5 Fixing Memory Problems In Production
5 Fixing Memory Problems In Production
5 Fixing Memory Problems In Production
5 Fixing Memory Problems In Production
5 Fixing Memory Problems In Production
5 Fixing Memory Problems In Production
5 Fixing Memory Problems In Production
6 Fixing Memory Problems In Production
6 Fixing Memory Problems In Production
1 2 3 4 5 6 Used a memory analysis tool before Manually took a heap dump Knows what trace-based memory analysis is Could describe what a dominator tree is Dummy Series 1 Dummy Series 2 Dummy Series 3
Fixing Memory Problems In Production 7
Fixing Memory Problems In Production 8
Fixing Memory Problems In Production 8
Fixing Memory Problems In Production 8
Top-down analysis
Fixing Memory Problems In Production 8
Top-down analysis GC root
Fixing Memory Problems In Production 8
Top-down analysis GC root
Fixing Memory Problems In Production 8
Top-down analysis GC root
Fixing Memory Problems In Production 8
Top-down analysis GC root
Fixing Memory Problems In Production 8
Top-down analysis GC root
Fixing Memory Problems In Production 8
Top-down analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive?
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive? GC root GC root
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive? GC root GC root 1 Mio. 10,000 10,000 10,000 1 Mio. 1 Mio.
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive? GC root GC root 1 Mio. 10,000 10,000 10,000 1 Mio. 1 Mio.
Fixing Memory Problems In Production 8
Top-down analysis Bottom-up analysis GC root
Which single object / GC root keeps the most other objects alive? GC root GC root 1 Mio. 10,000 10,000 10,000 1 Mio. 1 Mio.
Analyze why a certain group of
Fixing Memory Problems In Production 9
GC Roots L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
Object Reference Graph
Fixing Memory Problems In Production 9
GC Roots L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
Object Reference Graph Dominator Tree
Fixing Memory Problems In Production 10
GC Roots L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
Object Reference Graph Dominator Tree
Fixing Memory Problems In Production 10
GC Roots L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
Object Reference Graph Dominator Tree What happens if we could free L1?
Fixing Memory Problems In Production 11
GC Roots L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
L1 L2 N1 N2 N3 N4 N5 D1 D2 D3 D4 D5 D6 D7 D8 N6 N7 N8 N9
N10
Object Reference Graph Dominator Tree These can be collected too
Fixing Memory Problems In Production 12
Fixing Memory Problems In Production 12
Fixing Memory Problems In Production 12
Fixing Memory Problems In Production 12
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X()
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X()
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X()
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y()
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z()
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y()
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y()
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1 Heap 2
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1 Heap 2 Heap n
…
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1 Heap 2 Heap n
…
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1 Heap 2 Heap n
…
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1 Heap 2 Heap n
…
Fixing Memory Problems In Production 13
AntTracks VM Application t
Mutator phase
GC phase new X() new Y() new Z() new Y() move move move new Y() new Z()
Trace file AntTracks Analyzer Memory anomaly detection Memory leak analysis
Heap 1 Heap 2 Heap n
…
Fixing Memory Problems In Production 14
Fixing Memory Problems In Production 14
Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern.
Fixing Memory Problems In Production 14
Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern. Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually.
Fixing Memory Problems In Production 14
Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern. Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually. Explanation of the highlighted visualization helps users to interpret it and explains concepts that are needed for this interpretation.
Fixing Memory Problems In Production 14
Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern. Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually. Explanation of the highlighted visualization helps users to interpret it and explains concepts that are needed for this interpretation. Suggestions on which steps could / should be taken next make it easier for the user to understand what operations are possible and why they might be useful.
Fixing Memory Problems In Production 14
Detection describes the task of automatically detecting a potential problem, i.e., a suspicious pattern. Highlighting the relevant region on the UI helps users to understand where the automatically gained insight can be found if the view was inspected manually. Explanation of the highlighted visualization helps users to interpret it and explains concepts that are needed for this interpretation. Suggestions on which steps could / should be taken next make it easier for the user to understand what operations are possible and why they might be useful.
Fixing Memory Problems In Production 15
Fixing Memory Problems In Production 16
Fixing Memory Problems In Production 17
Fixing Memory Problems In Production 18
Fixing Memory Problems In Production 19
Fixing Memory Problems In Production 20
Fixing Memory Problems In Production 20
Date objects allocated in the constructor of Base are the major suspects for a memory leak since about 30% of the
heap growth is accounted to them. Inspect who keeps them alive.
Fixing Memory Problems In Production 21
Fixing Memory Problems In Production 22
Fixing Memory Problems In Production 23
Fixing Memory Problems In Production 23
Fixing Memory Problems In Production 23
Too many ArrayList<Location> are added here.
Fixing Memory Problems In Production 24
Fixing Memory Problems In Production 25
Fixing Memory Problems In Production 26
Fixing Memory Problems In Production 27
Fixing Memory Problems In Production 27
Fixing Memory Problems In Production 27
Fixing Memory Problems In Production 27
Fixing Memory Problems In Production 27
val response: Future[http.Response] = client(request)
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) }
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) }
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) } val h = { rep: http.Response => totalLength += rep.content.length } for (i <- 0 until NUM_REQUESTS { Await.result(response.onSuccess(h)) }
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) } val h = { rep: http.Response => totalLength += rep.content.length } for (i <- 0 until NUM_REQUESTS { Await.result(response.onSuccess(h)) }
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) } val h = { rep: http.Response => totalLength += rep.content.length } for (i <- 0 until NUM_REQUESTS { Await.result(response.onSuccess(h)) }
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) } val h = { rep: http.Response => totalLength += rep.content.length } for (i <- 0 until NUM_REQUESTS { Await.result(response.onSuccess(h)) }
Fixing Memory Problems In Production 28
val response: Future[http.Response] = client(request) for (i <- 0 until NUM_REQUESTS) { Await.result(response.onSuccess { rep: http.Response => totalLength += rep.content.length }) } val h = { rep: http.Response => totalLength += rep.content.length } for (i <- 0 until NUM_REQUESTS { Await.result(response.onSuccess(h)) }
Fixing Memory Problems In Production 28
Fixing Memory Problems In Production 29
Fixing Memory Problems In Production 29
Memory Analysis Top-down Bottom-up Heap dumps Trace-based ... and 100 other things
Fixing Memory Problems In Production 29
Memory Analysis Problem Top-down Bottom-up Heap dumps Trace-based ... and 100 other things New users need guidance to get started faster It is not enough to „throw tools at them“
Fixing Memory Problems In Production 29
Memory Analysis Problem Guided Exploration Top-down Bottom-up Heap dumps Trace-based ... and 100 other things New users need guidance to get started faster It is not enough to „throw tools at them“ Detection Highlighting Explanation Suggestion
Fixing Memory Problems In Production 29
Memory Analysis Problem Guided Exploration Top-down Bottom-up Heap dumps Trace-based ... and 100 other things New users need guidance to get started faster It is not enough to „throw tools at them“ Detection Highlighting Explanation Suggestion Markus Weninger
Johannes Kepler University Linz, Austria markus.weninger@jku.at http://mevss.jku.at/AntTracks http://bit.ly/weninger_ssw
Additional Notes
Some icons made by Freepik & Smashicons from https://www.flaticon.com Some photos made by bruce mars and Isaque Pereira from Pexels
Fixing Memory Problems In Production 30