Towards Automatic Inference of Task Hierarchies in Complex Systems - - PowerPoint PPT Presentation

▶
towards automatic inference of task hierarchies in
SMART_READER_LITE
LIVE PREVIEW

Towards Automatic Inference of Task Hierarchies in Complex Systems - - PowerPoint PPT Presentation

Nov 10, 2023 336 likes •568 views

Overview Design & Implementation Case Study Conclusion Acknowledgment References Towards Automatic Inference of Task Hierarchies in Complex Systems Haohui Mai Chongnan Gao Xuezheng Liu Xi Wang Geoffrey M. Voelker

slide-1
SLIDE 1

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Towards Automatic Inference of Task Hierarchies in Complex Systems

Haohui Mai◦ Chongnan Gao† Xuezheng Liu‡ Xi Wang§ Geoffrey M. Voelker∗

University of Illinois at Urbana-Champaign◦ MIT CSAIL§ Microsoft Research Asia‡ Tsinghua University† University of California, San Diego∗

December 7, 2008

slide-2
SLIDE 2

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Motivation

System models are valuable

Visualize the design and the implementation Understand the structures of components and their dependency Present dependability measures in an intuitive way Reason and verify the system

Developers can represent the system as a hierarchical task model

Encapsulate implementation details with high-level tasks Allow developers to address dependability problems at various task granularities

slide-3
SLIDE 3

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Our work

Explored how well the hierarchical task models can be automatically inferred

With minimal or no manual assistance

Designed and Implemented Scalpel to automatically infer hierarchical task models in complex systems Applied Scalpel to two systems

Apache HTTP Server PacificA distributed storage system [Lin et al. , 2008] Encouraging results

slide-4
SLIDE 4

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Challenges

All should be done automatically in complex systems Identify appropriate task boundaries Associate dependencies among tasks correctly Recover the hierarchical structure among tasks

Local Local Commit

Commit

Session:: RecvPacket Logic Logic Replica:: Mutate RpcCient RpcCient:: RpcAsync CallChain CallChain Session:: Session:: WSASend Packet LogicReplica LogicReplica:: Receive

PAThreadPool PAThreadPool::

Packet MutationAck MutationAck Session:: Session::

PAThreadPool:: ThreadInternal ::DoWork

RpcCient RpcCient:: RpcCall Reply LogicReplica LogicReplica:: Receive WSASend Packet Reply MutationAck Receive MutationAck

Remote Commit Acknowledgment Remote Commit Acknowledgment Worker Worker

slide-5
SLIDE 5

Overview Design & Implementation Case Study Conclusion Acknowledgment References

How it works

Collecting Execution Identifying Execution Traces Leaf Tasks Constructing Inferring Hierarchical Causal Graph Hierarchical Structure

slide-6
SLIDE 6

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Collecting Execution Traces

Trace down calls and their parameters of

Synchronization primitives (signal and wait) Socket communication (send and recv)

Leverage library-based record & replay tool named R2 [Guo et al. , 2008] in our implementation

slide-7
SLIDE 7

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Identifying Leaf Tasks

Leaf task: smallest unit of work in a task model Paritition the execution traces with synchronization points Synchronization point: where two threads synchronize their execution and establish a happens-before relation Rationale

Dependency only occurs at boundaries Relatively independent and self-contained

Recv(commit_ack); … Wait(queue_lock); enqueue(element); … ++seq_number; … enqueue(element); … SetEvent(qevent); …

slide-8
SLIDE 8

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Constructing Causal Graph

Use happens-before relation to infer causal dependency Distinguish causal dependency and occasional “run-after” relation

Producer - Consumer Mutual exclusion

Heuristics

OS-provided queues (I/O completion ports) Notification mechanisms (events) Efficient to catch shared queues

Wait(write_lock); … saveToDisk(data); Recv(commit_ack); … Send(commit_ack); … … ++seq_number; …

slide-9
SLIDE 9

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Inferring Hierarchical Structure

Idea: Identifying frequent patterns in causal graph Replace frequent patterns with “super nodes” recursively Identifying frequent patterns

Canonize sub graph and serialize it deterministically Use hash functions for exact matching

F D A B C E G D A B C E F G ABC(D)E

slide-10
SLIDE 10

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study

Effectiveness of the task models for debugging Effectiveness of capturing developers’ intuition All experiments on machines with 2.0 GHz Xeon dual-core CPUs, 4 GB memory, running Windows Server 2003 Service Pack 2, and interconnected via a 1 Gb switch

slide-11
SLIDE 11

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study: Performance Bug in PacificA (I)

Performance is not satisfactory under stress tests Task level profiling based on inferred task models Use a top-down approach to identify the problem

Use a profiler to collect performance numbers

Latency Network bandwidth CPU cycles

Aggregate profiling data in a per-task manner for each layer

Worker Thread Data Replication (2PC) Configuration Configuration Manager (Paxos) Liveness Monitoring

slide-12
SLIDE 12

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study: Performance Bug in PacificA (II)

The committing task could not saturate network bandwidth, while at the same time the CPU usage remained low

Local Local Commit

Commit

Session:: RecvPacket Logic Logic Replica:: Mutate RpcCient RpcCient:: RpcAsync CallChain CallChain Session:: Session:: WSASend Packet LogicReplica LogicReplica:: Receive

PAThreadPool PAThreadPool::

Packet MutationAck MutationAck Session:: Session::

PAThreadPool:: ThreadInternal ::DoWork

RpcCient RpcCient:: RpcCall Reply LogicReplica LogicReplica:: Receive WSASend Packet Reply MutationAck Receive MutationAck

Remote Commit Acknowledgment Remote Commit Acknowledgment Worker Worker

slide-13
SLIDE 13

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study: Performance Bug in PacificA (III)

Sender threads will block at a call to sleep() for 1 second

3 4

d ID

Time of RPC calls (1 time unit = 10ms)

1 2 50 100 150 200 250 300

Thread

Thr‐1 Thr‐2 Thr‐3 Thr‐4

int Session :: WSASendPacket(NetworkStream * pkt) { CAutoLock guard(_send_lock ); while (_send_size > (64 << 20)) // 64 MB Sleep (1000); ... int rt = WSASend(_socket , buf , buf_num , &bytes , 0, (OVERLAPPED *)ce , 0); ... }

slide-14
SLIDE 14

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study: Performance Bug in PacificA (IV)

Root cause: there is no flow control mechanism at RPC layer when it uses asynchronous communication

Thread sends messages in a non-blocking fashion Network layer blocks the thread when the internal buffer is full Threads will all be blocked by the network layer synchronously at high workload

Caused by poor interactions across software layers A clear hierarchical model helped developers to identify the location of the bug and also understand its root cause

slide-15
SLIDE 15

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study: Task Model of Apache HTTP Server

Successfully capture the Apache service cycle for SVN checkout operations

winnt_ t 2 ap_core_ accept_2 input_filter i i t winnt_get_ connection svn_io_ file_read winnt_ accept_1 connection svn_io_ file_getc … mpm_get_ completion_ mpm_recycle_ completion_ ap_linger l context context ing_close

slide-16
SLIDE 16

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Case Study: Statistics

Apache PacificA SLOC 819676 54458 Leaf Tasks 423952 10636 Events 47 IOCP 23 16 Socket 527 77 Mutex 210472 4950 Same thread 193972 11304 Running Time: Extracting Task Models 5.95s 32.02s Running Time: Native run 9.66s 20.79s Running Time: Execution Time 10.00s 28.36s Overhead 3.52% 36.41%

slide-17
SLIDE 17

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Conclusion & Future Work

Hierarchical task models of complex systems can be inferred with few or no annotations Future work

Extend the trace collecting method to collect memory

  • perations

More effective heuristics to prune “run-after” cases Experiment more graph mining algorithm for recovering task hierarchies Evaluate more systems

slide-18
SLIDE 18

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Acknowledgment

Our colleague Wenguang Chen for valuable feedback and discussions Zhenyu Guo for the R2 work Wei Lin and other PacificA developers Support from System Research Group in Microsoft Asia and High Performance Computing Group in Tsinghua University

slide-19
SLIDE 19

Overview Design & Implementation Case Study Conclusion Acknowledgment References

References

Guo, Zhenyu, Wang, Xi, Tang, Jian, Liu, Xuezheng, Xu, Zhilei, Wu, Ming, Kaashoek, M. Frans, & Zhang,

  • Zheng. 2008.

R2: An Application-Level Kernel for Record and Replay. In: OSDI ’08: Proceedings of the 8th symposium on Operating systems design and implementation. Lin, Wei, Yang, Mao, Zhang, Lintao, & Zhou, Lidong. 2008 (February). PacificA: Replication in Log-Based Distributed Storage Systems.

  • Tech. rept. MSR-TR-2008-25. Microsoft Research Asia.
slide-20
SLIDE 20

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Collect Execution Traces and Identify Leaf Tasks

1 3 4 Acceptor 2 1 3 4 2 5 6 7 8 9 A B 5 6 7 8 9 A B Worker A

winnt_accept() { while (…) { … mpm_get_completion_context(); worker_main() { while (…) { … winnt_get_connection(); l l ti t t() 1 5 Wait(qlock); … Wait(qwait event); mpm_recycle_completion_context(); Wait(qlock); SetEvent(qwait_event); … 3 6 2 (q _ ); … AcceptEx(conn); Wait(conn); GetQueue(ThrDispatch); … ap_process_connection(); 3 4 7 … PostQueue(ThrDispatch); … } } ap_core_input_filter(); svn_io_read(); svn_io_getc(); ap_lingering_close(); 8 9 A B … } }

slide-21
SLIDE 21

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Constructing Causal Graph

1 3 4 Acceptor 2 1 3 4 2 5 6 7 8 9 A B 5 6 7 8 9 A B Worker A

winnt_accept() { while (…) { … mpm_get_completion_context(); worker_main() { while (…) { … winnt_get_connection(); l l ti t t() 1 5 Wait(qlock); … Wait(qwait event); mpm_recycle_completion_context(); Wait(qlock); SetEvent(qwait_event); … 3 6 2 (q _ ); … AcceptEx(conn); Wait(conn); GetQueue(ThrDispatch); … ap_process_connection(); 3 4 7 … PostQueue(ThrDispatch); … } } ap_core_input_filter(); svn_io_read(); svn_io_getc(); ap_lingering_close(); 8 9 A B … } }

slide-22
SLIDE 22

Overview Design & Implementation Case Study Conclusion Acknowledgment References

Inferring Hierarchical Structure

1 4 Acceptor 2 1 4 2 5 C(3,6) 7 D(8,9,A,B) Worker 5 C(3,6) 7 D(8,9,A,B)

winnt_accept() { while (…) { … mpm_get_completion_context(); worker_main() { while (…) { … winnt_get_connection(); l l ti t t() 1 5 Wait(qlock); … Wait(qwait event); mpm_recycle_completion_context(); Wait(qlock); SetEvent(qwait_event); … 3 6 2 (q _ ); … AcceptEx(conn); Wait(conn); GetQueue(ThrDispatch); … ap_process_connection(); 3 4 7 … PostQueue(ThrDispatch); … } } ap_core_input_filter(); svn_io_read(); svn_io_getc(); ap_lingering_close(); 8 9 A B … } }