Roadmap for Section 11.1 Performance Evaluation and Prediction - - PDF document

1

Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze

Unit OS11: Performance Evaluation

11.1. System Performance

3

Roadmap for Section 11.1

Performance Evaluation and Prediction Tools for Monitoring Windows Internals Performance Monitor and mmc Scheduling-related Performance Counters Memory-related Performance Counters Windows Event Tracing

2

4

Performance Prediction and Evaluation

Constructing a model of the system and then using the model to predict the system's behavior

Model reflects system structure or organization as well as its workload or input Analyzed using mathematical techniques Alternatively, the model may be simulated

Benchmarking & Monitoring

Evaluating behavior of a live system Predefined workloads

5

Modeling Approaches

Analytic modeling techniques

Discrete- and continuous-time Markov chains Queueing theory, and queueing networks Approximate methods based on these techniques

Operational analysis

Non-stochastic, measurement-based perspective to the analysis of computer systems Modeled with discrete-event simulation Performance metrics from stochastic simulations are subject to statistical analysis (as are data

• btained from real systems)

3

6

Validity of Models

Models, whether analytic or simulation, can be inaccurate or implemented incorrectly

An important aspect of any kind of performance modeling study is to validate the model and its implementation to whatever extent is possible One way to do this is to study a system using more than one model, e.g., a simulation model and an analytic model

Analytic modeling of many systems is computationally demanding

7

Monitoring Windows - How to obtain Performance Data

Windows is thoroughly instrumented Performance counters allow for monitoring of most kernel objects

Many tools available to dig into Windows internals Helps to see internals behavior “in action”

Several sources of tools

Support Tools Resource Kit Tools Debugging Tools Sysinternals.com

Additional tool packages with internals information

Platform Software Development Kit (SDK) Device Driver Development Kit (DDK)

4

8

Tool Image Name Origin

File Monitor FILEMON www.sysinternals.com Global Flags GFLAGS Support Tools Handle Viewer HANDLE www.sysinternals.com Kernel debuggers WINDBG, KD Debugging tools, Platform SDK, Windows DDK Live Kernel Debugging LIVEKD www.sysinternals.com Open Handles OH Resource kits Page Fault Monitor PFMON Support Tools, Resource kits, Platform SDK Pending File Moves PENDMOVES www.sysinternals.com Performance tool PERFMON.MSC Windows built-in tool Pool Monitor POOLMON Support Tools, Windows DDK Process Explorer PROCEXP www.sysinternals.com Process Statistics PSTAT Support Tools, Windows 2000 Resource kits, Platform SDK, www.reskit.com Quick Slice QSLICE Windows 2000 resource kits Task (Process) List TLIST Debugging tools Task Manager TASKMGR Windows built-in tool TDImon TDIMON www.sysinternals.com

Tools for Windows Performance Monitoring

9

Process Explorer (Sysinternals)

Shows performance-related data

…plus full image path, command line, environment variables, parent process, security access token, open handles, loaded DLLs & mapped files

5

10

Obtain System Information with Process Explorer

Click View->System Information

11

Overview of Performance Data Collection

Windows defines performance data in terms of objects, counters, and instances

A performance object is any resource, application, or service that can be measured System Monitor and Performance Logs and Alerts allow to select performance objects, counters, and instances to collect and present performance data

Objects have performance counters

Objects may also have instances, which are unique copies of a particular object type Not all object types support multiple instances _Total instance represents the sum of the values for all instances of the object for a specific counter

6

12

Vast Array of Performance Data

13

Performance Counter Aggregation into Performance Logs (via mmc)

7

14

Real-time Data Collection with Performance Monitor

15

Windows Performance Counters -

Categories

Monitoring Memory Management

Memory\ Page Reads/sec Memory\ Page Writes/sec Memory\ Available Bytes Process\ Working Set Process\ Private Bytes

8

16

Windows Performance Counters -

Categories (contd.)

Monitoring Physical and Logical Disk I/O

PhysicalDisk\ % Disk Time PhysicalDisk\ Avg. Disk Queue Length PhysicalDisk\ Current Disk Queue Length PhysicalDisk\ Avg. Disk Sec/Read PhysicalDisk\ Avg. Disk Sec/Write PhysicalDisk\ Disk Read Bytes/sec PhysicalDisk\ Disk Write Bytes/sec PhysicalDisk\ Avg. Disk Bytes/Read PhysicalDisk\ Avg. Disk Bytes/Write PhysicalDisk\ Disk Reads/sec PhysicalDisk\ Disk Writes/sec

17

Windows Performance Counters -

Categories (contd.)

Monitoring Network Activities

Network Interface\ Bytes Total/sec Network Interface\ Bytes Sent/sec Network Interface\ Bytes Received/sec Protocol_layer_object\ Segments Received/sec Protocol_layer_object\ Segments Sent/sec Protocol_layer_object\ Frames Sent/sec Protocol_layer_object\ Frames Received/sec Server\ Bytes Total/sec Server\ Bytes Received/sec Server\ Bytes Sent/sec Network Segment\ % Network Utilization

9

18

Analyzing Processor Activity

Determine the baseline on normal workload (from several weeks to a month)

Processor\ % Processor Time counter System\Processor Queue Length counter

Be aware of the Idle process …

The Idle process runs a thread on each processor To measure the Idle process, use the Process(Idle)\ % Processor Time counter, or Processes tab in Task Manager Zero idle time could mean that the processor is handling a lot

• f work, but it could also mean that the processor or central

processing unit (CPU) is overloaded

19

Detecting Processor Bottlenecks

CPU bottlenecks are indicated by:

Processor\ % Processor Time often exceeds 80 percent (and there is no compute-bound workload) System\ Processor Queue Length is often greater than 2 on a single-processor system Queue Length is the single most important parameter

Other indications:

Unusually high values appear for the Processor(_Total)\ Interrupts/sec or System\ Context Switches/sec counters

10

20

Evaluating Memory Usage

Establish a reference point (or baseline) for physical memory usage under normal workload

Create logs of memory usage over an extended period (from several weeks to a month)

Relevant Performance Counters

\Memory\Available Bytes \Paging File(_Total)\% Usage

Exclude spikes; the range of values that seem to appear consistently constitutes your baseline

21

Detecting Memory Bottlenecks

Indication for insufficient memory:

Value for Memory\Available Bytes is consistently low (e.g. less than 5% of RAM)

If available memory is consistently low, the computer becomes unresponsive:

It is occupied exclusively with disk I/O operations During paging due to low memory, the processor is idle while waiting for the disk to finish

11

22

Examining Disk Performance

Monitor disk counters along with counters from other

• bjects. The following is a list of recommended counters.

LogicalDisk\% Free Space PhysicalDisk\Disk Reads/sec PhysicalDisk\Disk Writes/sec PhysicalDisk\Avg. Disk Queue Length Memory\Available Bytes Memory\Cache Bytes Memory\Pages/sec Processor(All_Instances)\% Processor Time System\Processor Queue Length

23

Detecting a Disk Bottleneck

• Avg. Disk Queue Length for LogicalDisk or PhysicalDisk

If the value of Avg. Disk Queue Length exceeds twice the number of spindles, then you are likely developing a bottleneck With a volume set, a queue that is never shorter than the number of active physical disks indicates that you are developing a bottleneck Notice that this might overstate the true length of the queue, because the counter includes both queued and in-service requests

12

24

Counters by Feature

Internet Information Service

Active Server Pages FTP Service Web Service Internet Information Services Global

Indexing Service

Indexing Service Indexing Service Filter HTTP Indexing Service

Message Queuing

MSMQ Session MSMQ IS MSMQ Queue MSMQ Service Quality of Service (QoS) Admission

Control

ACS/RSVP Service ACS/RSVP Interfaces ACS/RSVP Policy

Routing and Remote Access (RRAS)

RAS Port RAS Total

File Replication Service

FileReplicaConn FileReplicaSet

Terminal Service

Terminal Services Session

Active Directory™

NTDS

Windows services and apps may bring their

• wn performance objects

25

Kernel Event Tracing

Windows kernel and core device drivers are instrumented to record trace data Event Tracing for Windows (ETW)

Common infrastructure in the kernel that provides trace data to the user-mode facility

ETW is accessed by:

Controllers that start and stop logging sessions and manages buffer pools Providers that define GUIDs for the event classes they can produce traces for; act on Controllers’ commands Consumer select one or more trace sessions for which the want to read trace data (in real-time or in log files)

13

26

ETW Providers and Kernel Logger

Windows Server systems include several built-in providers in user mode

Active Directory, Kerberos, and Netlogon ETW defines a logging session with the name NT Kernel Logger (kernel logger) for use by the kernel and core drivers. NT Kernel Logger provider is implemented in the kernel

27

User mode controllers may enable kernel logger - ETW Operation

ETW library sends I/O control request to the WMI driver to enable tracing on a particular event class

ETW library is implemented in \Windows\System32\Ntdll.dll If file logging is configured the WMI driver creates a system thread in system process that creates a log file Alternatively, logging may use an in-memory buffer

The WMI driver records trace events to a buffer

File logging thread wakes up once per second to dump the contents

• f the buffers to the log file

Trace records generated for the kernel logger have a standard ETW trace event header

Header records timestamp, process, and thread IDs, info on event class Event classes can provide additional data specific to their events

14

28

Trace and Logging Data may be accessed via mmc

29

Kernel Logger Trace Classes

Traces classes and their generating components

Disk I/O - disk class driver File I/O - file system drivers Hardware Configuration - plug&play manager Image Load/Unload - system image loader in the kernele Page Faults - memory manager Process Create/Delete - process manager Thread Create/Delete - process manager Registry Activity - Configuration Manager TCP/UDP Activity - TCP/IP driver

ETW controllers/providers described in Platform SDK

15

30

Using Event Logs and Performance Counters for Optimization

Performance must be tuned to a workload

A sequence of service requests, commands, I/Os that exercise the software Often produced by workload generators rather than real-world service provision Allow for replay or generation of service requests at a maximum rate so that bottlenecks can be identified in systems

Most workloads are domain-specific

I.e.; TPC benchmarks and workloads No “optimal” system configuration per se

31

Further Reading

Mark E. Russinovich and David A. Solomon, Microsoft Windows Internals, 4th Edition, Microsoft Press, 2004.

Chapter 1 - Concepts and Tools Performance Tool, Support Tools, Resource Kits, pp. 25-34 Chapter 4 - Management Mechanisms Windows Management Instrumentation, pp. 237-249 Chapter 3 - System Mechanisms Event Tracing for Windows, pp. 177-180

Windows Performance Counter Reference

http://msdn.microsoft.com/library/default.asp?url=/library/en- us/counter/default.asp

16

32

Source Code References

Windows Research Kernel (WRK):

\base\ntos\perf – core performance logging support \base\ntos\ex\sysinfo.c – system/process performance query functions