Cost Dependability and Security Johan Karlsson Energy-aware - - PDF document

1

The trade-off betw een energy consumption and dependability consumption and dependability

Johan Karlsson

Department of Computer Science and Engineering Chalmers University of Technology Göteborg, Sweden

Trade-offs in Computer System Design

Cost

Dependability and Security

Johan Karlsson 2 Energy-aware computing

2

Layered fault tolerance

Catastrophic f il Benign f il Safe Sh td

System failure modes

Detected Error Undetected Error Error corrected Timing failure Bounded failure Value failure Fail silent Fail signal Error corrected failure failure Shutdow n

• st balancing

System failure modes

Software mechanisms

2 nd line of defense

System mechanisms

3 rdline of defense

Processor failure modes

Johan Karlsson 3 Energy-aware computing

SW Design Faults HW Design Faults Physical Faults Error Corrected Error Error

C Hardware mechanisms

1 st line of defense

Outline

• Trends in integrated circuit reliability

g y

• HP NonStop Advanced Architecture

– Traditional approach to fault tolerance in high-end servers

• IBM Power7 processor

– Energy control – Chip-level fault tolerance

• Software implemented hardware fault tolerance
• Final reamrks

Johan Karlsson 4 Energy-aware computing

3

Transistor variability and degradation

Shekhar Borkar, Intel Corp: “As technology scales, variability in transistor performance will continue to increase, making transistors less and less reliable. …. Finding solutions to these challenges will require a concerted effort on the part of all the players in a system design ”

Johan Karlsson 5 Energy-aware computing

the players in a system design.

Borkar, S.; "Designing reliable systems from unreliable components: the challenges of transistor variability and degradation," IEEE Micro, December 2005.

Trends in the bathtube curve

Infant mortality Constant failure rate Wear out

Failure rate

• Infant mortality: Increasing manufacturing defects
• Constant failure rate: Increasing rate of transient, intermittent and permanent faults
• Wearout: Acceleration of aging phenomena

Johan Karlsson 6 Energy-aware computing

Time

Source: Vikas Chandra, ARM R&D, Dependable Design in Nanoscale CMOS Technologies: Challenges and Solutions Keynote address, WDSN, Estoril, Portugal, June 29, 2009

1 – 20 weeks 3 – 10 years

4

Sources of transistor failures

• Process variations (intermittent and permanent faults)

( p )

– Random variations related to lithography, etching, dopant count – Voltage and temperature variations

• Wear out effects (intermittent and permanent faults)

– NBTI - negative bias temperature instability – HCI - hot carrier injection – Gate oxide breakdown – Electromigration – …

• Ionizing particle radiation (mostly transient faults)

– Cosmic neutrons, alpha particles, muons, … – Soft errors (single event upsets) – no permanent damage – Hard errors (permanent faults) – permanent damage

Johan Karlsson 7 Energy-aware computing Electromigration

Gate oxide breakdow ns

• Gate oxide breakdowns increase

leakage currents and change g g electrical characteristics of transistors

Johan Karlsson 8 Energy-aware computing

Gate oxide in 90 nm technology Thickness: 5 atom layers Gate oxide scaling

Source: Intel 2005

5

Development of Gate-Oxide Breakdow n

Johan Karlsson 9 Energy-aware computing

Single Event Effects (SEE)

Disturbance caused by a single ionizing particle Disturbance caused by a single ionizing particle Types of SEE:s

• Upset (SEU) – change in logic state (bit-flips) by direct hit in

memory element, e.g., flip-flop or SRAM cell

• Transient (SET) – voltage pulse in combinational network, may

lead to single bit or multiple bit upset

• Latchup (SEL) – triggering of parasitic pnpn structure
• Burnout (SEB) of high voltage device, e.g., power transistor

Johan Karlsson 10 Energy-aware computing

6

Soft Errors

• Soft errors (or single event upsets)

are particle induced upsets (bit-flips) are particle induced upsets (bit flips)

• Caused by highly energetic particles

such as neutron, protons and muons

SiO2 gate Poly Si gate Drain Source Particle trajectory

Bit-flips SRAM cell

Johan Karlsson 11 Energy-aware computing

Si substrate p

SiO2 gate n+ n+ Particle strike in n-channel MOSFET transistor Depletion region

Flux of cosmic ray-induced high-energy neutrons

– The neutron flux is influenced by latitude, longitude, altitude,

y g atmospheric pressure, and solar activity

– Reference point: New York City, sea-level, medium solar activity

• Total flux at NYC is 12.9 cm-2 h-1 for neutron energies > 10

MeV

• Roughly 10 times higher at an altitude of 3000 meters

– The neutron flux at a specific location can be calculated at

http://www seutest com http://www.seutest.com

– More information can be found in the JEDEC Standard:

JESD89A - Measurement and Reporting of Alpha Particle and Terrestrial Cosmic Ray-Induced Soft Errors in Semiconductor Devices (October, 2006)

Johan Karlsson 12 Energy-aware computing

7

Soft error rate trend for SRAM & Flip-Flops

(Radiation test data from Sun Microsystems)

Johan Karlsson 14 Energy-aware computing

Source: A. Dixit, R. Heald, and A. Wood, “Trends from Ten Years of Soft Error Experimentation, SELSE´09, Stanford, CA, USA.

1 FIT = 10-9 faults per hour

Raw soft error rate trend for microprocessors

(Data from Sun Microsystems)

Technology node (nm) Year introduced Relative SEU rate in FITs/kbit Mbits/processor Relative uncorrected SEU rate / FITs/kbit SEU rate / microproces sor

250 1998 3.2 1.52 5.0 180 1999 3.0 1.52 4.3 130 2000 2.4 3.28 7.9 90 2002 1.0 33.6 33.6 65 2006 0.7 44.3 30.5 40 2008 0.94 71 67

Johan Karlsson 15 Energy-aware computing

Source: A. Dixit, R. Heald, and A. Wood, “The Impact of New Technology on Soft Error Rates, SELSE-6, Stanford, CA, USA, 2010

1 FIT = 10-9 faults per hour

8

Circuit w ear out

Keane, J.; Kim, C.H.; , "An odometer for CPUs," IEEE Spectrum, May 2011

16 Johan Karlsson Energy-aware computing

HP’s NonStop Computer Systems

• Highly available computers for on-line transaction

Highly available computers for on line transaction processing (OLTP) systems

• Typical applications:

– Automatic teller machines, Stock trading, Funds transfer,

911 emergency centers, Medical records, Travel and hotel reservations, etc

• Availability: 0 99999

“five nines” or 5 min

• Availability: 0,99999 –

five nines , or 5 min downtime per year

• Data integrity: 1 FIT = 10-9 undetected errors per hour

(one undetected data error per billion hours)

Johan Karlsson 17 Energy-aware computing

9

associated hardw are announcements

18 Johan Karlsson Energy-aware computing

Marketing information from HP

(from 2005)

• Telecommunications

– 135 public telephone companies currently rely on NonStop

technology.

– More than half of all 911 calls in the United States and the

majority of wireless calls worldwide depend on NonStop servers.

• Finance

– Eighty percent of all ATM transactions worldwide and 66

percent of all point-of-sale transactions worldwide are percent of all point-of-sale transactions worldwide are handled by NonStop servers.

– NonStop technology powers 75 percent of the world’s 100

largest electronic funds transfer networks and 106 of the world’s 120 stock and commodity exchanges.

Johan Karlsson 19 Energy-aware computing

10

NonStop System w ith self-checked processors

Self-checked processors Self-checked processors

• Stop promptly if an error occurs
• Prevent error propagation

Process pairs

• Critical software is implemented as a

process pair, with one primary and one backup process executing on different processors

• Th

i t th

• The primary process execute the

program and sends state changes regularly to the backup process

• Backup process takes over if the primary

process fails by itself or as a result of a processor failure

Energy-aware computing 20 Johan Karlsson

Logical Processors

Johan Karlsson 21 Energy-aware computing

11

IBM Pow er7 processor

• Released in 2010, successor to the dual core Power 6 processor (released in

2007)

• Implements Power ISA v. 2.06 revision B (July 2010)
• Fabricated in 45 nm SOI, 567 mm2, 1.2 billion transistors
• 8 cores
• Each core has 12 executions units: two fixed-point units, two load-store units,

four double-precision floating-point units, one vector unit , one branch execution unit, one condition register unit, and decimal floating-point unit.

• Each core can fetch up to 8 instructions, decode and dispatch up to 6

p , p p instructions, and issue and execute up to 8 instructions in one clock cycle.

• Two on-chip memory controllers. Each memory controller supports four DDR3

memory channels, yielding a total memory bandwidth of 100 Gbytes/s

• Scales to 32 socket systems with 1024 threads

Johan Karlsson 22 Energy-aware computing

Pow er 7 High Volume Card

Johan Karlsson 23 Energy-aware computing

12

Pow er ISA v.2.06

• RISC load/store architecture.
• Thirty-two 32-bit or 64-bit General Purpose Registers (GPRs) for integer operations
• Sixty-four 128-bit Vector Scalar registers (VSRs) for vector operations and floating point
• perations.

– Thirty-two 64-bit Floating Point Registers (FPRs) as part of the VSRs for floating point

• perations.

– Thirty-two 128-bit Vector registers (VRs) as part of the VSRs for vector operations.

• Eight 4-bit Condition register fields (CRs) for comparison and flow control.
• Special registers: Counter Register (CTR), Link Register (LR), Time Base (TBU, TBL),

Alternate Time Base (ATBU, ATBL), Accumulator (ACC), Status registers (XER, FPSCR, VSCR, SPEFSCR).

• Instructions have a length of 32 bits, with the exception of the VLE (variable-length

encoding) subset that provides for higher code density for low-end embedded applications.

• Most instructions are triadic, i.e. have two source operands and one destination.
• Single and double precision IEEE-754 compliant floating point operations are supported,

including additional fused multiply–add (FMA) and decimal floating-point instructions.

• There are provisions for SIMD operations on integer and floating point data on up to 16

elements in a single instruction.

Johan Karlsson 24 Energy-aware computing Johan Karlsson 25 Energy-aware computing

13

Johan Karlsson 26 Energy-aware computing Johan Karlsson 27 Energy-aware computing

14

Pow er management – Idle modes

• Each core has two idle modes: Nap and Sleep

p p

• Nap mode favors wake-up latency over power saving

– All execution unit clocks are turned off – Caches and TLBs remain coherent to reduce make-up time – Frequency of clocks for cache and TLB logic can optionally be reduced

• Sleep mode favors power savings over wake-up latency

– All clocks (cache + core) turned off – Caches and TLB are purged – Voltage dropped to retention level – Retention voltage keeps state of configuration registers intact to reduce

wake-up time. Core re-initialization not required

– The 45 nm process has very low leakage current at retention voltage

Johan Karlsson 28 Energy-aware computing

Pow er management – non-idle load

• Dynamic voltage and frequency scaling (DVFS)

y g q y g ( )

– The clock frequency can be controlled individually for each core

• DVFS directed by an off-chip EnergyScale microcontroller
• The EnergyScale microcontroller monitors on-chip counters to measure

system utilization

• Uses “power proxies” to estimate switching power of cores and memory

subsystems

– Cannot measure power consumption of each core directly – Measures 50 different architectural events to estimate power consumption

• The EnergyScale microcontroller can enforce a “hard ceiling” on power

consumption for a system, or even a data center.

Johan Karlsson 29 Energy-aware computing

15

Johan Karlsson 30 Energy-aware computing Johan Karlsson 31 Energy-aware computing

16

Johan Karlsson 32 Energy-aware computing

RAS* features of Pow er7 Main memory

• Main memory protected by 64-byte error correcting code (ECC)

y p y y g ( )

– Corrects 8-bit device failure (chip kill) on the fly – General double-bit error detection and single-bit correction.

• Hardware assisted memory scrubbing
• The memory buffer chip supports use of spare memory devices
• Selective mirroring of main memory in different memory channels

– Used for protecting memory used by the hypervisor and critical applications.

• The bus transferring data between the processor and the memory is

protected by CRC error detection and a failed operation retry mechanism

*RAS = reliability, availability, serviceability Johan Karlsson 33 Energy-aware computing

17

RAS features of Pow er7 Caches

• L1 cache protected by parity-bit and instruction retry

p y p y y – Erroneous L1-D and L1-I data reloaded from L2 (L1-D is store-through)

• L2 and L3 caches protected with double-bit-detect and single-

bit-correct ECC

• Redundant repair bits in L1-I, L1-D and L2 caches

– Permanently faulty set in L1 can be deleted (marked as faulty) – Permanently faulty cache line in L2 can be deleted (marked as faulty)

• Special uncorrectable error (SUE) handling

– Handling of uncorrectable cache and main memory errors – Occurs when correct data cannot be fetch from the next lower level of the memory

hierarchy

– Hardware signals error to operating system or hypervisor – Reboot of process or OS using the corrupted data – Entire system must be rebooted if corrupted data belongs to the hypervisor

Johan Karlsson 34 Energy-aware computing

RAS features of Pow er7 Processor cores

• Execution units protected by error detection circuitry

p y y

• Instruction retry for transients fault

– Leverages mechanisms for speculative execution to flush errors detected in general-

purpose, floating-point and vector-scalar registers.

• Alternate processor recovery for permanent faults

– Threads are stopped and moved to another core if instruction retry is unsuccessful

multiple times

• Some core errors are not recoverable

Some core errors are not recoverable – Architected state corrupted – Generates a “core-contained checkstop” (hardware exception) – Causes a failure of the workload running on the core

Johan Karlsson 35 Energy-aware computing

18

Layered fault tolerance

Catastrophic f il Benign f il Safe Sh td

System failure modes

Detected Error Undetected Error Error corrected Timing failure Bounded failure Value failure Fail silent Fail signal Error corrected failure failure Shutdow n

• st balancing

System failure modes

Software mechanisms

2 nd line of defense

System mechanisms

3 rdline of defense

Processor failure modes

Johan Karlsson 36 Energy-aware computing

SW Design Faults HW Design Faults Physical Faults Error Corrected Error Error

C Hardware mechanisms

1 st line of defense

From Kellington et al., IBM POWER6 Processor Soft Error Tolerance Analysis Using Proton Radiation, available at www.selse.org

19

Softw are-based error detection/error masking mechanisms

• Triple Time Redundant execution with Forward

Triple Time Redundant execution with Forward Recovery (TTR-FR)

• Time Redundancy And More (TRAM)

– Double time redundant execution – plus 5 other error detection mechanisms

Johan Karlsson 38 Energy-aware computing

Triple time redundant execution w ith forw ard recovery (TTR-FR)

Purpose: Error masking and error detection Purpose: Error masking and error detection

• Executes each control loop three times
• Errors masked by majority voting
• Three copies of program state
• Forward recovery: erroneous program state replaced

ith t t f t i with program state of correct version

• Error signaled if no majority result found

Johan Karlsson 39 Energy-aware computing

20

Error coverage – TTR-FR

(Triple Time Redundant execution w ith Forw ard Recovery)

Coverage Over‐ h d No Effect Corrected by Detected by Detected by HW Program H Total C g head y Software y Software y Exception Hang Coverage Low compiler

• ptimization

Manual C

317% 34.5% 15.2% 0.9% 45.6% 0.2% 96.4%

AspectC++Opt

440% 33.2% 17.1% 0.5% 45.3% 0.3% 96.5%

High compiler

• ptimization

Manual C

285% 34.2% 18.4% 1.3% 41.9% 0.1% 95.9%

AspectC++Opt

297% 32.6% 20.7% 1.7% 40.4% 0.2% 95.6%

Error model: Single bit -flips in CPU registers and volatile main memory

• No. of injected errors for each program: 10.000

Johan Karlsson 40 Energy-aware computing

Time redundancy and more (TRAM)

Purpose: error detection Purpose: error detection

• Six checking mechanisms

– Double time redundant execution and result comparison – Stack pointer and stack frame pointer integrity checks – Check that writes are made to correct data set – Counter-based control flow checking – Check for fake resets

Johan Karlsson 41 Energy-aware computing

21

Error coverage – TRAM

( Double time Redundant execution + 5 other mechanisms)

Coverage Over‐ h d No Effect Corrected by Detected by Detected by HW Program H Total C g head y Software y Software y Exception Hang Coverage Low compiler

• ptimization

Manual C

187% 33.3% 0% 21.5% 44.8% 0.3% 100%

AspectC++Opt

271% 29.6% 0% 22.9% 47.4% 0.1% 100%

High compiler

• ptimization

Manual C

181% 34,2% 0% 24.2% 40.4% 0.1% 100%

AspectC++Opt

204% 30.6% 0% 30.9% 38.4% 0.2% 100%

Error model: Single bit -flips in CPU registers and volatile main memory

• No. of injected errors for each program: 10.000

Johan Karlsson 42 Energy-aware computing

Conclusion

• Finding good trade-offs between energy consumption and dependability will be

an important challenge in the design of future computer systems!

Cost

Detected Undetected Error corrected Timing failure Bounded failure Value failure Fail silent Fail signal Error corrected Catastrophic failure Benign failure Safe Shutdow n

st balancing Software mechanisms System mechanisms

Johan Karlsson 43 Energy-aware computing

Dependability and Security

SW Design Faults HW Design Faults Physical Faults Error Corrected Error Error

Co Hardware mechanisms

22

Questions?

44 Johan Karlsson Energy-aware computing