When Software meets Hardware Faults Hao Han hhan@cs.wm.edu 7 - - PowerPoint PPT Presentation

▶
when software meets hardware faults
SMART_READER_LITE
LIVE PREVIEW

When Software meets Hardware Faults Hao Han hhan@cs.wm.edu 7 - - PowerPoint PPT Presentation

Oct 16, 2022 361 likes •579 views

CSCI654 Advanced Computer Architecture CSCI654 Advanced Computer Architecture CSCI654 Advanced Computer Architecture CSCI654 Advanced Computer Architecture When Software meets Hardware Faults Hao Han hhan@cs.wm.edu 7 April 2009 Some slides

slide-1
SLIDE 1

When Software meets Hardware Faults

Hao Han

hhan@cs.wm.edu 7 April 2009

CSCI654 Advanced Computer Architecture CSCI654 Advanced Computer Architecture CSCI654 Advanced Computer Architecture CSCI654 Advanced Computer Architecture

Some slides are adapted from talks of "SWAT"[ASPLOS'08], "SymPlFIED" [DSN'08], "Trace- based diagnosis"[DSN'08], and "Likely program invariants"[DSN'08]

slide-2
SLIDE 2

2

Outline

  • Motivation
  • Background
  • Research points

– Program verification: SymPLFIED – Error detection: SWAT

  • Experimental methodology (see report)
  • Limitations
  • Conclusion
slide-3
SLIDE 3

3

Motivation

  • Goal

Goal Goal Goal: highly reliable systems

  • Conventional illusion: fault-free hardware devices to

software ⇒ Can not only focus on software bugs of programs

  • Hardware faults will happen in the field

– Traditional solutions: (1) Hardware redundancy (2) special circuits to verify hardware

⇒ Too expensive: area, power, and so on Today: Re-think about the reliability problem when considering hardware faults, especially in the core

slide-4
SLIDE 4

4

Background - Location of H/W faults

Microarchitectural structure Microarchitectural structure Microarchitectural structure Microarchitectural structure Faults Faults Faults Faults

Instruction decoder Instruction decoder Instruction decoder Instruction decoder Decoding instruction is corrupted Integer ALU Integer ALU Integer ALU Integer ALU Output latch of one of the ALUs FP ALU FP ALU FP ALU FP ALU Output latch of one of the ALUs Address or data bus Address or data bus Address or data bus Address or data bus Bus of register, cache, memory Physical reg file Physical reg file Physical reg file Physical reg file Physical regs in the reg file Reorder buffer (ROB) Reorder buffer (ROB) Reorder buffer (ROB) Reorder buffer (ROB) Src/dest reg of instr in ROB entry Address gen unit (AGEN) Address gen unit (AGEN) Address gen unit (AGEN) Address gen unit (AGEN) Virtual address generated by the unit Register alias table (RAT) Register alias table (RAT) Register alias table (RAT) Register alias table (RAT) Logical -> phys map of a logical reg

slide-5
SLIDE 5

5

Background - Hardware Faults

  • Category of H/W faults:

(1) permanent (2) transient (3) intermittent

  • Impact of H/W faults
slide-6
SLIDE 6

6

Research Points

  • Program verification under hardware faults

SymPLFIED [DSN'08] (Best paper award)

  • Error detection for hardware faults with low

cost

SWAT [ASPLOS ’08] SWAT Trace-Based Fault Diagnosis [DSN'08] Likely Program Invariants [DSN'08] Accurate Fault Models [HPCA'09]

slide-7
SLIDE 7

7

SymPLFIED [DSN'08]

Goal: Goal: Goal: Goal: A formal framework to evaluate the effects of hardware faults on arbitrary programs independent of the detection mechanism

Conceptual Design Flow of SymPLFIED

slide-8
SLIDE 8

8

Techniques of SymPFLIED

  • Model error propagation by representingl errors

in program as abstract symbol

<symbolic execution> – Represents all kinds of faults

– Avoids explosion of exhaustive fault injection

  • Automatically search possible values of symoblic

error that escape from detecion and cause SDC

<model checking> – Bounded model checking using satisfiability solving

slide-9
SLIDE 9

9

SWAT System

  • Assumptions:

– Multicore system where a fault-free core is always available – Checkpoint/rollback mechanism

  • Goals:

– Provide low-cost software-level detection methods for permanent hardware fault, and low-level diagnosis for recovery and possibly repair/reconfiguration

  • SWAT components

– Detection: Symptoms of software for detecting – Diagnosis: Identify the source of faulty unit

slide-10
SLIDE 10

10

Fault Fault Fault Fault Error Error Error Error Symptom Symptom Symptom Symptom detected detected detected detected Recovery Recovery Recovery Recovery Diagnosis Diagnosis Diagnosis Diagnosis Repair Repair Repair Repair Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint

  • 1. Detectors w/ simple symtoms
  • 1. Detectors w/ simple symtoms
  • 1. Detectors w/ simple symtoms
  • 1. Detectors w/ simple symtoms [ASPLOS

[ASPLOS [ASPLOS [ASPLOS ’ ’ ’ ’08] 08] 08] 08]

  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis [DSN

[DSN [DSN [DSN ’ ’ ’ ’08] 08] 08] 08]

  • 4. Accurate Fault Models
  • 4. Accurate Fault Models
  • 4. Accurate Fault Models
  • 4. Accurate Fault Models [HPCA

[HPCA [HPCA [HPCA’ ’ ’ ’09] 09] 09] 09]

  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support [DSN

[DSN [DSN [DSN ’ ’ ’ ’08] 08] 08] 08]

slide-11
SLIDE 11

11

Simple Symptoms

  • Observe anomalous symptoms for fault detection

– Incur low overheads for “always-on” detectors – Minimal support from hardware, no software support

  • Anomalous symptoms

– Fatal hardware traps Fatal hardware traps Fatal hardware traps Fatal hardware traps

  • For example, division by zero, RED State, etc.

– Abnormal application exit Abnormal application exit Abnormal application exit Abnormal application exit, indicated by OS

  • For example, application terminates due to segmentation fault

– Hangs Hangs Hangs Hangs

  • The whole system becomes unresponsive
  • Detected by setting up counter

– High OS activity High OS activity High OS activity High OS activity

  • Monitoring the amount of time the execution remains in the OS,

without returning to the application

slide-12
SLIDE 12

12

Fault Fault Fault Fault Error Error Error Error Symptom Symptom Symptom Symptom detected detected detected detected Recovery Recovery Recovery Recovery Diagnosis Diagnosis Diagnosis Diagnosis Repair Repair Repair Repair Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint

  • 1. SWAT
  • 1. SWAT
  • 1. SWAT
  • 1. SWAT [ASPLOS

[ASPLOS [ASPLOS [ASPLOS ’ ’ ’ ’08] 08] 08] 08]

  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis [DSN

[DSN [DSN [DSN ’ ’ ’ ’08] 08] 08] 08]

  • 4. Accurate Fault Models
  • 4. Accurate Fault Models
  • 4. Accurate Fault Models
  • 4. Accurate Fault Models [HPCA

[HPCA [HPCA [HPCA’ ’ ’ ’09] 09] 09] 09]

  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support [DSN

[DSN [DSN [DSN ’ ’ ’ ’08] 08] 08] 08]

slide-13
SLIDE 13

13

Likely Program Invariant

Training Phase Training Phase Training Phase Training Phase

Application Application Application Application Compiler Pass in LLVM Compiler Pass in LLVM Compiler Pass in LLVM Compiler Pass in LLVM

  • - - - -
  • - - - -
  • - - - -
  • - - - -

Application Application Application Application

  • - - - -
  • - - - -
  • - - - -
  • - - - -

Range Range Range Range i/p #1 i/p #1 i/p #1 i/p #1 . . . . . . . . . . . . . . . . Range Range Range Range i/p #n i/p #n i/p #n i/p #n Invariant Ranges Invariant Ranges Invariant Ranges Invariant Ranges

Invariant Invariant Invariant Invariant Monitoring Monitoring Monitoring Monitoring Code Code Code Code Test, Test, Test, Test, train, train, train, train, external external external external inputs inputs inputs inputs MIN ≤ value ≤ MAX

slide-14
SLIDE 14

14

Likely Program Invariant

Training Phase Training Phase Training Phase Training Phase

Application Application Application Application Compiler Pass in LLVM Compiler Pass in LLVM Compiler Pass in LLVM Compiler Pass in LLVM

  • - - - -
  • - - - -
  • - - - -
  • - - - -

Application Application Application Application

  • - - - -
  • - - - -
  • - - - -
  • - - - -

Ranges Ranges Ranges Ranges i/p #1 i/p #1 i/p #1 i/p #1 . . . . . . . . . . . . . . . . Range Range Range Range s i/p #n s i/p #n s i/p #n s i/p #n Invariant Ranges Invariant Ranges Invariant Ranges Invariant Ranges

Invariant Invariant Invariant Invariant Monitoring Monitoring Monitoring Monitoring Code Code Code Code

Compiler Pass in LLVM Compiler Pass in LLVM Compiler Pass in LLVM Compiler Pass in LLVM

  • - - - -
  • - - - -
  • - - - -
  • - - - -

Application Application Application Application

  • - - - -
  • - - - -
  • - - - -
  • - - - -

Invariant Invariant Invariant Invariant Checking Checking Checking Checking Code Code Code Code Full System Full System Full System Full System Simulation Simulation Simulation Simulation Inject Inject Inject Inject Faults Faults Faults Faults SWAT Diagnosis SWAT Diagnosis SWAT Diagnosis SWAT Diagnosis Invariant Invariant Invariant Invariant Violation Violation Violation Violation False Positive False Positive False Positive False Positive (Disable Invariant) (Disable Invariant) (Disable Invariant) (Disable Invariant) Fault Fault Fault Fault Detection Detection Detection Detection

Fault Detection Phase Fault Detection Phase Fault Detection Phase Fault Detection Phase

Test, Test, Test, Test, train, train, train, train, external external external external inputs inputs inputs inputs Ref Ref Ref Ref input input input input

slide-15
SLIDE 15

15

Fault Fault Fault Fault Error Error Error Error Symptom Symptom Symptom Symptom detected detected detected detected Recovery Recovery Recovery Recovery Diagnosis Diagnosis Diagnosis Diagnosis Repair Repair Repair Repair Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint

  • 1. SWAT
  • 1. SWAT
  • 1. SWAT
  • 1. SWAT [ASPLOS

[ASPLOS [ASPLOS [ASPLOS ’ ’ ’ ’08] 08] 08] 08]

  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis
  • 3. Trace-Based Fault Diagnosis [DSN

[DSN [DSN [DSN ’ ’ ’ ’08] 08] 08] 08]

  • 4. Accurate Fault Models
  • 4. Accurate Fault Models
  • 4. Accurate Fault Models
  • 4. Accurate Fault Models [HPCA

[HPCA [HPCA [HPCA’ ’ ’ ’09] 09] 09] 09]

  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support
  • 2. Detectors w/ compiler support [DSN

[DSN [DSN [DSN ’ ’ ’ ’08] 08] 08] 08]

slide-16
SLIDE 16

16

Diagnosis: first step

16

No symptom No symptom No symptom No symptom Symptom Symptom Symptom Symptom

Permanent h/w bug or Permanent h/w bug or Permanent h/w bug or Permanent h/w bug or deterministic s/w bug or deterministic s/w bug or deterministic s/w bug or deterministic s/w bug or false positive false positive false positive false positive

Symptom Symptom Symptom Symptom detected detected detected detected Faulty Good Faulty Good Faulty Good Faulty Good Rollback on Rollback on Rollback on Rollback on faulty faulty faulty faulty core core core core Rollback/replay Rollback/replay Rollback/replay Rollback/replay

  • n
  • n
  • n
  • n good

good good good core core core core Continue Continue Continue Continue Execution Execution Execution Execution

Transient h/w bug Transient h/w bug Transient h/w bug Transient h/w bug or

  • r
  • r
  • r

non-deterministic s/w bug non-deterministic s/w bug non-deterministic s/w bug non-deterministic s/w bug

Symptom Symptom Symptom Symptom

Permanent Permanent Permanent Permanent h/w fault, h/w fault, h/w fault, h/w fault, needs repair! needs repair! needs repair! needs repair!

No symptom No symptom No symptom No symptom

False positive False positive False positive False positive or

  • r
  • r
  • r

deterministic s/w bug deterministic s/w bug deterministic s/w bug deterministic s/w bug (send to s/w layer) (send to s/w layer) (send to s/w layer) (send to s/w layer)

slide-17
SLIDE 17

17

Diagnosis: second step

Permanent Permanent Permanent Permanent fault fault fault fault

Microarchitecture-Level Microarchitecture-Level Microarchitecture-Level Microarchitecture-Level Granularity Diagnosis Granularity Diagnosis Granularity Diagnosis Granularity Diagnosis

Unit X is faulty Unit X is faulty Unit X is faulty Unit X is faulty

Symptom Symptom Symptom Symptom detected detected detected detected

Diagnosis Diagnosis Diagnosis Diagnosis

Software Software Software Software bug bug bug bug Transient Transient Transient Transient fault fault fault fault Goal: Goal: Goal: Goal: to efficiently diagnose the source (microarchitecture-level unit) of a permanent fault Advantages: Advantages: Advantages: Advantages: do not disable the entire core, only repair or disable/reconfigure the faulty µarch-level unit

slide-18
SLIDE 18

18

Trace-Based Fault Diagnosis (TBFD)

Permanent Permanent Permanent Permanent fault detected fault detected fault detected fault detected Rollback faulty- Rollback faulty- Rollback faulty- Rollback faulty- core to checkpoint core to checkpoint core to checkpoint core to checkpoint Load checkpoint Load checkpoint Load checkpoint Load checkpoint

  • n fault-free core
  • n fault-free core
  • n fault-free core
  • n fault-free core

Replay execution, Replay execution, Replay execution, Replay execution, collect collect collect collect µ µ µ µarch info arch info arch info arch info Fault-free Fault-free Fault-free Fault-free instruction exec instruction exec instruction exec instruction exec Faulty trace Faulty trace Faulty trace Faulty trace Test trace Test trace Test trace Test trace =? =? =? =? Invoke TBFD Invoke TBFD Invoke TBFD Invoke TBFD

Diagnosis Algorithm: Diagnosis Algorithm: Diagnosis Algorithm: Diagnosis Algorithm:

  • 1. Front-end
  • 1. Front-end
  • 1. Front-end
  • 1. Front-end
  • 2. Meta-datapath
  • 2. Meta-datapath
  • 2. Meta-datapath
  • 2. Meta-datapath
  • 3. Datapath
  • 3. Datapath
  • 3. Datapath
  • 3. Datapath

Mismatch!! Mismatch!! Mismatch!! Mismatch!!

  • Faults in front-end is

related to Instruction Decoder;

  • Fault in meta-datapath

indicates faults in ROB

  • r RAT;
  • Faults in datapath is

related to ALU, data bus, and register file.

slide-19
SLIDE 19

19

Limitations

  • Do not consider the off-core faults, such as faults in

crossbar

  • Most work only considers single error for simplicity, but

in practice hardware faults can be multi-types and multi- sources

  • Pure software level detection has inherent shortcomings,

hybrid method (combining hardware and software) may be a better choice

  • SWAT is passive scheme, need more aggressive

detection method ...

slide-20
SLIDE 20

20

Conclusion

  • Verifying program and detecting hardware

faults are vital for reliable system

  • For SymPLFIED

Verify programs automatically with symbolic execution and model checking

  • For SWAT

High-level detection, low-level diagnosis Treats hardware faults as software bugs Handles all faults that matter, and oblivious to masked faults