Trust in Computers How Reliable are they really? 1 Some Newspaper - - PDF document

1

Trust in Computers

How Reliable are they really?

2

Some Newspaper Headlines

• Data Entry Typo Mutes Millions of US. Pagers
• Software Errors Cause Radiation Overdose
• IRS Computer Sends Bill For $68 Billion in Penalties
• Robot Kills Worker
• California Junks $100 Million Child Support System
• Man Arrested Five Times Due to Faulty FBI Computer Data
• High-Tech Baggage System ‘Eats’ Luggage
• Computer Predicts We Will Run Out of Copper by 1985

3

Computer Errors

Errors can occur in almost every computer

system

Errors may have a minor effect or a major

• ne

How prevalent are errors? How about serious errors? Should the probability of errors prevent us

from applying computer systems to certain applications?

4

Facts About Computer Errors

Error-free software is not possible Errors are often caused by more than one

factor

Errors can be reduced by following good

procedures and professional practices

Q: How do we distinguish between tolerable or unavoidable errors in

software versus careless software development?

5

Causes of Errors

Faulty design Sloppy Implementation Insufficiently trained users Poor user interfaces

6

Implications of Errors

Implications for Computer Users Implications for Computer Professionals Implications for Society

7

Some Types of Errors (1)

Errors Affecting Individuals: Billing Errors Database Accuracy Problems Contributing Factors: Similar names Automated processing Overconfidence in computerized data Data entry errors Outdated information Lack of accountability

8

Some Types of Errors (2)

System Failures: Network Failures (phones, computer

networks, …etc.)

Business System Failures (financial

systems, reservations systems, …etc.)

Abandoned Systems Safety-Critical Applications: Aviation Control Medical Systems

9

Case Study: The Therac-25

The Therac-25 was a software-controlled

radiation-therapy machine used to treat people with cancer.

Overdoses of radiation

• Normal dosage is 100–200 rads.
• It is estimated that 13,000 and 25,000 rads were

given to six people.

• Three of the six people died.

Q: What determines whether the risks associated with a safety-critical

application are acceptable?

10 10

Case Study: The Therac-25

Therac-25 Radiation Overdose Multiple Causes:

• Poor safety design.
• Insufficient testing and debugging.
• Software errors.
• Lack of safety interlocks.
• Overconfidence.
• Inadequate reporting and investigation of

accidents.

11 11

Factors in System Failures (1)

Interaction with physical devices that do not work

as expected

Incompatibility of software and hardware, or of

application software and the operating system

Management problems, including business and/or

political pressure to get a product out quickly

Inadequate attention to potential safety risks Not planning and designing for unexpected inputs

• r circumstances

Specification modification midway through the

development phase

Insufficient testing

12 12

Factors in System Failures (2)

Reuse of software from another system without adequate

checking.

Overconfidence in software Carelessness. Misrepresentation; hiding problems; inadequate response

when problems are reported.

Problems with management of the use of a system: Data-entry errors. Inadequate training of users. Errors in interpreting results or output. Overconfidence in software by users. Insufficient planning for failures; no backup systems or

procedures.

Lack of market or legal incentives to do a better job.

13 13

Good Practices

Study and follow established professional

techniques

Software developers should learn enough

about the target domain to be able to understand potential pitfalls

Human factors should be incorporated in

user interfaces

Redundancy in safety critical systems Self testing of systems Testing, testing, testing

14 14

Laws and Regulations

Do we need computer liability laws? Warranties: Individual responsibility Legal status Regulation of safety critical applications? Regulations for database accuracy? Licensing of software developers: Mandatory Voluntary

15 15

Errors in Perspective

What percentage of computing is erroneous? What percentage is acceptable? What if the automated systems produces less

errors than the manual systems they replace?

What if the cost to identify and fix the errors is

smaller than the cost of preventing them?

The balance between risk and progress Are computer errors used as a scapegoat?

16 16

Computer Modeling

How much faith should we place on

predictions by computer models?

Assumptions? Testing?

17 17

Points To Consider

Developers have incomplete knowledge

• f the system being modeled

Data might be incomplete or inaccurate Power of the computer might be

inadequate

Variables are difficult to numerically

quantify

Political and economic motivation to

distort results

18 18

Why Models Might Not Be Accurate

Developers have incomplete knowledge

• f the system being modeled

Data might be incomplete or inaccurate Power of the computer might be

inadequate

Variables are difficult to numerically

quantify

Political and economic motivation to

distort results

19 19

News Item

In June 2005, United Airlines scrapped the complex, trouble- plagued automated baggage system described in the text. The system had cost hundreds of millions of dollars. United will move baggage manually.

20 20

News Item

ChoicePoint, a major data broker, agreed to pay $15 million for inadvertently allowing a fraud ring posing as a legitimate business customer to buy personal data, including credit histories, for 163,000 people, according to the FTC. Approximately 800 people became victims

• f identity theft as a result. After the incident, which

was made public in 2005, other data brokers announced similar security breaches. Roughly half the states require that companies notify consumers of breaches of their personal information. (Jan. 27, 2006)

21 21

News Item

According to the Transportation Security Administration, more than 30,000 innocent people have been delayed at airports because their names matched names on the federal "watch list" (Jan. 27, 2006)