Trust in Computers How Reliable are they really? 1 Some Newspaper - PDF document

Trust in Computers How Reliable are they really? 1 Some Newspaper Headlines • Data Entry Typo Mutes Millions of US. Pagers • Software Errors Cause Radiation Overdose • IRS Computer Sends Bill For $68 Billion in Penalties • Robot Kills Worker • California Junks $100 Million Child Support System • Man Arrested Five Times Due to Faulty FBI Computer Data • High-Tech Baggage System ‘Eats’ Luggage • Computer Predicts We Will Run Out of Copper by 1985 2 Computer Errors � Errors can occur in almost every computer system � Errors may have a minor effect or a major one � How prevalent are errors? � How about serious errors? � Should the probability of errors prevent us from applying computer systems to certain applications? 3

Facts About Computer Errors � Error-free software is not possible � Errors are often caused by more than one factor � Errors can be reduced by following good procedures and professional practices Q : How do we distinguish between tolerable or unavoidable errors in software versus careless software development? 4 Causes of Errors � Faulty design � Sloppy Implementation � Insufficiently trained users � Poor user interfaces 5 Implications of Errors � Implications for Computer Users � Implications for Computer Professionals � Implications for Society 6

Some Types of Errors (1) � Errors Affecting Individuals: � Billing Errors � Database Accuracy Problems � Contributing Factors: � Similar names � Automated processing � Overconfidence in computerized data � Data entry errors � Outdated information � Lack of accountability 7 Some Types of Errors (2) � System Failures: � Network Failures (phones, computer networks, …etc.) � Business System Failures (financial systems, reservations systems, …etc.) � Abandoned Systems � Safety-Critical Applications: � Aviation Control � Medical Systems 8 Case Study: The Therac-25 � The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer. � Overdoses of radiation • Normal dosage is 100–200 rads. • It is estimated that 13,000 and 25,000 rads were given to six people. • Three of the six people died. Q : What determines whether the risks associated with a safety-critical application are acceptable? 9

Case Study: The Therac-25 � Therac-25 Radiation Overdose � Multiple Causes: • Poor safety design. • Insufficient testing and debugging. • Software errors. • Lack of safety interlocks. • Overconfidence. • Inadequate reporting and investigation of accidents. 10 10 Factors in System Failures (1) � Interaction with physical devices that do not work as expected � Incompatibility of software and hardware, or of application software and the operating system � Management problems, including business and/or political pressure to get a product out quickly � Inadequate attention to potential safety risks � Not planning and designing for unexpected inputs or circumstances � Specification modification midway through the development phase � Insufficient testing 11 11 Factors in System Failures (2) � Reuse of software from another system without adequate checking. � Overconfidence in software � Carelessness. � Misrepresentation; hiding problems; inadequate response when problems are reported. � Problems with management of the use of a system: � Data-entry errors. � Inadequate training of users. � Errors in interpreting results or output. � Overconfidence in software by users. � Insufficient planning for failures; no backup systems or procedures. � Lack of market or legal incentives to do a better job. 12 12

Good Practices � Study and follow established professional techniques � Software developers should learn enough about the target domain to be able to understand potential pitfalls � Human factors should be incorporated in user interfaces � Redundancy in safety critical systems � Self testing of systems � Testing, testing, testing 13 13 Laws and Regulations � Do we need computer liability laws? � Warranties: � Individual responsibility � Legal status � Regulation of safety critical applications? � Regulations for database accuracy? � Licensing of software developers: � Mandatory � Voluntary 14 14 Errors in Perspective � What percentage of computing is erroneous? � What percentage is acceptable? � What if the automated systems produces less errors than the manual systems they replace? � What if the cost to identify and fix the errors is smaller than the cost of preventing them? � The balance between risk and progress � Are computer errors used as a scapegoat? 15 15

Computer Modeling � How much faith should we place on predictions by computer models? � Assumptions? � Testing? 16 16 Points To Consider � Developers have incomplete knowledge of the system being modeled � Data might be incomplete or inaccurate � Power of the computer might be inadequate � Variables are difficult to numerically quantify � Political and economic motivation to distort results 17 17 Why Models Might Not Be Accurate � Developers have incomplete knowledge of the system being modeled � Data might be incomplete or inaccurate � Power of the computer might be inadequate � Variables are difficult to numerically quantify � Political and economic motivation to distort results 18 18

News Item In June 2005, United Airlines scrapped the complex, trouble- plagued automated baggage system described in the text. The system had cost hundreds of millions of dollars. United will move baggage manually. 19 19 News Item ChoicePoint, a major data broker, agreed to pay $15 million for inadvertently allowing a fraud ring posing as a legitimate business customer to buy personal data, including credit histories, for 163,000 people, according to the FTC. Approximately 800 people became victims of identity theft as a result. After the incident, which was made public in 2005, other data brokers announced similar security breaches. Roughly half the states require that companies notify consumers of breaches of their personal information. (Jan. 27, 2006) 20 20 News Item According to the Transportation Security Administration, more than 30,000 innocent people have been delayed at airports because their names matched names on the federal "watch list" (Jan. 27, 2006) 21 21