An Empirical Study on Configuration Errors in Commercial and Open - PowerPoint PPT Presentation

An Empirical Study on Configuration Errors in Commercial and Open Source Systems Zuoning Yin, Xiao ma, Lakshmi N. Bairavasundaram, Jing Zheng, Yuanyuan Zhou Shankar Pasupathy University of California Netapp Inc. At San Diego

Configuring computers is not easy

Configuring server systems is much harder

Root Causes of Customer Reported Issues User Knowledge 9% Customer Environment 25% Configuration 31% Bug 15% Hardware Failure 20%

Who should take responsibility for configuration errors, users or developers?

• Developers do not think carefully when they design configuration interface • Once an issues turns out to be a configuration error, developers move on

How to Reduce Configuration Errors? • What kind of configuration errors do users make? • Which types of configuration parameters are more error-prone? • Which user actions may cause configuration errors? • ...... We need to understand real-world configuration errors first

Objectives and Challenges • Objectives • Understand the characteristics of real-world configuration errors • Reveal their implications to developers • Challenges • Configuration errors are not recorded rigorously • Configuration errors are difficult to understand

Methodology • Random sampling configuration errors • Choose resolved cases in recent 2 years • Ensure the sample size is big enough • Calculate statistical error • Categorizing errors with best effort • Cross-validation among co-authors • Help from developers

Data Source Number of System ystem Sampled Errors Commercial COMP-A 309 CentOS 60 MySQL 55 Open Source Open Source Apache 60 Open LDAP 62 Total Total 546

Limitation • We study only user-reported errors • Configuration errors may be resolved in other means • We focus on server-side systems • Other types of systems may have different characteristics

Type Reaction Impact Cause

Type Reaction Impact Cause

Type Impact Reaction Cause Software Configuration Errors Parameter Compatibility Other Errors Errors Errors

Type Impact Reaction Cause Parameter Errors Compatibility Errors Other Errors COMP-A 79.6% 10% 10.4% CentOS 70.0% 18.3% 11.7% MySQL 85.5% 14.5% Apache 83.4% 8.3% 8.3% Open LDAP 79.0% 11.2% 9.7% Parameter errors dominate

Type Impact Reaction Cause Parameter Errors Compatibility Errors Other Errors COMP-A 79.6% 10% 10.4% • Systems should expose as few CentOS 70.0% 18.3% 11.7% configuration parameters as possible MySQL 85.5% 14.5% • Automatic configuration is preferred Apache 83.4% 8.3% 8.3% Open LDAP 79.0% 11.2% 9.7% Parameter errors dominate

Type Impact Reaction Cause Parameter Errors Compatibility Errors Other Errors COMP-A 10% 10.4% CentOS 18.3% 11.7% MySQL 14.5% Apache 8.3% 8.3% Open LDAP 11.2% 9.7% Other types of configuration errors are not negligible

Type Impact Reaction Cause Parameter Errors COMP-A MySQL InitiatorName: iqn_DEV_domain AutoCommit = True Lower-case Error! “True” value may affect only value performance

Type Impact Reaction Cause Parameter Errors Illegal Parameters Legal Parameters COMP-A MySQL InitiatorName: iqn_DEV_domain AutoCommit = True Lower-case Error! “True” value may affect only value performance

Type Impact Reaction Cause Illegal Parameters Legal Parameters COMP-A 53.7% 46.3% CentOS 38.1% 61.9% MySQL 48.9% 51.1% Apache 46.0% 54.0% Open LDAP 53.1% 46.9% Illegal and legal parameters have similar contribution to parameter errors

Type Impact Reaction Cause Illegal Parameters Legal Parameters COMP-A 53.7% 46.3% • Illegal parameters are relatively easy to CentOS 38.1% 61.9% detect MySQL 48.9% 51.1% • About half of parameter errors involve illegal parameters (“good” news!) Apache 46.0% 54.0% Open LDAP 53.1% 46.9% Illegal and legal parameters have similar contribution to parameter errors

Type Impact Reaction Cause Illegal Parameters Value Errors Format Errors Inconsistent Other Values Errors Value Errors

Type Impact Reaction Cause Inconsistent Value Errors Format Errors Other Value Errors COMP-A 80% CentOS 75% MySQL 65% Apache 52% Open LDAP 27% Inconsistent values dominate illegal parameter errors for most systems

Type Impact Reaction Cause Inconsistent Value Errors Format Errors Other Value Errors COMP-A 11% CentOS 6% MySQL 4% Apache 26% Open LDAP 69% All systems have format errors, in particular, Open LDAP has 69%

Type Impact Reaction Cause Inconsistent Value Errors A c r o s M s u l t i p l e S y s t e m s ! PHP + MySQL MySQL PHP configuration: log_output=”Table” ... mysql.max_persistent = 400 log=query.log MySQL configuration: max_connections = 300 Not consistent. They should be: log_output=”Table” The value in PHP should or not be bigger than the log_output=”File” value in MySQL log=query.log

Type Impact Reaction Cause Inconsistent Value Errors A c r o s M s u l t i p l e S y • Value consistency constraints are error- s t e m s ! prone; they account for most illegal PHP + MySQL MySQL parameter errors PHP configuration: log_output=”Table” ... mysql.max_persistent = 400 • Consistency constraints could be across log=query.log multiple systems, which is more MySQL configuration: max_connections = 300 difficult for users to follow Not consistent. They should be: log_output=”Table” The value in PHP should or not be bigger than the log_output=”File” value in MySQL log=query.log

Type Impact Reaction Cause Format Errors COMP-A InitiatorName: iqn_DEV_domain Lower-case Error only value OpenLDAP Apache include schema/ppolicy.schema extension = mysql.so ...... ...... overlay ppolicy extension = recode.so recode.so must be Missing put before mysql.so

Type Impact Reaction Cause Format Errors • Format constraints are difficult to COMP-A follow, especially non-intuitive ones, InitiatorName: iqn_DEV_domain e.g., upper-case vs. lower-case or ordering Lower-case Error only value • Format errors are relatively easier to OpenLDAP Apache detect compared to value errors include schema/ppolicy.schema extension = mysql.so ...... ...... overlay ppolicy extension = recode.so recode.so must be Missing put before mysql.so

Type Reaction Impact Cause

Type Impact Reaction Cause System Reaction Pinpoint Indeterminate Quiet Failure Reaction Reaction

Type Impact Reaction Good Reaction #1 Cause • Symptom : the user cannot create new directories in “ /vol/vol1/data/ ” • Reaction : the system prints this message: [COMP-A – dir.size.max:warning]: Directory /vol/vol1/data/ reached the maxdirsize Limit. Reduce the number of files or use the vol options command to increase this limit.

Type Impact Reaction Good Reaction #2 Cause MySQL log_output=”Table” ... log=query.log Patch: +if (opt_logname + && !(log_output_options & LOG_FILE) + && !(log_output_options & LOG_NONE)) + sql_print_warning("Although a path was specified + for the --log option, log tables are used. To enable + logging to files use the --log-output option.");

Type Impact Reaction Good Reaction #2 Cause MySQL MySQL log_output=”Table” ... log=query.log Patch: +if (opt_logname + && !(log_output_options & LOG_FILE) + && !(log_output_options & LOG_NONE)) + sql_print_warning("Although a path was specified + for the --log option, log tables are used. To enable + logging to files use the --log-output option.");

Type Impact Reaction Cause Pinpoint Reaction Indeterminate Reaction Quiet Failure Unknown COMP-A 15.5% 49.5% 23.9% 11.0% CentOS 11.7% 54.9% 26.7% 6.7% MySQL 7.2% 47.3% 23.6% 21.8% Apache 13.3% 46.7% 26.7% 13.3% Open LDAP 14.5% 45.2% 22.6% 17.7%

Type Impact Reaction Cause Pinpoint Reaction Indeterminate Reaction Quiet Failure Unknown COMP-A 15.5% CentOS 11.7% MySQL 7.2% Apache 13.3% Open LDAP 14.5% Today’s systems do not react to configuration errors in a user-friendly way

Type Impact Reaction Cause Pinpoint Reaction Indeterminate Reaction Quiet Failure Unknown COMP-A 23.9% CentOS 26.7% MySQL 23.6% Apache 26.7% Open LDAP 22.6% Big portion of quiet failures makes diagnosis difficult

Type Impact Reaction Cause Pinpoint Reaction Indeterminate Reaction Quiet Failure Unknown • Systems should avoid “bug-like” COMP-A 23.9% symptoms when configuration errors CentOS 26.7% happen, such as quite failures, crash or hang MySQL 23.6% Apache 26.7% Open LDAP 22.6% Big portion of quiet failures makes diagnosis difficult

Type Impact Reaction Cause Do systems react better to errors with “illegal” parameters?

Type Impact Reaction Cause Overall Illegal Parameter 30.0% Percentage of Pinpoint Reaction 22.5% 15.0% 7.5% COMP-A CentOS MySQL Apache OpenLDAP