Applying Computational Learning Theory to Software Testing Neil - - PowerPoint PPT Presentation

Applying Computational Learning Theory to Software Testing

Neil Walkinshaw

Computational Learning Theory (CLT)

• Theoretical analysis of questions of learnability.
• Is it feasible to learn a given type of concept from a set
• f examples?
• What are the performance bounds? Time complexity?
• Several theoretical learning frameworks to

characterise a learning problem.

• Identification In the Limit (IIL)
• Version Spaces
• Probably Approximately Correct (PAC) learning
• Vapnik-Chervonenkis Theory
• …

Valiant, Leslie G. "A theory of the learnable." Communications of the ACM (1984) Vapnik, Vladimir Naumovich, and Vlamimir Vapnik. Statistical learning theory, Wiley, 1998. Gold, E. Mark. "Language identification in the limit." Information and control 10.5 (1967) Mitchell, Tom M. "Generalization as search." Artificial intelligence(1982)

Machine Learning Insights from CLT

• Numerous positive / negative learnability results.
• Finite languages, supra-finite languages from positive

data, regular grammars with a helpful teacher, conjunctive concepts (in PAC), disjunctions of conjunctive concepts (in PAC), …

• The number of required examples can be

bounded explicitly…

• … in PAC if we can bound the size of the

Version Space or…

• … in PAC if we can bound the size of the

Vapnik Chervonenkis dimension…

• Derived by close examination of relationship

between learning algorithm and subject system.

Gold, E. Mark. "Language identification in the limit." Information and control 10.5 (1967): 447-474. Angluin, Dana. "Queries and concept learning." Machine learning 2.4 (1988): 319-342. Valiant, Leslie G. "A theory of the learnable." Communications of the ACM 27.11 (1984): 1134-1142. Haussler, David. "Quantifying inductive bias: AI learning algorithms and Valiant's learning framework." Artificial intelligence 36.2 (1988): 177-221. Blumer, Anselm, et al. "Learnability and the Vapnik-Chervonenkis dimension." Journal of the ACM (JACM) 36.4 (1989): 929-965.

Applications to Testing?

• Several theoretical frameworks that link testing and ML
• Learnability ➔ Testability
• Bounds on training set size ➔ Bounds on test set

size.

• Current results can rarely simply be carried over.
• Software representations are necessarily complex.
• Rarely valid to represent software as (e.g.) a FSM.
• CLT results assume that learning is `distribution free’ -

i.e. apply to any distribution over the training sample

• Reasoning wrt. distributions of test inputs is

notoriously problematic.

Walkinshaw, "Assessing test adequacy for black-box systems without specifications." ICTSS (2011) Weyuker "Assessing test data adequacy through program inference." ACM TOPLAS (1983) Budd, and Angluin. "Two notions of correctness and their relation to testing." Acta Informatica 18.1 (1982): 31-45. Romanik, "Approximate testing and its relationship to learning." Theoretical Computer Science 188.1 (1997): 79-99. Romanik and Vitter (1996). Using Vapnik–Chervonenkis Dimension to Analyze the Testing Complexity of Program Segments. Information and Computation, 128(2), 87-108. Hamlet, R. (1994). Random testing. Encyclopedia of software Engineering.

Goal: Improved theoretical frameworks to reason about learnability (and testability) of generic software systems