O and V , to yield h We cannot report L V ( h ) as the measure of - PowerPoint PPT Presentation

Training, Validation, Testing Testing • A machine learning system has been trained, using both T O and V , to yield ˆ h • We cannot report L V (ˆ h ) as the measure of performance • The set V is tainted since we used it during training • Performance measures are accepted only on pristine sets, not used in any way for training • We need to test the system on a third set S , the test set • Estimate the true risk L p (ˆ h ) = E p [ ` ( y , ˆ h ( x ))] by computing the empirical risk L S (ˆ P | S | n = 1 ` ( y n , ˆ 1 h ) = h ( x n )) on S | S | COMPSCI 527 — Computer Vision Basics of Machine Learning 16 / 21

Training, Validation, Testing Summary of Sets Involved • A training set T to train the predictor given a specific set of hyper-parameters (if any) • A validation set V to choose good hyper-parameters, or for deciding termination • A test set S to evaluate the generalization performance of the predictor ˆ h learned by training on T and validating on V • Resampling techniques (“cross-validation”) exist for making the same set play the role of both T and V • S must still be entirely separate COMPSCI 527 — Computer Vision Basics of Machine Learning 17 / 21

The State of the Art of Image Classification The State of the Art of Image Classification • ImageNet Large Scale Visual Recognition Challenge (ILSVRC) • Based on ImageNet,1.4 million images, 1000 categories (Fei-Fei Li, Stanford) • Three different competitions: • Classification : • One label per image, 1.2M images available for training, 50k for validation, 100k withheld for testing • Zero-one loss for performance evaluation • Localization : Classification, plus bounding box. Correct if ≥ 50% overlap with true box • Detection : Same as localization, but find every instance in the image. Measure the fraction of mistakes (false positives, false negatives) COMPSCI 527 — Computer Vision Basics of Machine Learning 18 / 21

The State of the Art of Image Classification [Image from Russakovsky et al. , ImageNet Large Scale Visual Recognition Challenge, Int’l. J. Comp. Vision 115:211-252, 2015] COMPSCI 527 — Computer Vision Basics of Machine Learning 19 / 21

The State of the Art of Image Classification Difficulties of ILSVRC • Images are “natural.” Arbitrary backgrounds, different sizes, viewpoints, lighting. Partially visible objects • 1,000 categories, subtle distinctions. Example: Siberian husky and Eskimo dog • Variations of appearance within one category can be significant (how many lamps can you think of?) • What is the label of one image? For instance, a picture of a group of people examining a fishing rod was labeled as “reel.” COMPSCI 527 — Computer Vision Basics of Machine Learning 20 / 21

The State of the Art of Image Classification Performance for Image Classification 1 3.7 • 2010: 28.2 percent t • 2017: 2.3 percent (ensemble of several deep networks) • Improvement results from both architectural insights (residuals, squeeze-and-excitation networks, ...) and persistent engineering • A book on “tricks of the trade in deep learning!” • We will see some after studying the basics COMPSCI 527 — Computer Vision Basics of Machine Learning 21 / 21