Life-long Learning with Applications in Monitoring Biodiversity - - PowerPoint PPT Presentation
Life-long Learning with Applications in Monitoring Biodiversity Joachim Denzler Computer Vision Group Michael Stifel Center Jena http://www.inf-cv.uni-jena.de/ http://www.mscj.uni-jena.de/ Friedrich Schiller University Jena Computer Vision
Joachim Denzler
Computer Vision Group Michael Stifel Center Jena http://www.inf-cv.uni-jena.de/ http://www.mscj.uni-jena.de/
Friedrich Schiller University Jena Computer Vision Group
Life-Long Learning 1
Friedrich Schiller University Jena Computer Vision Group
1 Motivation 2 WALI: Watch, Ask, Learn, and Improve 3 First Results for Biodiversity Research 4 Conclusion
Life-Long Learning 2
Friedrich Schiller University Jena Computer Vision Group
Motivation
Life-Long Learning 3
Friedrich Schiller University Jena Computer Vision Group
Motivation
from Wolfgang W¨ agele: Technical Concept for AMMOD Automated Multi-Sensor Station for Monitoring of Species Diversity (part of BioM-D)
Life-Long Learning 4
Friedrich Schiller University Jena Computer Vision Group
Motivation
Life-Long Learning 5
Friedrich Schiller University Jena Computer Vision Group
Motivation
Life-Long Learning 6
Friedrich Schiller University Jena Computer Vision Group
Motivation
Life-Long Learning 7
Friedrich Schiller University Jena Computer Vision Group
Motivation
Life-Long Learning 8
Friedrich Schiller University Jena Computer Vision Group
Motivation
Life-Long Learning 9
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings Classification of expected species
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings Classification of expected species Activity detection (eating, sleeping, etc.)
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings Classification of expected species Activity detection (eating, sleeping, etc.) High accuracy, small error rate
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings Classification of expected species Activity detection (eating, sleeping, etc.) High accuracy, small error rate Open to changing setups, new species to be detected, etc.
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings Classification of expected species Activity detection (eating, sleeping, etc.) High accuracy, small error rate Open to changing setups, new species to be detected, etc. Possibility of human to check, correct, and understand the automatic results (keep the human in the loop)
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
Motivation
Monitoring does not only mean to record data, but also to evaluate it: Million of images per day possible Presorting of errorneous or useless recordings Classification of expected species Activity detection (eating, sleeping, etc.) High accuracy, small error rate Open to changing setups, new species to be detected, etc. Possibility of human to check, correct, and understand the automatic results (keep the human in the loop)
Life-Long Learning 10
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
1 Motivation 2 WALI: Watch, Ask, Learn, and Improve 3 First Results for Biodiversity Research 4 Conclusion
Life-Long Learning 11
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Christoph K¨ ading, Erik Rodner, Alexander Freytag, Joachim Denzler: Watch, Ask, Learn, and Improve: a lifelong learning cycle for visual recognition. European Symposium on Artificial Neural Networks (ESANN). 2016
Life-Long Learning 12
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Key incredients: Multi-class active learning
K¨ ading et al.. Active Learning and Discovery of Object Categories in the Presence of Unnameable Instances. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4343-4352. 2015. Freytag et al.. Selecting Influential Examples: Active Learning with Expected Model Output Changes European Conference on Computer Vision (ECCV). 562-577. 2014.
Novelty detection
Bodesheim et al.. Local Novelty Detection in Multi-class Recognition Problems IEEE Winter Conference on Applications of Computer Vision (WACV). 813-820. 2015. Bodesheim et al.. Kernel Null Space Methods for Novelty Detection IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3374-3381. 2013.
Large-scale learning
Fr¨
Segmentation and Facade Recognition Machine Vision and Applications. 24(5):1043-1053 2013. Rodner et al.. Large-Scale Gaussian Process Classification with Flexible Adaptive Histogram Kernels European Conference on Computer Vision (ECCV). 85-98. 2012.
Life-Long Learning 13
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Idea: Expected model output change (EMOC), i.e. ask for a label for that sample that maximizes the change of the model output after knowing the label
Life-Long Learning 14
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Idea: Expected model output change (EMOC), i.e. ask for a label for that sample that maximizes the change of the model output after knowing the label △f (x′) =
p(y′|f (x′)) 1 |D|
L(f (xj), f ′(xj)) We have sample set of labeled and unlabeled samples D, classifier output f (xj) given by Gaussian process classifier
Life-Long Learning 14
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Idea: Expected model output change (EMOC), i.e. ask for a label for that sample that maximizes the change of the model output after knowing the label △f (x′) =
p(y′|f (x′)) 1 |D|
L(f (xj), f ′(xj)) We have sample set of labeled and unlabeled samples D, classifier output f (xj) given by Gaussian process classifier We need an appropriate loss function: L1-loss estimate for the label probability of the given sample: given by the predictive posterior of the Gaussian process classifier used for MC sampling of the labels efficient model update: possible in the Gaussian process framework
Life-Long Learning 14
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Evaluation: number of discovered classes over time, error on the detected classes, error for all classes
Life-Long Learning 15
Friedrich Schiller University Jena Computer Vision Group
WALI: Watch, Ask, Learn, and Improve
Evaluation: Long term behaviour
Life-Long Learning 16
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
1 Motivation 2 WALI: Watch, Ask, Learn, and Improve 3 First Results for Biodiversity Research 4 Conclusion
Life-Long Learning 17
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Data: Initially annotated (large) data set of species to be detected (standard in DNA-barcoding)
Life-Long Learning 18
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Data: Initially annotated (large) data set of species to be detected (standard in DNA-barcoding) No handcrafting: Generic classifiers that can be learned without manual feature design and that will work in large scale scenarios (40.000 species) (Demo)
Life-Long Learning 18
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Data: Initially annotated (large) data set of species to be detected (standard in DNA-barcoding) No handcrafting: Generic classifiers that can be learned without manual feature design and that will work in large scale scenarios (40.000 species) (Demo) Where to look at: Automatic decision of where to look at for species identification in fine-grained setups (Demo)
Life-Long Learning 18
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Data: Initially annotated (large) data set of species to be detected (standard in DNA-barcoding) No handcrafting: Generic classifiers that can be learned without manual feature design and that will work in large scale scenarios (40.000 species) (Demo) Where to look at: Automatic decision of where to look at for species identification in fine-grained setups (Demo) “Oops“: Detection of unexpected or new species violating closed world assumption followed by incrementally updating the knowledge of the system (Demo)
Life-Long Learning 18
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Data: Initially annotated (large) data set of species to be detected (standard in DNA-barcoding) No handcrafting: Generic classifiers that can be learned without manual feature design and that will work in large scale scenarios (40.000 species) (Demo) Where to look at: Automatic decision of where to look at for species identification in fine-grained setups (Demo) “Oops“: Detection of unexpected or new species violating closed world assumption followed by incrementally updating the knowledge of the system (Demo) Ask for help: Feedback of humans to decide complicated cases and to support in annotating (a few, very) important images (Demo)
Life-Long Learning 18
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
CUB200 dataset: 200 bird species, aprox. 12000 images
Marcel Simon and Erik Rodner. Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks. International Conference on Computer Vision (ICCV). 2015.
Life-Long Learning 19
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 20
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Erik Rodner, Marcel Simon, Gunnar Brehm, Stephanie Pietsch, J. Wolfgang W¨ agele, Joachim
Fine-grained Visual Classification (CVPR-WS). 2015.
Life-Long Learning 21
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 22
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 23
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Possible use: support species identification (complement field guides) Alexander Freytag, Alena Schadt, Joachim Denzler. Interactive Image Retrieval for Biodiversity Research German Conference on Pattern Recognition (GCPR). 129-141. 2015.
Life-Long Learning 24
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Joint work with Hjalmar K¨ uhl (Max Planck Institute for Evolutionary Anthropology), Tilo Burkhardt (U. Bristol)
Freytag et al.: Chimpanzee Faces in the Wild: Log-Euclidean CNNs for Predicting Identities and Attributes of Primates. German Conference on Pattern Recognition (GCPR). 2016.
Life-Long Learning 25
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Joint work with Andrea Perino (thanks for the slide!)
Life-Long Learning 26
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Christoph K¨ ading, Alexander Freytag, Erik Rodner, Andrea Perino, Joachim Denzler: Large-scale Active Learning with Approximations of Expected Model Output Changes. German Conference on Pattern Recognition (GCPR). 2016.
Life-Long Learning 27
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 28
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 29
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 30
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 31
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 32
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Christoph K¨ ading, Alexander Freytag, Erik Rodner, Paul Bodesheim, Joachim Denzler. Active Learning and Discovery of Object Categories in the Presence of Unnameable Instances IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4343-4352. 2015.
Life-Long Learning 33
Friedrich Schiller University Jena Computer Vision Group
First Results for Biodiversity Research
Life-Long Learning 34
Friedrich Schiller University Jena Computer Vision Group
Conclusion
+ No handcrafting: Generic classifiers that can be learned and will work in large scale scenarios (most recent results: identification of great apes in the wild, joint work with Hjalmar K¨ uhl, MPI Leipzig) +/- Where to look at: Automatic decision of where to look at for species identification in fine-grained setups
assumption followed by incrementally updating the knowledge of the system +/- Ask for help: Feedback of humans to decide complicated cases and to support in annotating (a few, very) important images
Life-Long Learning 35
Friedrich Schiller University Jena Computer Vision Group
Conclusion
For automatic, visual monitoring we need
1 annotated datasets 2 generic methods that learn with weak supervision 3 tools for keeping the human in the loop 4 you (the ecologist) for providing us with interesting data, challenging research questions,
and feedback
Life-Long Learning 36
Friedrich Schiller University Jena Computer Vision Group
Conclusion
Life-Long Learning 37
Friedrich Schiller University Jena Computer Vision Group
Conclusion
For more information (details, datasets, papers, videos) please visit
1 Computer Vision Group: http://www.inf-cv.uni-jena.de 2 Youtube Channel CVG:
https://www.youtube.com/channel/UCpnLVdxmvF0zEHHIVfqSxag
3 Google+: https://plus.google.com/+ComputerVisionGroupJena/videos 4 Michael Stifel Center Jena: http://www.mscj.uni-jena.de
Life-Long Learning 38
Friedrich Schiller University Jena Computer Vision Group
Related publications and more detailed information can be found at http://www.inf-cv.uni-jena.de
Life-Long Learning 39