SLIDE 1
(Source: Softbank Pepper)
Audio recognition, context-awareness, and its applications - - PowerPoint PPT Presentation
Audio recognition, context-awareness, and its applications - - PowerPoint PPT Presentation
Audio recognition, context-awareness, and its applications Yoonchang Han Co-founder & CEO, Cochlear.ai 26 March, 2018 Rule-based Deep learning methods (Source: Softbank Pepper) See Computer vision Understand Natural language
SLIDE 2
SLIDE 3
See Understand language Listen Computer vision Natural language processing Speech recognition
(Source: Softbank Pepper)
SLIDE 4
Taking an umbrella Closing the window
SLIDE 5
(Audio source: http://www.freesound.org/people/Damiaan/)
Foot step sound High heels
SLIDE 6
(Source: BBC)
SLIDE 7
Easy for Humans Hard for Machines
SLIDE 8
Evolution of data processing technique
Feature engineering Early days Traditional ML Prediction ML Classifier Data Feature engineering Deep learning Deep learning More human effort More automatic Better performance
SLIDE 9
To tackle each topic (make some “rules”) To simulate how human understand the sound (and prepare data)
Domain knowledge
SLIDE 10
Required domain knowledge
Signal Processing Psychoacoustics Cognitive Sciences Acoustics Music Machine Learning
SLIDE 11
“Modern” audio identification pipeline
Audio Neural Network Time-frequency representation Output
butterfly flower
- bjects in an image ≈ instruments in a spectrogram
voice violin piano
SLIDE 12
“Machine listening” is the use of signal processing and machine learning for making sense of natural / everyday sounds, and recorded music.
- Machine listening lab, Queen Mary, Univ. of London
SLIDE 13
Voice
Language Age Gender Emotion Health
…
Music
Genre Mood Tempo Chord Pitch
…
SLIDE 14
Voice
…
Music
Machine listening
driving bus library train park city centre home market cafe
Acoustic scenes
… water boil glass break car horn footstep knock dog bark gun shot bird chirping snoring
Acoustic events
…
“Any” sound we hear everyday
sneeze crying
SLIDE 15
Acoustic scene/event detection
Computer vision
(Sources: Tensorflow, Facebook , Microsoft, Apple, Shazam)
Optical Character Recognition (OCR)
Machine listening
Facial recognition Object detection Voice recognition Music search Speaker identification
SLIDE 16
70 80 90 100
2013 2017
Scene classification accuracy (IEEE DCASE) 76 % 92 %
(Source: http://www.cs.tut.fi/sgn/arg/dcase2017/, http://c4dm.eecs.qmul.ac.uk/sceneseventschallenge/resultsSC.html)
SLIDE 17
Artificial Intelligence Deep Learning Machine Learning
SLIDE 18
Perceive Think Act
SLIDE 19
Cat Five, Zero
SLIDE 20
Know what it is (with input restriction) Know what it is Know what/where it is Know what/where it is + why
Simple Identification Closer to human
SLIDE 21
Sense (closed alpha release in April)
Acoustic event Music analysis Speech analysis
Genre/Mood /Key/Tempo Age/Gender /Emotion
Activity detection Scene classification
Dogbark/Babycry Carhorn/Snoring... Music,Speech,Others Indoor/Outdoor /Vehicles
SLIDE 22
Activity detection Unified model Why do we need…
SLIDE 23
It is really challenging because…
Recording environment Recording device Noises Local characteristics Overlapped / Polyphonic
SLIDE 24
Probability or Saliency ?
SLIDE 25
Example: AI speakers
“Alexa, turn on the light” “Alexa, play dance music” “Alexa, turn on TV”
(footstep sound, door slam, cough, Someone got back home, got a bad cold) turn on light / TV play suitable music adjust room temperature warmer (not just a pattern, there is a “reason”) ask to take cold medicine before sleep
Simple voice control IoT control-tower with context-awareness
SLIDE 26
Example: Humanoid robots
See things Understand speech
+
Listen things other than voice Know who they talk to
(Source: Atlas, Boston Dynamics)
SLIDE 27
Example: Autonomous car
(Source: NVIDIA)
Outside - Car horn (normal, air horn), Siren (fire truck, police, ambulance) Inside - Music mood, snoring, baby, anomaly detection (malfunction warning)
SLIDE 28
Musician
AI researcher
Architect Visual artist
Contemporary dancer
ATMO: Generative music for spatial atmo-sphere
+
SLIDE 29
Generative Music with contextual information
SLIDE 30
Ambient music Background music
Generative Music with contextual information
SLIDE 31
Analysis Result : Typing in a rainy day…
Contextual Information
Typing… Reading a book… Raining outside…
SLIDE 32
Microphone Speaker
SLIDE 33
SLIDE 34
SLIDE 35