Weakly-Supervised Temporal Localization via Occurrence Count - - PowerPoint PPT Presentation

▶
weakly supervised temporal localization via occurrence
SMART_READER_LITE
LIVE PREVIEW

Weakly-Supervised Temporal Localization via Occurrence Count - - PowerPoint PPT Presentation

Jul 20, 2023 171 likes •1.46k views

Weakly-Supervised Temporal Localization via Occurrence Count Learning Julien Schroeter schroeterj1@cardiff.ac.uk Dr Kirill Sidorov Prof David Marshall C ONTEXT Temporal Localization DNN Weakly-Supervised Temporal Localization via Occurrence

slide-1
SLIDE 1

Weakly-Supervised Temporal Localization via Occurrence Count Learning

Julien Schroeter schroeterj1@cardiff.ac.uk

Dr Kirill Sidorov Prof David Marshall

slide-2
SLIDE 2

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONTEXT

Temporal Localization DNN

slide-3
SLIDE 3

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONTEXT

Temporal Localization

Input Data

DNN

Temporal Sequence

slide-4
SLIDE 4

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONTEXT

Temporal Localization

Precise Localization

Input Data Target

DNN

Impulse-like Events Temporal Sequence

slide-5
SLIDE 5

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONTEXT

Temporal Localization DNN

Training

Fully-Supervised

slide-6
SLIDE 6

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONTEXT

Temporal Localization DNN

Fully-Supervised

Training

Input Data

slide-7
SLIDE 7

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONTEXT

Temporal Localization DNN

Training Labels

Training

Input Data

Fully-Supervised

slide-8
SLIDE 8

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Our approach

Training

slide-9
SLIDE 9

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Our approach

Training

Input Data

slide-10
SLIDE 10

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Our approach

Training

Input Data Training Labels

slide-11
SLIDE 11

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training Labels

Our approach

Training

Input Data

Occurrence Counts

slide-12
SLIDE 12

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training

Occurrence Counts

Inference

LoCo

slide-13
SLIDE 13

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training

Occurrence Counts

Inference

LoCo

Input

slide-14
SLIDE 14

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training

Occurrence Counts

Inference

LoCo

Output Input

slide-15
SLIDE 15

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training

Occurrence Counts

Inference

LoCo

Occurrence Counts

Output Input

slide-16
SLIDE 16

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training

Occurrence Counts

Inference

LoCo

Precise Localization Occurrence Counts

Output Input

slide-17
SLIDE 17

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

LoCo

Training

Occurrence Counts

Inference

LoCo

Precise Localization

Weakly-Supervised

Occurrence Counts

Output Input

slide-18
SLIDE 18

Is it useful?

slide-19
SLIDE 19

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

Label Piano Music

slide-20
SLIDE 20

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

slide-21
SLIDE 21

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

Precise hand-labeling is very tedious

slide-22
SLIDE 22

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

Precise hand-labeling is very tedious Prone to labeling inaccuracy

slide-23
SLIDE 23

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

Proposed Approach

slide-24
SLIDE 24

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

How many notes per pitch?

slide-25
SLIDE 25

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

How many notes per pitch?

slide-26
SLIDE 26

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

How many notes per pitch?

slide-27
SLIDE 27

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

How many notes per pitch?

slide-28
SLIDE 28

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

OBJECTIVE

Weakening the annotation requirement

2

How many notes per pitch?

slide-29
SLIDE 29

The Model

slide-30
SLIDE 30

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Main Idea

Unlike existing methods, in which localization is explicitly achieved by design,

  • ur

model learns localization implicitly as a byproduct of learning to count instances.

slide-31
SLIDE 31

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Probability of Event occurrence

slide-32
SLIDE 32

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Event occurrence

slide-33
SLIDE 33

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Occurrence Count

slide-34
SLIDE 34

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Input Data

slide-35
SLIDE 35

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Input Data Occurrence Count

slide-36
SLIDE 36

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Input Data Occurrence Count

slide-37
SLIDE 37

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Counting Occurrences

Input Data Occurrence Count Estimated through RNN (e.g. LSTM)

slide-38
SLIDE 38

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Loss

Occurrence Count

slide-39
SLIDE 39

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Loss

Occurrence Count Compare them to true observed counts.

slide-40
SLIDE 40

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Loss

Occurrence Count Observed Count Compare them to true observed counts.

slide-41
SLIDE 41

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Loss

Occurrence Count Observed Count Compare them to true observed counts.

slide-42
SLIDE 42

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Loss

Occurrence Count Observed Count Compare them to true observed counts.

slide-43
SLIDE 43

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Loss

Occurrence Count Optimized with standard backpropagation

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Observed Count Compare them to true observed counts.

slide-44
SLIDE 44

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Full Pipeline

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

slide-45
SLIDE 45

Why does it work?

slide-46
SLIDE 46

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Poisson Binomial Counts

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Y follows a Poisson-binomial distribution

slide-47
SLIDE 47

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Bin k of count distribution at time t

slide-48
SLIDE 48

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Bin k of count distribution at time t

slide-49
SLIDE 49

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒖 = 𝟏

slide-50
SLIDE 50

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟕%

slide-51
SLIDE 51

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟕%

slide-52
SLIDE 52

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟒𝟏%

slide-53
SLIDE 53

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟒𝟏%

slide-54
SLIDE 54

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟒𝟏%

slide-55
SLIDE 55

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟒𝟏%

slide-56
SLIDE 56

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Recursion

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

𝒒𝒋 = 𝟒𝟏%

slide-57
SLIDE 57

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

slide-58
SLIDE 58

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-59
SLIDE 59

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-60
SLIDE 60

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-61
SLIDE 61

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-62
SLIDE 62

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-63
SLIDE 63

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-64
SLIDE 64

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-65
SLIDE 65

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Mass moves to the right

slide-66
SLIDE 66

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Consequence: Mass shifts are irreversible

Mass moves to the right

slide-67
SLIDE 67

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

No early triggering

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Consequence: Mass shifts are irreversible

  • prevents the model from triggering early
  • prevents the model from false alarms

Mass moves to the right

slide-68
SLIDE 68

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

slide-69
SLIDE 69

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Counting Loss

slide-70
SLIDE 70

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Counting Loss

slide-71
SLIDE 71

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Counting Loss

slide-72
SLIDE 72

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Learns to count Counting Loss

slide-73
SLIDE 73

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Learns to count Converge towards 0,1 extremes Counting Loss

slide-74
SLIDE 74

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

slide-75
SLIDE 75

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

A detection cannot be split into numerous small 𝐪𝐣(∙) contributions

slide-76
SLIDE 76

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

As the model learns to count event occurrences:

slide-77
SLIDE 77

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

As the model learns to count event occurrences:

  • pi(∙) converge towards 0,1 extremes
slide-78
SLIDE 78

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

As the model learns to count event occurrences:

  • pi(∙) converge towards 0,1 extremes
  • A detection cannot be split into

numerous small pi(∙) contributions

slide-79
SLIDE 79

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Mass Convergence

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • pi(∙) converge towards 0,1 extremes
  • A detection cannot be split into

numerous small pi(∙) contributions As the model learns to count event occurrences: A single 𝐪𝐣 ∙ will contain almost all of them mass for an event.

slide-80
SLIDE 80

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 1. Almost binary predictions
slide-81
SLIDE 81

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 2. No early triggering
  • 1. Almost binary predictions
slide-82
SLIDE 82

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 3. No systematic late bias

Not a theoretical property

  • 2. No early triggering
  • 1. Almost binary predictions
slide-83
SLIDE 83

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 3. No systematic late bias

Not a theoretical property Achieved trough an implementation trick:

  • 2. No early triggering
  • 1. Almost binary predictions
slide-84
SLIDE 84

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 3. No systematic late bias

Not a theoretical property Achieved trough an implementation trick: Feeding sequences of variable length

  • 2. No early triggering
  • 1. Almost binary predictions
slide-85
SLIDE 85

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 3. No systematic late bias
  • 2. No early triggering
  • 1. Almost binary predictions
slide-86
SLIDE 86

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

MODEL

Properties

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

  • 3. No systematic late bias

If the model accurately learns to count occurrences and if the events are detectable, then a coherent localization will emerge naturally.

  • 2. No early triggering
  • 1. Almost binary predictions
slide-87
SLIDE 87

Experiments

slide-88
SLIDE 88

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Experiment Specifications Detection of three different drum types in drum audio extracts

slide-89
SLIDE 89

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Experiment Specifications Detection of three different drum types in drum audio extracts

  • Tight tolerance of 50ms for a prediction to be correct
slide-90
SLIDE 90

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Experiment Specifications Detection of three different drum types in drum audio extracts

  • Tight tolerance of 50ms for a prediction to be correct
  • Comparison with fully-supervised benchmark models
slide-91
SLIDE 91

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Our approach

Signal

slide-92
SLIDE 92

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Fourier

Our approach

Signal Mel-spectrogram 1st order derivative

slide-93
SLIDE 93

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Fourier

Our approach

CNNs

Signal Mel-spectrogram 1st order derivative

Convolutional Representations

slide-94
SLIDE 94

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Fourier

Our approach

CNNs Convolutional Representations LSTM

Signal Mel-spectrogram 1st order derivative

slide-95
SLIDE 95

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Fourier

Our approach

CNNs Convolutional Representations LSTM FCs

Signal Mel-spectrogram 1st order derivative

slide-96
SLIDE 96

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Fourier

Our approach

CNNs Convolutional Representations LSTM Localization FCs

Signal Mel-spectrogram 1st order derivative

slide-97
SLIDE 97

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Fourier

Our approach

CNNs Convolutional Representations LSTM Localization Trained with our loss (using only occurrence counts) FCs

Signal Mel-spectrogram 1st order derivative

slide-98
SLIDE 98

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Results

slide-99
SLIDE 99

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Results State-of-the-art

slide-100
SLIDE 100

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Results State-of-the-art Great Overall F1-Score

slide-101
SLIDE 101

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Results Further tests on HH reveal that: Detection of three different drum types in drum audio extracts

slide-102
SLIDE 102

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DRUM DETECTION

Results Further tests on HH reveal that:

  • In that setting, the standard deviation is only of 4.35ms

for the distance between true and predicted hits.

Detection of three different drum types in drum audio extracts

slide-103
SLIDE 103

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

PIANO ONSET DETECTION

Results Detection of piano notes in audio extracts

slide-104
SLIDE 104

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

PIANO ONSET DETECTION

Results Detection of piano notes in audio extracts

  • Complex task with 88 channels
slide-105
SLIDE 105

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

PIANO ONSET DETECTION

Results Detection of piano notes in audio extracts

  • Complex task with 88 channels
  • Tight tolerance of 50ms for a prediction to be considered correct
slide-106
SLIDE 106

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

PIANO ONSET DETECTION

Results Detection of piano notes in audio extracts

  • Complex task with 88 channels
  • Tight tolerance of 50ms for a prediction to be considered correct
  • Comparison with fully-supervised benchmark models
slide-107
SLIDE 107

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

PIANO ONSET DETECTION

Results

slide-108
SLIDE 108

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

PIANO ONSET DETECTION

Results

slide-109
SLIDE 109

Digit Detection Experiment

slide-110
SLIDE 110

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Initial Image

Not a sequence

slide-111
SLIDE 111

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Initial Image Hilbert

Space-filling curve Not a sequence

slide-112
SLIDE 112

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Initial Image Hilbert

Space-filling curve Not a sequence A sequence

slide-113
SLIDE 113

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Input

slide-114
SLIDE 114

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Input Model

slide-115
SLIDE 115

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Input Model

1

Predictions

slide-116
SLIDE 116

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Predictions

1

slide-117
SLIDE 117

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Predictions

1

Hilbert

Space-filling curve

slide-118
SLIDE 118

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Main Idea

Predictions

1

Hilbert

Space-filling curve

Object Detection

slide-119
SLIDE 119

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Representations VGG-13 Ours

slide-120
SLIDE 120

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Detection Performance

Mean absolute distance between true and estimated bounding box centers: 9:04 pixels (approx. step size of the space filling curve)

slide-121
SLIDE 121

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Conclusion

The model learnt:

slide-122
SLIDE 122

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Conclusion

The model learnt:

  • 1. Feature representation
slide-123
SLIDE 123

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Conclusion

The model learnt:

  • 1. Feature representation
  • 2. Space-mapping
slide-124
SLIDE 124

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Conclusion

The model learnt:

  • 1. Feature representation
  • 2. Space-mapping
  • 3. Object detection
slide-125
SLIDE 125

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

DIGIT DETECTION EXPERIMENT

Conclusion

The model learnt:

  • 1. Feature representation
  • 2. Space-mapping
  • 3. Object detection

Using only occurrence counts as training labels

slide-126
SLIDE 126

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONCLUSION

This work shows that implicit model constraints can be used to ensure that accurate localization emerges as a byproduct

  • f learning to count occurrences.
slide-127
SLIDE 127

Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

CONCLUSION

This work shows that implicit model constraints can be used to ensure that accurate localization emerges as a byproduct

  • f learning to count occurrences.

Competitive results against fully-supervised state-of-the-art models.

slide-128
SLIDE 128

Questions?

Poster #255