Action Recognition in Low Quality Videos by Jointly Using Shape, - - PowerPoint PPT Presentation

▶
action recognition in low quality videos by
SMART_READER_LITE
LIVE PREVIEW

Action Recognition in Low Quality Videos by Jointly Using Shape, - - PowerPoint PPT Presentation

Oct 18, 2023 137 likes •500 views

Action Recognition in Low Quality Videos by Jointly Using Shape, Motion and Texture Features Saimunur Rahman, John See and Ho Chiung Ching Center of Visual Computing Multimedia University, Cyberjaya Motivation Local space-time features

slide-1
SLIDE 1

Action Recognition in Low Quality Videos by Jointly Using Shape, Motion and Texture Features

Saimunur Rahman, John See and Ho Chiung Ching Center of Visual Computing Multimedia University, Cyberjaya

slide-2
SLIDE 2

Motivation

  • Local space-time features have become popular for action

recognition in videos.

  • Current methods focus on high quality videos which are not

suitable for real-time video processing applications.

  • Current methods handles various complex video problems

(such as camera motion) but problem of video quality is still relatively unexplored [Oh et al’11].

IEEE ICSIPA '15 2

slide-3
SLIDE 3

Goal of this work

  • Investigate and analyze the performance of action recognition

under two low quality conditions:

− Spatial downsampling − Temporal downsampling

  • Joint utilization of shape, motion and texture features for

robust recognition of actions from downsampled videos.

  • Investigate ‘good’ feature combinations for action recognition

in low quality video.

IEEE ICSIPA '15 3

slide-4
SLIDE 4

Related Works

  • Shape and motion features
  • Space-time interest points [Laptev’05]
  • Dense Trajectories [Wang et al.’11]
  • Textural features
  • Local Binary Pattern on three orthogonal planes [Kellkompu et al.’08]
  • Extended Local binary pattern on three orthogonal planes [Mattivi and

Shao’09]

IEEE ICSIPA '15 4

slide-5
SLIDE 5

Outline

  • Spatio-temporal video features
  • Action recognition framework
  • Video downsampling
  • Experiments

IEEE ICSIPA '15 5

slide-6
SLIDE 6

Spatio-temporal video features Action recognition framework Video downsampling Experiments

IEEE ICSIPA '15 6

slide-7
SLIDE 7

Spatio-temporal video features

  • Shape and Motion Features (structures and its change with time)
  • Feature detector – Harris3D
  • Feature descriptor – HOG and HOF
  • Textural Features (change of statistical regularity with time)
  • Feature detector and descriptor – LBP-TOP

IEEE ICSIPA '15 7

slide-8
SLIDE 8

Harris3D detector [Laptev’05]

  • Space-time corner detector
  • Capable of detecting any spatial and temporal interest point
  • Dense scale sampling (no explicit scale selection)

IEEE ICSIPA '15 8

slide-9
SLIDE 9

HOG/HOF descriptor [Laptev’08]

  • Based on gradient and optical flow information
  • HOG – Histogram of oriented gradients
  • HOF – Histogram of Optical Flow
  • Detected 3D patch (xyt) is divided into grid of cells
  • Each cell is described with HOG and HOF.

IEEE ICSIPA '15 11

slide-10
SLIDE 10

LBP-TOP detector + descriptor [Zhao’07]

  • Extension of popular local binary pattern (LBP) operator into

three orthogonal planes (TOP)

  • Encodes shape and motion on three orthogonal planes (XY,

XT and YT)

  • Calculate occurrence of different plane histograms to form

final histogram (𝐼 = ℎ𝑌𝑍 ∙ ℎ𝑌𝑈 ∙ ℎ𝑍𝑈)

IEEE ICSIPA '15 12

LBP − TOP PXYPXTPYTRXRYRT 2 ∙ 3P

slide-11
SLIDE 11

LBP-TOP in action

IEEE ICSIPA '15 13 XY Plane XT Plane YT Plane XY Plane XT Plane YT Plane

+

Final histogram

slide-12
SLIDE 12

Spatio-temporal video features Action recognition framework Video downsampling Experiments

IEEE ICSIPA '15 15

slide-13
SLIDE 13

Evaluation framework

IEEE ICSIPA '15 16

Input Video Harris3D HOG/HOF Codebook SVM LBP-TOP Feature Encoding Classification

Feature Detection+Description Bag-of-words +

slide-14
SLIDE 14

Detection + description of features

IEEE ICSIPA '15 17 t t t

Interest Points Textures Shape - Motion Dynamic Textures Feature Detection Spatio-temporal Description Feature Vector Representation Input Video

slide-15
SLIDE 15

Bag-of-words representation

IEEE ICSIPA '15 18

Bag of space-time features + SVM with 𝜓2 kernel [Vedaldi’08]

Training feature vectors are clustered with k-means Each feature vector is assigned to its closest cluster center (visual word) An entire video sequence is represented as occurrence histogram of visual words Classification with multi-class non-linear SVM and 𝜓2 kernel

slide-16
SLIDE 16

Spatio-temporal video features Action recognition framework Video downsampling Experiments

IEEE ICSIPA '15 19

slide-17
SLIDE 17

Video Downsampling

  • Spatial downsampling (SD) decrease the spatial resolution.
  • Temporal downsampling (TD) reduces temporal sampling rate.

IEEE ICSIPA '15 20

SD Factor Description 𝑇𝐸1 Original Res. 𝑇𝐸2

1 2 Res. of Original

𝑇𝐸3

1 3 Res. of Original

𝑇𝐸4

1 4 Res. of Original

TD Factor Description T𝐸1 Original F.R. T𝐸2

1 2 F.R. of Original

T𝐸3

1 3 F.R. of Original

T𝐸4

1 4 F.R. of Original

Fig: Temporal Downsampling; (a) Original video (b) T𝐸2 (c) T𝐸3 Fig: Spatially downsampled videos. (a) 𝑇𝐸1 (b) 𝑇𝐸2 (c) 𝑇𝐸3 (d) 𝑇𝐸4 .

slide-18
SLIDE 18

Preview of downsampled videos

IEEE ICSIPA '15 21

Original Video

SD2 SD3 SD4 TD2 TD3 TD4

slide-19
SLIDE 19

Spatio-temporal video features Action recognition framework Video downsampling Experiments

IEEE ICSIPA '15 22

slide-20
SLIDE 20

Datasets

  • Two popular publicly available dataset
  • KTH action [Schuldt et al.’04]
  • Weizmann [Blank et al.’05]
  • Both captured in a controlled environment with homogeneous

background.

IEEE ICSIPA '15 23

slide-21
SLIDE 21

Feature combination used

  • Five different feature combinations

− Combination I : (HOG + HOF) - linear kernel − Combination II : (HOG + HOF) - χ2 kernel − Combination III : (HOG + HOF + LBP-TOP) - linear kernel − Combination IV : (HOG + HOF) + LBP-TOP - χ2 kernel − Combination V : (HOG + HOF + LBP-TOP) - χ2 kernel

IEEE ICSIPA '15 24

slide-22
SLIDE 22

KTH actions [Schuldt et al.’04]

  • Total 599 videos divided in 6 action classes
  • 25 people performed in 4 different scenarios
  • Frame resolution: 160 x 120 pixels
  • Frames per second: 25 (average duration 10-15 sec.)
  • Followed author specified setup for training-testing splits.
  • Performance measure: average accuracy over all classes

IEEE ICSIPA '15 25

slide-23
SLIDE 23

KTH original dataset - results

IEEE ICSIPA '15 28

KTH dataset HOGHOF vs. HOG+HOF

slide-24
SLIDE 24

KTH original dataset – results (2)

  • Best result for HOG+HOF (94.91%)
  • HOG+HOF helps to elevate the overall accuracy by 3–8% 
  • Kernelization of specific features are able to strengthen results
  • HOF + LBP-TOP : 93.06%
  • HOF + LBP-TOP - χ2 kernel : 94.44% 
  • HOF is more effective than HOG but improves when paired

with LBP-TOP 

IEEE ICSIPA '15 29

slide-25
SLIDE 25

KTH downsampled videos – results

IEEE ICSIPA '15 30

Spatial downsampling (k=2000) Temporal downsampling (k=2000)

slide-26
SLIDE 26

KTH downsampled videos – results (2)

  • STIPs and kernalized LBP-TOP appear to dominate the best

results within each mode 

  • LBP-TOP contributes more with the deterioration of spatial or

temporal quality (more significant in case of SD4 & TD4) 

  • Shape information are more important for low temporal resolution 
  • Motion information are more important for low spatial resolution 
  • Note: for STIPs detection in SD modes different k parameters

are used

IEEE ICSIPA '15 31

slide-27
SLIDE 27

Weizmann [Blank et al’05]

  • Total 93 videos divided in 10 action classes
  • 9 people performed different actions
  • Frame resolution: 180 x 144 pixels
  • Frames per second: 50 (average duration 2-3 sec.)
  • Performance measure: leave-one-out-cross-validation

IEEE ICSIPA '15 32

slide-28
SLIDE 28

IEEE ICSIPA '15 33

Weizmann video sample

slide-29
SLIDE 29

Weizmann original dataset - results

IEEE ICSIPA '15 35

  • Best result 94.44% for HOF.
  • HOF+LBP-TOP dominate best

result within each mode 

  • Kernelization of LBP-TOP features

are able to strengthen results 

  • Kernelization is less effective for

HOF features 

  • Shape is largely poor on all

combinations  but performs better after combining with LBP-TOP 

slide-30
SLIDE 30

Weizmann downsampled videos – results

IEEE ICSIPA '15 36

Spatial downsampling SD2, SD3 (k=2000) & SD4 (k=1500) Temporal downsampling SD2 (k=2000), SD3 (k=400)

slide-31
SLIDE 31
  • STIPs and kernalized LBP-TOP appear to dominate the best

results within each mode 

  • LBP-TOP contributes significantly more as the resolution

quality decreases 

  • Kernelized LBP-TOP achieves best accuracy rate at α = 4

and β = 3 

IEEE ICSIPA '15 37

Weizmann downsampled videos – results (2)

slide-32
SLIDE 32

Effects of kernelization

IEEE ICSIPA '15 38

Recognition accuracy with and without χ2-kernel, on the original KTH videos.

slide-33
SLIDE 33

Conclusion

  • This work utilizes a new notion of joint feature utilization for action

recognition in low quality videos

  • This woks shows how downsampled videos can particularly get

benefitted from textural information with shape and motion.

  • The combined usage of all three features (HOG+HOF+LBP-TOP)
  • utperforms the other competing methods across a majority of

cases.

  • Our best method is able to limit the drop in accuracy to around 8-

10% when the video resolutions and frame rates deteriorate to a fourth of their original values.

IEEE ICSIPA '15 39

slide-34
SLIDE 34

Future Works

  • Extend our evaluation to videos from more complex and

uncontrolled environments [Laptev et al.’04], [Oh et al.’11]

  • Investigate the simultaneous effects of both spatial and

temporal downsampling on videos

  • Explore other spatio-temporal textural features that might

exhibit more robustness towards video quality

IEEE ICSIPA '15 40

slide-35
SLIDE 35

Thank You

IEEE ICSIPA '15 41