OVERVIEW: RULE BASED DEFINITIONS BEHAVIOUR RECOGNITION 1. TEMPORAL - - PowerPoint PPT Presentation

School of Informatics, University of Edinburgh

OVERVIEW: RULE BASED BEHAVIOUR RECOGNITION

• 1. CROWLEY’S REPRESENTATION
• 2. ROLE RECOGNITION
• 3. SITUATION RECOGNITION
• 4. PREPROCESSING FOR SEQUENCE

RECOGNITION

• 5. SEQUENCE RECOGNITION
• 6. WHAT’S MISSING

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 1 School of Informatics, University of Edinburgh

DEFINITIONS

• 1. TEMPORAL GROUP: A TRACKED

INDIVIDUAL OR GROUP THROUGH A VIDEO SEQUENCE

• 2. ROLE: HOW A TEMPORAL GROUP

PARTICIPATES IN A SITUATION

• 3. SITUATION: A PARTICULAR

ASSIGNMENT OF ENTITIES TO ROLES + RELATIONS BETWEEN THE ENTITIES

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 2 School of Informatics, University of Edinburgh

• 4. SCENARIO: A PARTICULAR

SEQUENCE OF SITUATIONS

• 5. CONTEXT: A GRAPH/NETWORK

DESCRIBING ALL POSSIBLE SEQUENCES OF SITUATIONS FOR A GIVEN SCENARIO

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 3 School of Informatics, University of Edinburgh

BROWSE CONTEXT

MOVE MOVE BROWSE

SITUATION ROLES MOVE WALKER BROWSING BROWSER TYPICAL SCENARIOS: MB, MBM, MBMB, BM, . . .

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 4

School of Informatics, University of Edinburgh

RULE BASED BEHAVIOUR RECOGNITION USE CROWLEY’S REPRESENTATION FOCUS ON BROWSE CONTEXT MODEL AS EXAMPLE

MOVE MOVE BROWSE

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 5 School of Informatics, University of Edinburgh

SITUATION ROLE ACTIVITY LEVEL MOVE WALKER MOVEMENT BROWSING BROWSER STATIONARY

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 6 School of Informatics, University of Edinburgh

ROLE RECOGNITION RULES

ROLE RULE BROWSER IN BROWSABLE LOCATION ≥ 20 FRAMES WALKER DEFAULT IF NOT IN OTHER ROLES APPLIED TO TRACKED PERSON, MAPS (ROWT , COLT , T) → ROLET GIVES A SEQUENCE OF ROLE CLASSIFICATIONS: ROLE1, ROLE2, ROLE3, . . . ROLEN ALTERNATIVE ROLE LABELS POSSIBLE: BROWSER COULD ALSO BE AN IDLE PERSON, INSPECTOR

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 7 School of Informatics, University of Edinburgh

ROLE RECOGNITION RESULTS

ROLE GROUND CORRECTLY PERCENT TRUTH LABELED COUNT COUNT BROWSER 3584 2481 69.2% WALKER 33502 31783 94.9% SEE LABELED VIDEO

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 8

School of Informatics, University of Edinburgh

SCENARIO RECOGNITION CAN THE SEQUENCE OF ROLE CLASSIFICATIONS: ROLE1, ROLE2, ROLE3, . . . ROLEN BE RECOGNIZED AS A SCENARIO INSTANCE? SEQUENCE OF SITUATIONS FIT THE CONTEXT MODEL?

MOVE MOVE BROWSE

VALID SEQUENCE OF SITUATIONS: M, MB, MBM, MBMB, ... B, BM, BMB, BMBM, ...

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 9 School of Informatics, University of Edinburgh

PREPROCESSING I: ROLE NOISE FILTERING ASSUME ISOLATED ROLE LABEL IS NOISE EG: ...WWWWWWBWWWWWW... W: WALKER, B: BROWSER FILTER TO RELABEL: ROLET = MOST COMMON MEMBER({ROLEI : T − N ≤ I ≤ T + N}) (N = 63)

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 10 School of Informatics, University of Edinburgh

PREPROCESSING II: TEMPORAL GROUPING OBSERVATION: ROLES DON’T CHANGE QUICKLY, EG. PERSIST FOR AT LEAST 25 FRAMES (1 SECOND) COMPRESS ROLE LIST TO GROUPS OF CONSECUTIVE IDENTICAL ROLES WALKB

A =

{WALKA, WALKA+1, . . . WALKB} SEQUENCE OF ROLE GROUPS: W A

0 , BB A, T C B , W D C , B∞ D

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 11 School of Informatics, University of Edinburgh

PREPROCESSING III: GROUP FILTERING OBSERVATION: CAN’T HAVE GROUPS SMALLER THAN A GIVEN LENGTH (FUNCTION OF ROLE) ACTION: MERGE SHORT GROUPS WITH LONGEST NEIGHBOUR (OR MOST SIMILAR, OR ...) W A

0 , BB A, T C B , W D C , B∞ D → W A 0 , BC A, W D C , B∞ D

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 12

School of Informatics, University of Edinburgh

RECOGNITION INPUTS

• SEQUENCE OF ROLE GROUPS:

W A

0 , BC A, W D C , B∞ D

• TABLE OF ALLOWABLE ROLES IN A

GIVEN SITUATION

• TRANSITION DIAGRAM FOR EACH

CONTEXT

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 13 School of Informatics, University of Edinburgh

RECOGNITION ISSUES

• ERRONEOUS ROLE GROUPS
• LOOPS IN CONTEXT TRANSITION

DIAGRAM

• MULTIPLE CONTEXTS TO CONSIDER
• AMBIGUOUS ROLE GROUPS

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 14 School of Informatics, University of Edinburgh

RECOGNITION ALGORITHM MODIFIED INTERPRETATION TREE ALGORITHM PRUNED COMBINATORIAL SEARCH TREE EACH NODE IN THE TREE AT LEVEL T IS A PAIR: (SEQUENCE OF ROLES: R1, R2, . . . RT, SEQUENCE OF SITUATIONS: S1, S2, . . . ST)

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 15 School of Informatics, University of Edinburgh

SUCH THAT:

• ROLE RI OCCURS IN SITUATION SI
• ROLE RI FOLLOWS ROLE RI−1 IN THE

ROLE GROUP LIST

• SITUATION SI IS A VALID SUCCESSOR

SITUATION TO SI−1 IN THE CONTEXT TRANSITION DIAGRAM

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 16

School of Informatics, University of Edinburgh

WILDCARDS - * SOME ROLE GROUPS MAY BE INCORRECTLY LABELED DON’T WANT TO FAIL TO MATCH IF JUST A SMALL PROBLEM WILDCARD SITUATION MATCHES ANY ROLE GROUP, SATISFYING ALL PREVIOUS CONSTRAINTS

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 17 School of Informatics, University of Edinburgh

GIVES AUGMENTED TRANSITION DIAGRAM:

MOVE MOVE BROWSE

* * *

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 18 School of Informatics, University of Edinburgh

SEARCH TREE POTENTIAL FULL SEARCH TREE

MBM MB*

R0,R1,R2 R0,R1 R0 ROLES MATCHED

M B * BM M* MB B* *M *B **

...

EVERY POSSIBLE INTERPRETATION TO ROLE GROUP SEQUENCE R1, R2, . . . RN WITH BROWSE CONTEXT MODEL

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 19 School of Informatics, University of Edinburgh

SEARCHING THE TREE

• 1. DEPTH-FIRST SEARCH DOWN THE LEFT EDGES
• 2. IF ROLE GROUP TYPE DOES NOT MATCH THE

SITUATION, THEN FAIL THIS PATH AND BACKTRACK

• 3. SUCCESS ONLY IF YOU REACH LEVEL N (EXPLAINS

ALL N ROLE GROUPS)

• 4. MULTIPLE SUCCESSES POSSIBLE (EG. RIGHT EDGE IS

‘***...*’)

• 5. COUNT NUMBER OF FRAMES (NOT ROLE GROUPS)

MATCHED TO ‘*’

• 6. SELECT SOLUTION WITH THE SMALLEST COUNT

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 20

School of Informatics, University of Edinburgh

EXTENSIONS

• 1. MULTIPLE CONTEXTS: TRY ALL AND SELECT

CONTEXT WITH SMALLEST * FRAME COUNT

• 2. AMBIGUITY OF ROLE TYPES: MATCH WITH BOTH

LABELS

• 3. MODEL FOR TEMPORAL LENGTH OF SITUATIONS
• 4. CONTEXT CHANGES OR CONSECUTIVE SCENARIOS
• 5. GEOMETRY OF TRAJECTORIES (IGNORED)
• 6. INTERACTIONS (SEE LATER)

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 21 School of Informatics, University of Edinburgh

RECOGNITION RESULTS

• ROLE INSTANCES BEFORE PREPROCESSING: 37086
• MISLABELED ROLE INSTANCES BEFORE & AFTER

PREPROCESSING I: 4083 & 2843

• MERGED GROUPS OF CONSECUTIVE ROLES BEFORE

PREPROCESSING II: 170 (2822 STILL MISLABELLED ROLES)

• GROUPS JOINED TO NEIGHBOURS IN PREPROCESSING

II: 6

• ROLE GROUPS AFTER PREPROCESSING II: 164
• FRAMES RELABELLED IN PREPROCESSING II: 90
• FRAMES MATCHED TO WILDCARD: 449 OF 25665

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 22 School of Informatics, University of Edinburgh

RECOGNITION RESULTS II

CORRECTLY SCENARIO POSSIBLE RECOGNISED BROWSE 10 7 WALK 70 44 INACTIVE 46 29 DROP DEAD 5 1 TOTAL 131 81 (62%) FAILURE CAUSES: MULTIPLE SCENARIOS IN SINGLE TRACKING; TOO SHORT TIME IN SITUATION MORE WORK IN PROGRESS

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 23 School of Informatics, University of Edinburgh

WHAT WE HAVE LEARNED

• 1. SYMBOLIC BEHAVIOUR RECOGNITION

ALGORITHM

• 2. HOW TO APPLY CROWLEY’S

BEHAVIOUR MODEL

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 24

School of Informatics, University of Edinburgh

Lecture Problem DESCRIBE THE RECOGNITION RESULT FOR THE BROWSE CONTEXT ON THIS SEQUENCE OF ROLE GROUPS {MOVE, BROWSE, ENTER, MOVE}. DRAW THE INTERPRETATION TREE.

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 25 School of Informatics, University of Edinburgh

EXPECTATION GRAMMARS USE DOMAIN KNOWLEDGE TO EXPLAIN SEQUENCES EXPRESS IN SYMBOLIC FORM USING GRAMMAR USEFUL IN HIGHLY CONSTRAINED DOMAINS BASED ON MINNEN, ESSA, STARNER

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 26 School of Informatics, University of Edinburgh

TOWER OF HANOI TASK GOAL: MOVE WHOLE STACK TO NEW PEG CAN ONLY MOVE 1 RING EACH TIME ONLY SMALLER RINGS ON LARGER

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 27 School of Informatics, University of Edinburgh

EXAMPLE PROBLEM IMAGE UNDERSTAND WHAT IS HAPPENING IN THIS FRAME

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 28

School of Informatics, University of Edinburgh

ASSUMPTIONS FOREGROUND BLOB EXTRACTION (EG BY BACKGROUND IMAGE COMPARISON) BLOBS MAY CONTAIN MULTIPLE OBJECTS INTERESTING OBJECTS MOVE MOVING BLOBS MUST CONTAIN HAND

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 29 School of Informatics, University of Edinburgh

EXPLANATION GRAMMAR ALLOWABLE EVENTS AND THEIR SEQUENCE

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 30 School of Informatics, University of Edinburgh

KEY PROCESS STEPS

• 1. EXTRACT BLOBS
• 2. ASSIGN POTENTIAL LABELS

({ hand, block A, B, C, noise }) BY PERSISTENCE, MOTION, COLOR, POSITION

• 3. HYPOTHESIZE EVENTS BY BLOB INTERACTIONS

(SPLIT/MERGE, ...)

• 4. EXTEND CURRENT PARSINGS INTO NEW FRAME
• 5. REMOVE INCONSISTENCIES
• 6. RANK AT END BY PROBABILITIES

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 31 School of Informatics, University of Edinburgh

STOCHASTIC PARSING KEEP PROBABILITY ASSOCIATED WITH EACH PARSING PROBABILITY COMES FROM BLOB AND TARGET SIMILARITIES LOW PROBABILITIY HYPOTHESES REMOVED DETAILS UNCLEAR

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 32

School of Informatics, University of Edinburgh

PARTIAL PARSE TRACE

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 33 School of Informatics, University of Edinburgh

CONCLUSIONS GRAMMAR ENFORCES “BIG PICTURE” EVEN WITH LOTS OF INDIVIDUAL FRAME AMBIGUITY DETAILS OF SPEED, NUMBER OF HYPOTHESES MAINTAINED, COMPUTATIONAL COMPLEXITY LARGER GRAMMARS COULD BECOME INFEASIBLE

ECVision Summer School: 5 - Behaviour Understanding Fisher slide 34