WHAT CAN MACHINE LEARNING BRING TO CROWD ANALYSIS, MODELLING AND - - PowerPoint PPT Presentation

WHAT CAN MACHINE LEARNING BRING TO CROWD ANALYSIS, MODELLING AND SIMULATION? CONSIDERATIONS AND EXPERIMENTS

GIUSEPPE VIZZARI

CROWDS: MODELS AND CONTROL CIRM MARSEILLE, FRANCE, JUNE 3-7, 2019

OUTLINE

• AI, Machine Learning (ML), a turbulent moment
• A general schema for pedestrian and crowd research, and

where can ML provide support

• Sample applications of ML on pedestrian/crowd behaviour

analysis

• Sample applications of ML on pedestrian/crowd modelling

and simulation

• A look ahead

OUTLINE

• AI, Machine Learning (ML), a turbulent moment
• A general schema for pedestrian and crowd research, and

where can ML provide support

• Sample applications of ML on pedestrian/crowd behaviour

analysis

• Sample applications of ML on pedestrian/crowd modelling

and simulation

• A look ahead

AI, MACHINE LEARNING (ML), A TURBULENT MOMENT

• I don’t have to tell you about the

legitimate, sometimes excessive, and sometimes completely hyped interest around AI

• Two main factors behind this return of

attention, especially on machine learning

• Growth of computational power of devices

(especially GPUs and even dedicated devices, e.g. TPUs)

• Growing availability of data (especially

pictures, videos, but also text in different languages)

• This led to the development of novel open

source machine learning frameworks

• Legitimate research question: what can

machine learning bring to crowd analysis, modelling and simulation?

Next Generation AI Development Plan, State Council

• f China 201 (attain AI supremacy by 2030)

OK, MACHINE LEARNING… BUT WHICH MACHINE LEARNING?

Introducing Machine Learning

OK, MACHINE LEARNING… BUT WHICH MACHINE LEARNING?

Introducing Machine Learning

OUTLINE

• AI, Machine Learning (ML), a turbulent moment
• A general schema for pedestrian and crowd research, and

where can ML provide support

• Sample applications of ML on pedestrian/crowd behaviour

analysis

• Sample applications of ML on pedestrian/crowd modelling

and simulation

• A look ahead

A GENERAL SCHEMA FOR PEDESTRIAN AND CROWD RESEARCH

Model and simulator Target system Simulation results Empirical data

Simulation campaign execution Modelling and design of a simulator

Analysis of results and interpretation

Analysis of the dynamics of target system Synthesis Analysis

(i) Formalisation of phenomenologies (ii) Metrics, indicators, techniques (iii) Generation of synthetic datasets (i) Motivations/goals for model innovation (ii) Data for calibration, validation, learning

OUTLINE

• AI, Machine Learning (ML), a turbulent moment
• A general schema for pedestrian and crowd research, and where can

ML provide support

• Sample applications of ML on pedestrian/crowd behaviour

analysis

• A little state of the art
• Clustering for lane identification and characterization
• Sample applications of ML on pedestrian/crowd modelling and

simulation

• A look ahead

RELEVANT WORKS ON ML FOR AUTOMATED ANALYSIS OF PEDESTRIAN / CROWD BEHAVIOUR

• Within the Computer Vision area

lots of relevant work has been carried out on the border between surveillance and pedestrian/crowd studies

• A particularly interesting case is

represented by a work on PAMI on group detection

• Francesco Solera, Simone

Calderara, Rita Cucchiara: Socially Constrained Structural Learning for Groups Detection in Crowd. IEEE

• Trans. Pattern Anal. Mach. Intell.

38(5): 995-1008 (2016)

• Authors came up wit a ML approach

employing basic proxemic and group behavioural aspects that we developed in prior works…

• …plus some annotated data we

gathered GT!trajectories!

CLUSTERING FOR LANE IDENTIFICATION AND CHARACTERIZATION

• Bi-directional flows are generally characterized by the formation of

lanes

• Few approaches proposed means of automated identification and

quantitative characterization of this phenomenon

• Order parameter [Rex & Loewen, 2007]
• Clustering analysis [Hoogendoorn & Daamen, 2005]
• Rotation measurement [Feliciani & Nishinari, 2016]

DBSCAN [ESTER, KRIEGEL, SANDER AND XU, 1996]

• Unsupervised learning algorithm

based on the concept of density

• Parameters of the base version: 𝜁,

minPoints

• Clusters determined through the

concept of neighborhood:

• if distance between 2 points is less

than 𝜁, they are neighbors

• when one point has at least

minPoints neighbors it is a core point

• a cluster is defined as the set of

neighboring core points, plus neighboring points (border points)

• Remaining points are considered

noise

The choice of a suitable distance metric is crucial, just as the values for parameters

A TWO STEP DBSCAN APPROACH

• We employed a two step

clustering approach

• The first application of DBCAN

considers velocity vectors to separate main flows according to the direction

• The second one further subdivides

clusters achieved from the previous step according also to positions

• Different distance metrics,

essentially evaluating in step (i) angle among velocities and in step (ii) distance among pedestrians (discounted for )

• Overall 4 parameters (different 𝜁

and minPoints in the two steps)

• 1. Velocity

vectors

• 2. Positions &

identified flow directions

ACHIEVED RESULTS IN DIFFERENT EXPERIMENTS

AGREEMENT WITH HUMAN ANNOTATOR

• Cohen’s Kappa coefficient is used to measure the level of inter-rater

agreement between two coders in classifying a certain subject

• Pedestrians have been classified considering:
• their condition of belonging or not to any lane (i.e. gross classification – lane

identification)

• their belonging to a certain lane (i.e. granular classification – lane

characterization)

FURTHER WORK

• Make the model more robust to changes in density
• [SPOILER ALERT] Some ongoing work by Luca Crociani with

Francesco Zanlungo to be discussed at TGF19

• Connect the dots: move from a frame by frame clustering to a

time window, characterizing lanes in time

• Intuition: the set of pedestrians in a lane can change, to a certain

extent, without dissolving it or creating something new…

• Jaccard similarity can properly represent this intuition
• Additional validation elements are required
• Could provide additional means of validation for models

OUTLINE

• AI, Machine Learning (ML), a turbulent moment
• A general schema for pedestrian and crowd research, and where can

ML provide support

• Sample applications of ML on pedestrian/crowd behaviour analysis
• Sample applications of ML on pedestrian/crowd modelling and

simulation

• A little state of the art
• Learning observables for multi-scale modelling and simulation
• Deep neural networks for operational level behaviour modelling?
• A look ahead

RELEVANT WORKS ON ML FOR SIMULATION OF PEDESTRIAN / CROWD BEHAVIOUR

• Several past attempts of relatively little

success

• Works really worth attention
• Reiforcement Learning approaches
• Francisco Martinez-Gil, Miguel Lozano,

Fernando Fernández: Emergent behaviors and scalability for multi-agent reinforcement learning-based pedestrian

• models. Simulation Modelling Practice

and Theory 74: 117-133 (2017)

• Generative Adversarial Networks

approaches

• Javad Amirian, Jean-Bernard Hayet,

Julien Pettré: Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs. CVPR Workshops 2019

LEARNING OBSERVABLES FOR MULTI- SCALE MODELLING AND SIMULATION

• Allows simulating relatively large

populations in urban areas

• A multi-scale methodology is

employed:

• MATSim for road traffic
• A CA-based microscopic model

for pedestrian traffic

• The environment is mainly

represented by a network, but some parts can be simulated with higher fidelity using the CA, to better capture interaction among pedestrians

• Iterative approach for the

computation of routes…

Crociani, L., Lämmel, G., & Vizzari, G. (2016). Simulation-aided crowd management: A multi-scale model for an urban case study. In International Workshop on Agent Based Modelling of Urban Systems (pp. 151-171)

MULTI-SCALE MODEL – ROUTE CHOICE

• Sequential iterations of the simulation

abstract this concept:

• 1st iteration: shortest path
• Nth iteration: some agents re-

compute their path, based on the experienced cost in the previous iteration

• This allows the search of:
• Nash Equilibria, if the cost is the

individual travel time of the agent

• System Optimum, if the marginal

cost of the population is considered

• We basically add a more detailed

account of some specific edge of the graph through a CA

• But it comes at a cost!

Routes of agents (shortest path at the 1st iteration) Lo Loop p for # # iterat ations ns MA MATSim Routing algorithm Update experienced travel times simulation iteration St Star art

THE ETT MODEL (1/5)

• Goal: estimate the travel time
• f agents in the area of the

environment associated to each link, given the surrounding conditions

THE ETT MODEL (2/5)

• Goal: estimate the travel time
• f agents in the area of the

environment associated to each link, given the surrounding conditions

• ccupation

THE ETT MODEL (3/5)

• Goal: estimate the travel time
• f agents in the area of the

environment associated to each link, given the surrounding conditions

THE ETT MODEL (4/5)

• Goal: estimate the travel time of

agents in the area of the environment associated to each link, given the surrounding conditions

• The new model (ETT) is applied in

substitution of the CA, saving many computations and time

THE ETT MODEL (5/5)

• Goal: estimate the travel time of

agents in the area of the environment associated to each link, given the surrounding conditions

• The new model (ETT) is applied in

substitution of the CA, saving many computations and time

• Regression is a suitable technique

for the estimation of relationships between variables

• In particular we employed SVR

WORKFLOW WITH ETT

• 1. Train the SVR of each link from CA simulation results in the scenario, and

compose ETT

• 2. Load agents plans and configure their generation schedule into learnt links
• 3. Simulate:

For each generated agent:

1.

Ask the TT of the first link of the plan to the link model, according to current occupations

2.

Update the occupancy of the new/old link

3.

Let the agent wait

4.

Remove the link from the agent plan and repeat from 1 until the plan is completed

APPLICATION ON BENCHMARK SCENARIOS

Scenario 2: Scenario with bottleneck Scenario 1: Corridor with uni/bi-directional flow

SIMULATION RESULTS OF SCENARIO 1

• Diagrams show the number of agents in each link over the time of the simulation, achieved with the

ETT model (red) and CA model (blue and cyan)

• The ETT model reproduces well the dynamics of this scenario: the range and trend given by the CA-

based simulations are respected

SIMULATION RESULTS OF SCENARIO 1

• Diagrams show the number of agents in each link over the time of the simulation, achieved with the

ETT model (red) and CA model (blue and cyan)

• The ETT model reproduces well the dynamics of this scenario: the range and trend given by the CA-

based simulations are respected

SIMULATION RESULTS OF SCENARIO 2

• Results of ETT fit well the CA

simulations also in this case

• The configuration of the

routes of the tenth iteration also leads to very close results to the CA

1st iteration 10th iteration Link 7à8 Link 8à9

ABOUT COMPUTATION TIMES…

• Python implementation on a single threaded process
• Once trained, the ETT model is extremely efficient: speedup around 30 with 1000 agents

(it is about 5 with the microscopic model)

• The training phase is still a bottleneck, requiring a relatively large dataset (15-20

iterations) and a few minutes to train the model using cross-validation

DATA-DRIVEN MODELLING OF PEDESTRIANS PROXEMIC BEHAVIOUR

• We start to have available datasets from

experiments, much information on the pedestrian proxemics behaviour…

• … this information can be synthetized with current

machine learning and deep learning techniques, to build new data-driven models

• In this preliminary work we tested the

accuracy of a fully-connected neural network for regression of pedestrian velocities, as function of their internal state and neighbourhood: 𝑊

# $%& = 𝑔 𝑌# $ ,

𝑌#

$ is a vector including the state and

neighbourhood information of the pedestrian 𝑏

[2] [3] [1]

[1] Gorrini, A., Crociani, L., Feliciani, C., Zhao, P., Nishinari, K., & Bandini, S. (2016). Social groups and pedestrian crowds: experiment on dyads in a counter flow scenario. PED 2016. [2-3] from http://ped.fz-juelich.de

THE DEEP-NN INPUT

• To provide a discrete and

regular input to the NN, an

• val-shaped neighbourhood
• f the pedestrian is sampled

with a set of rays fired from its position

• if a ray hits an obstacle or

another pedestrian, its state is represented in the input

• The desired and actual

velocity of the subject pedestrian are also included in the network input

Coarse sampling in the rear area Fine sampling in the front area

Nothing hit: <rayLength, 0,0,empty> hit: <rayLength, 0,0, obstacle> hit: <rayLength, !

"#,%, ! "#,&, pedestrian>

FIRST RESULTS – UNIDIRECTIONAL CORRIDOR

Real / Simulated

• Trajectories from corridor, corners and t-junctions experiments (both 1-dir and 2-

dir) are used for the training, dataset is split in training/test part in the 70/30 proportion

• Best result show an error (Mean Absolute Error) of about .025 m/s on the

prediction for the next frame

FIRST RESULTS – CORNER

• … so it’s a good result?
• Well, certainly not yet…
• … I’m not hiding dust under the

carpet, I’m not selling you anything

• We are just looking at a situation

in which some pedestrians employ the learned model…

• Goal driven component of the

behaviour seems to be underestimated

• Also, the model seems slow in

responding to changes in the contextual situation

• We probably need to make a step

back and really ask ourselves “what are we providing to the network learning algorithm? Is it consistent to what we want to achieve?”

Real / Simulated

A LOOK AHEAD

CONSIDERATIONS

• ML can represent a source of inspiration, approaches, concrete

techniques for researchers trying to analyse and synthetize pedestrian and crowd behaviour…

• … it is not a silver bullet, and although these techniques are data driven,

knowledge on the subject is needed to make these techniques work properly and interpret results (also negative ones)

• Several works on this line of research are “popping up”, research

interest is legitimate and these topics are current…

• … but for some time it will be difficult to understand the actual value, limits,

and applicability of the propose techniques

• Systematically gathering and sharing data, better if annotated,

represents an invaluable contribution to the overall community

WORKS IN COLLABORATION WITH:

• STEFANIA BANDINI
• LUCA CROCIANI
• ANDREA GORRINI
• GREGOR LÄMMEL
• KATSUHIRO NISHINARI

THANKS!

Giuseppe Vizzari

University of Milano-Bicocca, Milano, Italy giuseppe.vizzari@unimib.it