The Graph Hawkes Neural Network for Forecasting on Temporal - - PowerPoint PPT Presentation

▶

Dec 03, 2022 134 likes •431 views

The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs By Zhen Han, Yunpu Ma, Yuyi Wang, Stephan Gnnemann, Volker Tresp Page 1 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC

SLIDE 1

Page 1 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs

By Zhen Han, Yunpu Ma, Yuyi Wang, Stephan Günnemann, Volker Tresp

SLIDE 2

Page 2 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Temporal Knowledge Graph (tKG)

Each Quadruple represents an events: (subject, predicate, object, timestamp) (Obama, visit, Turkey, 2009-04-05) Global Database of Events, Language, and Tone (GDELT)

SLIDE 3

Page 3 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Graph View of a Temporal Knowledge Graph

Each Quadruple represents an events: (subject, predicate, object, timestamp) (Obama, visit, Turkey, 2009-04-05)

SLIDE 4

Page 4 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

From Temporal Knowledge Graph to Event Sequence

t t1 (e1, p1，e2), (e1, p2, e5), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Timeline of a Sequence of Events.

2 6 3 5 1 4

p1 p2 p1 p3 p3

t1

SLIDE 5

Page 5 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

From Temporal Knowledge Graph to Event Sequence

t t2 t1 (e1, p1，e2), (e1, p2, e5), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Timeline of a Sequence of Events.

(e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e1), (e3, p1, e2)

2 6 3 5 1 4

p1 p2 p1 p3 p3

2 6 3 5 1 4

p1 p1 p2 p3 p1

t1 t2

SLIDE 6

Page 6 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

From Temporal Knowledge Graph to Event Sequence

t t3 t2 t1 (e1, p1，e2), (e1, p2, e5), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Timeline of a Sequence of Events.

(e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e1), (e3, p1, e2) (e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e2), (e3, p2, e2)

2 6 3 5 1 4

p1 p2 p1 p3 p3

2 6 3 5 1 4

p1 p1 p2 p3 p1

2 6 3 5 1 4

p3 p2 p2 p1 p2

t1 t2 t3

Slices of a Discrete-time Temporal Knowledge Graph.

SLIDE 7

Page 7 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

From Temporal Knowledge Graph to Event Sequence

t4 t3 t2 t1 (e1, p1，e2), (e1, p2, e5), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Timeline of a Sequence of Events.

(e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e1), (e3, p1, e2) (e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e2), (e3, p2, e2)

2 6 3 5 1 4

p1 p2 p1 p3 p3

2 6 3 5 1 4

p1 p1 p2 p3 p1

2 6 3 5 1 4

p3 p2 p2 p1 p2 p1 p3

t1 t2 t3

Slices of a Temporal Knowledge Graph.

2 6 3 5 1 4

p3 p

p2 p2

？

t4

SLIDE 8

Page 8 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Hawkes Process & Neural Hawkes Process

Hawkes Process[2] 𝜇𝑙 𝑢 = 𝜈! + ∑":$!%$ 𝛽!!,!exp −𝜀!!,! 𝑢 − 𝑢"

Intensity function of event type 𝑙 Base intensity Mutual excitation Exponential decaying with time

SLIDE 9

Page 9 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Hawkes Process & Neural Hawkes Process

Hawkes Process[2] 𝜇𝑙 𝑢 = 𝜈! + ∑":$!%$ 𝛽!!,!exp −𝜀!!,! 𝑢 − 𝑢"

Neural Hawkes Process[3] 𝜇! 𝑢 = 𝑔(𝒙!

'𝒊(𝑢)).

Intensity function of event type 𝑙 Base intensity Mutual excitation Exponential decaying with time Intensity function of event type 𝑙 Activation function Event-specific weight vector Hidden state vector

An Event Stream from the Neural Hawkes Process.

EventType-1 EventType-2 EventType-1 EventType-2 BaseIntensity-1 BaseIntensity-2 Intensity-1 Intensity-2

t1 t2 t3 t4 t

Excitation Inhibition

SLIDE 10

Page 10 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Challenge: Characteristics of Temporal Knowledge Graphs

Scalability: a huge amount of event types in tKGs.
Number of probable event types in our tKG dataset: 1.4 ⋅ 1010
Existing event types in our dataset: 1.2 ⋅ 106

t t3 t2 t1 (e1, p1，e2), (e1, p2, e5), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Event Sequence Extracted from a Temporal Knowledge Graph

(e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e1), (e3, p2, e2) (e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e2), (e3, p2, e2) (subject, predicate, object)

SLIDE 11

Page 11 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

How to improve the scalability of Hawkes process?

Considering an object prediction query (e1, p1, ?, t4).

SLIDE 12

Page 12 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

How to improve the scalability of Hawkes process?

Considering an object prediction query (e1, p1, ?, t4).
Modelling intensity functions inspired by score functions of KGs

SLIDE 13

Page 13 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

How to improve the scalability of Hawkes process?

Considering an object prediction query (e1, p1, ?, t4).
Modelling intensity functions inspired by score functions of KGs
Investigating the influence of the following historical event sequence:

eh,sp(e1, p1, t4) = {(e1, p1，e3, t1), (e1, p1，e4, t1), (e1, p1，e2, t2), (e1, p1，e4, t2), (e1, p1, e3, t3)}.

t t3 t2 t1 (e1, p1，e3), (e1, p1, e4), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Event Sequence Extracted from a Temporal Knowledge Graph

(e1, p1，e2), (e1, p1, e4), (e1, p2, e5), (e6, p3, e1), (e3, p1, e2) (e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e2), (e3, p2, e2)

SLIDE 14

Page 14 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Neighborhood Aggregation

Considering an object prediction query (e1, p1, ?, t4).
Neighborhood Aggregation Module[1] :

g O"! e#, 𝑞$ = 1 O"! e#, p$ (𝐟% + 𝐟&)

t t3 t2 t1 (e1, p1，e3), (e1, p1, e4), (e6, p3, e1), (e3, p2, e2), (e3, p3, e4)

Event Sequence Extracted from a Temporal Knowledge Graph

(e1, p1，e2), (e1, p1, e4), (e1, p2, e5), (e6, p3, e1), (e3, p1, e2) (e1, p1，e3), (e1, p2, e4), (e1, p2, e5), (e6, p3, e2), (e3, p2, e2)

Ot1(e1, p2)

Neighborhood Aggregation

t1 p

p1 e1 e3 e4 g(Ot1(e1, p2))

={e3, e4}

Embedding of the 3-th entity Embedding of the 4-th entity

SLIDE 15

Page 15 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Graph Hawkes Process

Object prediction query e'", e(", ? , t) .
Hidden state computed by a continuous-time LSTM (cLSTM) network[3]

𝐢'*+ e'", e(", t), e)

,,'(

= cLSTM 𝐟'", 𝐟(" , ∪./#

)

g O"#(e'", e(")

Historical event sequence Subject embedding Predicate embedding Neighborhood aggregation module

SLIDE 16

Page 16 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Graph Hawkes Process

Object prediction query e'", e(", ? , t) .
Hidden state computed by a continuous-time LSTM (cLSTM) network[3]

𝐢'*+ e'", e(", t), e)

,,'(

= cLSTM 𝐟'", 𝐟(" , ∪./#

)

g O"#(e'", e(")

Subject-centric intensity function

λ'*+ e0|e'", e(", t), e)

,,'(

= f 𝐗1 𝐟'" ⊕ 𝐗,𝐢'*+ e'", e(", t), e)

,,'( ⊕ 𝐟(" > 𝐟0

Historical event sequence Subject embedding Predicate embedding Neighborhood aggregation module Historical event sequence Subject embedding Predicate embedding Object embedding Hidden state vector Inner product

SLIDE 17

Page 17 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Link Prediction Task

Consider an object prediction query e'", e(", ? , t) and the corresponding 𝑓2

3,45.

Choose the object candidate with the highest intensity.

e!! e"! ti e1 e2 e3 e4 eN Query Candidates 1 2 3 4 N λ#$%(e!!, e"!, e&, t', e'

(,!")

λ#$%(e!!, e"!, e*, t', e'

(,!")

λ#$%(e!!, e"!, e+, t', e'

(,!")

λ#$%(e!!, e"!, e,, t', e'

(,!")

λ#$%(e!!, e"!, e-, t', e'

(,!")

Rank

SLIDE 18

Page 18 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Time Prediction Task

Given a time prediction query e'", e(", e0", t = ?

for t > tL.

Last occurrence time of the given event type

SLIDE 19

Page 19 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Time Prediction Task

Given a time prediction query e'", e(", e0", t = ?

for t > tL.

Computing conditional probability density that the given event type (esi, epi, eoi) occurs at time t based on the

survival analysis theory: p t e'", e(", e0", e)

,,'(, e) ,,0() = λ" e'", e(" , e0", t, e) ,,'(, e) ,,0( exp − ∫ "$ " λ6 e'", e(", e7%, τ, e) ,,'( , e) ,,0( dτ

Intensity function Last occurrence time of the given event type Historical event sequences Last occurrence time of the given event type

SLIDE 20

Page 20 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Time Prediction Task

Given a time prediction query e'", e(", e0", t = ?

for t > tL.

Computing conditional probability density that the given event type (esi, epi, eoi) occurs at time t based on the

survival analysis theory: p t e'", e(", e0", e)

,,'(, e) ,,0() = λ" e'", e(" , e0", t, e) ,,'(, e) ,,0( exp − ∫ "$ " λ6 e'", e(", e7%, τ, e) ,,'( , e) ,,0( dτ

The expectation of the next happening time:

E 𝑢2 = ∫

6& ⋈ τ > p τ e'", e(", e0", e) ,,'(, e) ,,0() 𝑒𝜐

Intensity function Last occurrence time of the given event type Historical event sequences Last occurrence time of the given event type Probability density function Historical event sequences Last occurrence time of the given event type

SLIDE 21

Page 21 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Experimental Results - Link Prediction

Table 1: Link prediction results: Mean Reciprocal Rank (MRR, %) and Hits@1/3/10 (%).

Datasets GDELT – filtered ICEWS14 – filtered

Models MRR Hits@1 Hits@3 Hits@10 MRR Hits@1 Hits@3 Hits@10 T-TransE 5.45 0.44 4.89 15.10 7.15 1.39 6.91 18.93 TA-TransE 9.57 0.00 12.51 27.91 11.35 0.00 15.23 34.25 TA-Dismult 10.28 4.87 10.29 20.43 10.73 4.86 10.86 22.52 LiTSEE 6.64 0.00 8.10 18.72 6.45 0.00 7.00 19.40 GHN 23.55 15.66 25.51 38.92 28.71 19.82 31.59 46.47

SLIDE 22

Page 22 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

How to Fairly Compare the Time Prediction Performance?

Our model (GHN) is nontrivial for time prediction.

SLIDE 23

Page 23 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Experimental Results - Time Prediction

108.00 1.78 6.10 20 40 60 80 100 120 LiTSEE Know-Evolve Our Model

MAE on the ICEWS14 Dataset (days)

303.78 110.80 7.18 50 100 150 200 250 300 350 LiTSEE Know-Evolve Our Model

MAE on the GDELT Dataset (hours)

SLIDE 24

Page 24 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Applications

Integrated conflict early warning Supporting clinical decisions in terms of personalized healthcare

SLIDE 25

Page 25 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Conclusion

Solving the challenge of massive event types.
Proposing the Graph Hawkes Process for forecasting on temporal knowledge graphs.
Define new evaluation metrics on temporal knowledge graph reasoning tasks.

SLIDE 26

Page 26 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Conclusion

Solving the challenge of massive event types.
Proposing the Graph Hawkes Process for forecasting on temporal knowledge graphs.
Define new evaluation metrics on temporal knowledge graph reasoning tasks.

Future Work

Enabling induction on new nodes.
Explainability.

SLIDE 27

Page 27 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Thank you!

Link to our paper: https://openreview.net/forum?id=kXVazet_cB

SLIDE 28

Page 28 Zhen Han, The Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs, AKBC 2020, June 22th.

Reference

[1] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pages 1024–1034, 2017. [2] Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90, 1971. [3] Hongyuan Mei and Jason Eisner. The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process. In Advances in Neural Information Processing Systems, 2017. [4] Woojeong Jin and Changlin Zhang and Pedro Szekely and Xiang Ren. Recurrent Event Network for Reasoning over Temporal Knowledge Graphs. arXiv preprint arXiv:1904.05530, 2019. [5] M. S. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling. Modeling relational data with graph convolutional networks. In ESWC, 2018. [6] Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. Know-Evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 3462–3471, 2017. [7] Odd Aalen, Ornulf Borgan, and Hakon Gjessing. Survival and event history analysis: a process point of

view. Springer Science & Business Media, 2008.