Graph Algorithms for Community Detection & Recommendations Mark - - PowerPoint PPT Presentation

Graph Algorithms for Community Detection & Recommendations

Mark Needham & Amy Hodler, Neo4j

Mark Needham

@markhneedham

2

Neo4j Labs Engineer

Amy Hodler

@amyhodler

Analytics & AI Programs

Graph Algorithms for Community Detection & Recommendations Investigating the Graph Community

3

• Graph Algorithms
• Neo4j Social Network
• Finding Influencers
• Identifying Communities

What are Graph Analytics and Algorithms?

4

Query (e.g. Cypher/Python)

Real-time, local decisioning and pattern matching

Graph Algorithms Libraries

Global analysis and iterations You know what you’re looking for and making a decision You’re learning the overall structure of a network, updating data, and predicting

Local Patterns Global Computation

6

Don’t Need Graph Algorithms to Answer . . .

• Questions with just a few connections or flat (not nested)
• Questions solved with specific, well-crafted queries
• Simple statistical results (sums, averages, ratios)
• Example:
• Regular reporting based on defined criteria and

well-organized data

What Do People Do with Graph Algorithms?

7

Understand & Predict Complex Behavior

Requires Understanding Relationships and Structures Flow & Dynamics Interactions & Resiliency Propagation Pathways

Using Graph Algorithms

Explore, Plan, Measure

Find significant patterns and plan for optimal structures Score outcomes and set a threshold value for a prediction

Machine Learning

Use the measures as features to train an ML model

1st Node 2nd Node Common Neighbors Preferential Attachment label 1 2 4 15 1 3 4 7 12 1 5 6 1 1

10

Neo4j Graph Algorithms Library

Graph & ML Algorithms in Neo4j

+45

neo4j.com/

graph-algorithms- book/

Pathfinding & Search Centrality / Importance Community Detection Link Prediction

Finds optimal paths

• r evaluates route

availability and quality Determines the importance of distinct nodes in the network Detects group clustering or partition

• ptions

Evaluates how alike nodes are Estimates the likelihood

• f nodes forming a

future relationship

Similarity

Graph and ML Algorithms in Neo4j

• Parallel Breadth First Search &

DFS

• Shortest Path
• Single-Source Shortest Path
• All Pairs Shortest Path
• Minimum Spanning Tree
• A* Shortest Path
• Yen’s K Shortest Path
• K-Spanning Tree (MST)
• Random Walk
• Degree Centrality
• Closeness Centrality
• CC Variations: Harmonic, Dangalchev,

Wasserman & Faust

• Betweenness Centrality
• Approximate Betweenness Centrality
• PageRank
• Personalized PageRank
• ArticleRank
• Eigenvector Centrality
• Triangle Count
• Clustering Coefficients
• Connected Components (Union Find)
• Strongly Connected Components
• Label Propagation
• Louvain Modularity – 1 Step &

Multi-Step

• Balanced Triad (identification)
• Euclidean Distance
• Cosine Similarity
• Jaccard Similarity
• Overlap Similarity
• Pearson Similarity

Pathfinding & Search Centrality / Importance Community Detection Similarity

neo4j.com/docs/

graph-algorithms/current/

Updated April 2019

Link Prediction

• Adamic Adar
• Common Neighbors
• Preferential Attachment
• Resource Allocations
• Same Community
• Total Neighbors

• 1. Call as Cypher procedure
• 2. Pass in specification (Label, Prop, Query) and

configuration

• 3. stream variant returns (a lot) of results

CALL algo.<name>.stream('Label','TYPE',{conf}) YIELD nodeId, score

• 4. non-stream variant writes results to graph returns

statistics CALL algo.<name>('Label','TYPE',{conf})

How To…

Pathfinding & Search Centrality / Importance Community Detection Link Prediction Similarity

Pass in Cypher statement for node- and relationship-lists. CALL algo.<name>( 'MATCH ... RETURN id(n)', 'MATCH (n)-->(m) RETURN id(n) as source, id(m) as target', {graph:'cypher'})

Cypher Projection

15

Cypher Projection Example

Russian Twitter Trolls

https://www.nbcnews.com/pages/author/ben-popken

Inferred Relationships AMPLIFIED

CALL algo.pageRank( "MATCH (t:Troll) RETURN id(t) AS id", "MATCH (r1:Troll)-[:POSTED]->(:Tweet)<-[:RETWEETED]- (:Tweet)<-[:POSTED]-(r2:Troll) RETURN id(r2) as source, id(r1) as target", {graph:'cypher'})

PageRank on Inferred AMPLIFIED Graph https://www.nbcnews.com/tech/social-media/russian-trolls-went-a ttack-during-key-election-moments-n827176

How does it work?

Procedures Neo4j In Memory Projected Graph

Read projected graph Load projected graph

Graph Loader

Execute algorithm Store results

1 2 4 3

Everything is concurrent

1) Read projected graph 2) Load projected graph 3) Execute algorithm 4) Store results

Architecture Considerations

• Parallelization - everything, leverage lots of CPUs

○ Community Edition restricted to 4 cores!

• Memory

○ Need enough heap to fit projected graph in memory ○ Memory requirements vary per algorithm

• Causal Clusters

○ Do not run graph algos on core members ○ Streaming method only for read replicas ○ Consider snapshot

Enter the NEuler

20

install.graphapp.io

21

Investigating the Neo4j Social Graph

22

Neo4j Twitter Graph

Twint: Twitter scraping tool

Neo4j Twitter Graph

Determines the importance of distinct nodes in the network Developed for distinct uses or types

• f importance.

Centrality Algorithms

Degree Centrality

Degree Centrality

Tip / Caution This is the simplest of the centrality algorithms. Can measure in-degree,

• ut-degree, or both. When

globally averaged, it can be skewed by supernodes. Other algorithms are better for determining influence over more than just direct neighbors.

Measures the number of direct relationships

In-Degree

Degree Centrality - Uses

Use When Understanding immediate connectedness or direct influence Popularity & Gregariousness Quick estimation of network densities such as min/max and mean degrees

Likelihood of Flu

Individual probabilities

31

Degree Centrality

PageRank

Tip / Caution Test your dampening factor as it will change results (default works well for power law distributions.) Spark uses a inverse dampening factor resetProbability=0.15 is equal to dampingFactor:0.85 in Neo4j and other libraries Careful with mixing node types

PageRank

Measures the transitive (directional) influence of nodes and considers the influence of neighbors and their neighbors

PageRank Calculation

Node Being Ranked Nodes Linking To -> “u” Dampening Factor Outdegree of that Node CALL algo.pageRank('Page', 'LINKS', {iterations:20, dampingFactor:0.85, sourceNodes: [siteA]})

Personalized

PageRank - Uses

Fraud Detection Feature engineering for machine learning

Use When Anytime you’re looking for broad influence over a network

Many domain specific variations for differing analysis, e.g. Personalized PageRank for personalized recommendations

Recommendations Who To Follow with personalized PR

36

PageRank

Betweenness Centrality

Betweenness Centrality

Tip / Caution Computationally intensive: use RA Brandes approximation on large graphs. Assumes all communication between nodes happens along the shortest path and with the same frequency (not always the case in real life)

The sum of the % shortest paths that pass through a node, calculated by pairs

1. For a node, find the shortest paths that go through it

• B,C, E have no shortest paths and are assigned 0 value

2. For each shortest path in step one, calculate it’s percentage of the total possible shortest paths for that pair 3. Add together all the values in step two; this is a nodes Betweenness Centrality score 4. Repeat for each node

A B D E C

3.5 Pairs with Shortest Paths Through D Total Possible Shortest Paths for that Pair % of Total Through D (1/Total) A,E 1 1 B,E 1 1 C,E 1 1 B,C 2 (through D & A) 0.5 Betweenness Score 3.5

Node D Calculation

0.5

A

Betweenness Centrality

Betweenness Centrality - Uses

Use When Identify bridges Uncover control points Find bottlenecks and vulnerabilities

Network Resilience

Key points of cascading failure

41

Betweenness Centrality

Evaluates how a group is clustered

• r partitioned

Different approaches to define a community

Community Detection Algorithms

Louvain Modularity

Louvain Modularity

Continually maximizes the modularity by comparing relationship weights and densities to an estimate /average

Tip / Caution ALL Modularity algorithms:

• Merge smaller communities

into larger ones

• Review intermediates
• Can plateau with similar

modularity on several partitions - forming local maxima & stalling progress

• Treat as a guide and

test/validate results

2 1 4 1 1 2 14 4 14

Louvain Modularity - Uses

Use When Community detection in large networks Uncover hierarchical structures in data Evaluate different grouping thresholds

Understanding the Brain

Mapping hierarchy of functions

Louvain Modularity - Uses

Use When Community detection in large networks Uncover hierarchical structures in data Evaluate different grouping thresholds

Detecting Fraud Rings

Thresholds for bad apples vs rings

47

Graph Communities

48

Graph Communities

Degree Centrality to get a message directly in front of the most people with a single, immediate offer

49

Targeting Recommendations - EXAMPLES

PageRank to use an influencer with the broadest reach to all communities for a greater ripple effect Communities for use of different promoted tweets &

• ffers per sub-community

Betweenness Centrality + Louvain to find the person who can best connect to a specific sub-community

D

Learn More

50

51

Online Training!

Applied Graph Algorithms Data Science With Neo4j

https://neo4j.com/graphacademy/online-training/data-science https://neo4j.com/graphacademy/online-training/applied-graph-algorithms

Free O’Reilly Book neo4j.com/ graph-algorithms-book

• Spark & Neo4j Examples
• Machine Learning Chapter

Hunger Games Questions for

"Graph Algorithms for Community Detection & Recommendations"

1. Easy: Which of these algorithms is a community detection algorithm?

a. PageRank b. Common Neighbors c. Louvain Modularity

2. Medium: What is NOT a good use for a Graph Projection?

a. Infer Relationships b. Shape a graph for your algorithm to run on c. Save a new subgraph

3. Hard: The default dampening factor for the PageRank algorithm is 0.85. This default works well for graphs that have what distribution of relationships?

a. Power Law b. Poisson c. Normal

Answer here: r.neo4j.com/hunger-games