Multi-dimensional Indexing GIS applications (maps): GIS - - PDF document
Advanced Data Structures NTUA 2007 NTUA 2007 R-trees and Grid File Multi-dimensional Indexing GIS applications (maps): GIS applications (maps): Urban planning, route optimization, fire or pollution monitoring, utility networks, etc.
GIS applications (maps): GIS applications (maps):
Urban planning, route optimization, fire or
Other applications:
VLSI design CAD/CAM model of human VLSI design, CAD/CAM, model of human
Traditional applications:
Multidimensional records
Point : 2 real numbers Line : sequence of points
Region : area included inside n-points
Topological relationships: Topological relationships:
adjacent, inside, disjoint, etc
Direction relationships:
Above, below, north_of, etc
Metric relationships:
“distance < 100”
And operations to express the
Selection queries: “Find all objects inside Selection queries: Find all objects inside
Nearest Neighbor-queries: “Find the
Spatial join queries: Two spatial relations S1 and
Point Access Methods (PAMs):
Index methods for 2 or 3-dimensional
Spatial Access Methods (SAMs):
Index methods for 2 or 3-dimensional
Approximate each region with a simple
Filtering step: Find all the MBRs (using
Refinement step:For each qualified
Point Access Methods (PAMs) vs Spatial Point Access Methods (PAMs) vs Spatial
PAM: index only point data
Hierarchical (tree-based) structures Multidimensional Hashing Space filling curve
SAM: index both points and regions
Transformations Overlapping regions Clipping methods
Given a point set and a rectangular query find the Given a point set and a rectangular query, find the
We allow insertions/deletions on line
Hashing methods for multidimensional points
Idea: Use a grid to partition the space each
Two disk access principle (exact match)
The Grid File: An Adaptable, Symmetric Multikey File Structure
Start with one bucket Start with one bucket
Select dividers along
Dividers cut all the
Each cell corresponds
Many cells can point
Cell directory
Dynamic structure using a grid directory Dynamic structure using a grid directory
Grid array: a 2 dimensional array with
Linear scales: Two 1 dimensional arrays that
/ g memory.
First use liner scales to determine the index into the cell
directory
access the cell directory to retrieve the bucket address (may
cause 1 I/O if cell directory does not fit in memory)
access the appropriate bucket (1 I/O)
use linear scales to determine the index into the cell directory. Access the cell directory to retrieve the bucket addresses of
buckets to visit.
Access the buckets.
Determine the bucket into which insertion must
If space in bucket, insert. Else, split bucket
how to choose a good dimension to split? ans: create convex regions for buckets.
If bucket split causes a cell directory to split do so
insertion of these new entries potentially requires a
Deletions may decrease the space utilization Deletions may decrease the space utilization.
We need to decide which cells to merge and
Buddy system and neighbor system
A bucket can merge with only one buddy in each
Merge adjacent regions if the result is a rectangle
Basic assumption: Finite precision in the Basic assumption: Finite precision in the
The address space is a square (image) and
Each element is called a pixel
Impose a linear ordering on the pixels
10 11
A
00 01 10 11 00 01
B
Given a point (x, y) and the precision K
Given a set of points, use a B+ -tree to
A range (rectangular) query in 2-d is
Find the z-values that contained in the
10 11
00 01 10 11 00 01
We want points that are close in 2d to
Note that in 2d there are 4 neighbors
Z-curve has some “jumps” that we
Hilbert curve avoids the jumps :
It has been shown that in general Hilbert is better It has been shown that in general Hilbert is better
Hi (order-i) Hilbert curve for 2ix2i array
Given a collection of geometric objects Given a collection of geometric objects
[Guttman 84] Main idea: extend B+ -tree to [Guttman 84] Main idea: extend B+ tree to
(only deal with Minimum Bounding Rectangles
A multi-way external memory tree A multi-way external memory tree Index nodes and data (leaf) nodes All leaf nodes appear on the same level Every node contains between t and M
The root node has at least 2 entries
eg., w/ fanout 4: group nearby rectangles
F= 4
F G D E H I J A B C
F= 4
P1 P2 P3 P4
F G D E H I J A B C
{ (MBR; obj_ptr)} for leaf nodes P1 P2 P3 P4 A B C
A B C
{ (MBR; node_ptr)} for non-leaf nodes P1 P2 P3 P4
A B C
i
E
4 6 8 10
y axis
b
E
f
1
E
2 e d a h g
E
5
E
6
E
4
E7
8 contents
E
9 i
E1 E2 E3 E4 E5 E6 E7 E8 Root E9 E1 E2
2 4 6 8 10 2
x axis
c
E
3
a b c d e f h g i E4 E5 E8
P1 P2 P3 P4
F G D E H I J A B C
P1 P2 P3 P4 A C
F G D E H I J A B C
Main points: Main points:
every parent node completely covers its ‘children’ a child MBR may be covered by more than one
a point query may follow multiple branches. everything works for any(?) dimensionality
P1 P2 P3 P4
F G D E H I J A B C
X
P1 P2 P3 P4
F G D E H I J A B C
Extend the parent MBR
P1 P2 P3 P4
F G D E H I J A B C
Y
How to find the next node to insert the
Using ChooseLeaf: Find the entry that
Other methods (later)
If node is full then Split : ex. Insert w
P1 P2 P3 P4
F G D E H I J A B C K
If node is full then Split : ex. Insert w
Q1 Q2 A B
P1 P5 P2 P3 P4
F G D E H I J C K W
Split node P1: partition the MBRs into two groups.
pick two rectangles as ‘seeds’; pick two rectangles as seeds ; assign each rectangle ‘R’ to the ‘closest’ ‘seed’
pick two rectangles as ‘seeds’; pick two rectangles as seeds ; assign each rectangle ‘R’ to the ‘closest’ ‘seed’: ‘closest’: the smallest increase in area
How to pick Seeds: How to pick Seeds:
Linear:Find the highest and lowest side in each
Quadratic: For each pair E1 and E2, calculate the
Use the ChooseLeaf to find the leaf
If leaf node is full, then Split, otherwise
Propagate the split upwards, if necessary
Adjust parent nodes
Find the leaf node that contains the entry E Find the leaf node that contains the entry E Remove E from this node If underflow:
Eliminate the node by removing the node entries
Reinsert the orphaned (other entries) into the tree
Other method (later)
R+ -tree: DO not allow overlapping so split R+ tree: DO not allow overlapping, so split
R* -tree: change the insertion, deletion
Hilbert R-tree: use the Hilbert values to insert
The original R-tree tries to minimize the
Is there any other property that can be
Optimization Criteria:
(O1) Area covered by an index MBR (O2) Overlap between index MBRs (O3) Margin of an index rectangle (O4) Storage utilization
Sometimes it is impossible to optimize
ChooseSubtree:
If next node is a leaf node, choose the node
Least overlap enlargement Least area enlargement Smaller area
Else
Least area enlargement Smaller area
SplitNode
Choose the axis to split Choose the two groups along the chosen axis
ChooseSplitAxis
Along each axis, sort rectangles and break them
ChooseSplitIndex
Along the chosen axis, choose the grouping that
Forced Reinsert:
defer splits, by forced-reinsert, i.e.: instead
Which ones to re-insert? Which ones to re insert? How many? A: 30%
Given a collection of geometric objects (points, lines,
point queries range queries k-nn queries
spatial joins (‘all pairs’ queries)
Given a collection of geometric objects (points, lines,
point queries range queries k-nn queries
spatial joins (‘all pairs’ queries)
Given a collection of geometric objects (points, lines,
point queries range queries k-nn queries
spatial joins (‘all pairs’ queries)
Given a collection of geometric objects (points, lines,
point queries range queries k-nn queries
spatial joins (‘all pairs’ queries)
Given a collection of geometric objects (points, lines,
point queries range queries k-nn queries
spatial joins (‘all pairs’ queries)
4 5 6 7 8 9 10 11 2 3
1 2 3 12 13 1
Q: How? (find near neighbor; refine
Q: How? (find near neighbor; refine...)
A1: depth-first search; then range query A1: depth first search; then range query
A1: depth-first search; then range query A1: depth first search; then range query
A1: depth-first search; then range query A1: depth first search; then range query
A2: [Roussopoulos+
A2: [Roussopoulos+ , sigmod95]:
At each node, priority queue, with promising
main idea: Every face of any MBR contains at least
MBR is a d-dimensional rectangle which is the MBR is a d dimensional rectangle, which is the
MBR f.p.: Every face of the MBR contains at least one
Visit an MBR (node) only when necessary How to do pruning? Using MINDIST and MINMAXDIST
MINDIST(P R) is the minimum distance between a MINDIST(P, R) is the minimum distance between a
If the point is inside R, then MINDIST= 0 If P is outside of R, MINDIST is the distance of P to
(p, ) p corner points l and u
the closest point in R is at least this distance away
=
d i i i
1 2
u=(u1, u2, …, ud)
i i
i
MINDIST = 0
l=(l1, l2, …, ld)
MINMAXDIST(P,R): for each dimension, find the
MINMAXDIST(P,R) is the smallest possible upper
MINMAXDIST guarantees that there is at least one
MINDIST(P, R) < = NN(P) < = MINMAXDIST(P,R)
MINDIST MINMAXDIST MINDIST
MINDIST MINMAXDIST MINMAXDIST
Downward pruning: An MBR R is discarded if there exists Downward pruning: An MBR R is discarded if there exists
Downward pruning: An object O is discarded if there
Upward pruning: An MBR R is discarded if an object O is
Downward pruning: An MBR R is discarded if there exists Downward pruning: An MBR R is discarded if there exists
MINDIST
MINMAXDIST
Downward pruning: An object O is discarded if there Downward pruning: An object O is discarded if there
Actual-Dist
MINMAXDIST
Upward pruning: An MBR R is discarded if an object O is Upward pruning: An MBR R is discarded if an object O is
MINDIST
Actual-Dist
MINDIST is an optimistic distance where MINMAXDIST is MINDIST is an optimistic distance where MINMAXDIST is
MINDIST MINMAXDIST
1
Initialize the nearest distance as infinite distance
1.
Initialize the nearest distance as infinite distance
2.
Traverse the tree depth-first starting from the root. At each Index node, sort all MBRs using an ordering metric and put them in an Active Branch List (ABL).
3.
Apply pruning rules 1 and 2 to ABL
4.
Visit the MBRs from the ABL following the order until it is empty
5.
If Leaf node, compute actual distances, compare with the best NN f d t if NN so far, update if necessary.
6.
At the return from the recursion, use pruning rule 3
7.
When the ABL is empty, the NN search returns.
Keep the sorted buffer of at most k current nearest
Pruning is done using the k-th distance
Global order [HS99]
Maintain distance to all entries in a common Priority
Use only MINDIST Repeat
Inspect the next MBR in the list
Inspect the next MBR in the list Add the children to the list and reorder
Until all remaining MBRs can be pruned
Best-first (BF) algorihm:
i
E
2 4 6 8 10 2 4 6 8 10
x axis y axis
b
E
f query point
1
E
2 e d c a h g
E
3
E
5
E
6
E
4
E7
8 search region contents
E
9 i
a 5 b 13 c 18 d 13 e 13 f 10 h 2 g 13 E1 1 E2 2 E3 8 E4 5 E5 5 E6 9 E7 13 E8 2 Root E9 17 i 10 E1 E2 E4 E5 E8
2 4 6 8 10 E 1 1 E 2 2
Visit Root
E 13 7
follow E
1 E 2 2 E 5 4 E 5 5 E 8 3 E 9 6 E 8 3
Action Heap follow E
2 E 2 8 E 5 4 E 5 5 E 8 3 E 9 6
follow E
8
Report h and terminate
E 17 9 E 13 7 E 5 4 E 5 5 E 8 3 E 9 6 E 17 9
Result {empty} {empty} {empty} {(h,
2 )}
E4 E5 8
i 10 E 5 4 E 5 5 E 8 3 E 9 6 E 13 7 g 13
R= Dequeue(PQ) If R is an object
Report R and exit (done!)
If R is a leaf page node
For each O in R, compute the Actual-Dists, InsertQueue(PQ, O)
If R is an index node
For each MBR C, compute MINDIST, insert into PQ
Best-First is the “optimal” algorithm in the sense that Best First is the optimal algorithm in the sense that
But needs to store a large Priority Queue in main
BB uses small Lists for each node. Also uses
Find all parks in each city in MA Find all parks in each city in MA Find all trails that go through a forest in MA Basic operation
find all pairs of objects that overlap
Single-scan queries Single-scan queries
nearest neighbor queries, range queries
Multiple-scan queries
spatial join
No existing index structures
Transform data into 1-d space [O89]
z-transform; sensitive to size of pixel
Partition-based spatial-merge join [PW96]
partition into tiles that can fit into memory plane sweep algorithm on tiles
Spatial hash joins [LR96, KS97] Sort data using recursive partitioning [BBKK01]
With index structures [BKS93, HJR97]
k-d trees and grid files R-trees
Join1(R,S) Repeat Find a pair of intersecting entries E in R and F in S If R and S are leaf pages then add (E,F) to result-set Else Join1(E,F)
Two ways to improve CPU – time
Restricting the search space Spatial sorting and plane sweep Spatial sorting and plane sweep
Join2(R,S,IV) Repeat Find a pair of intersecting entries E in R and F in S that overlap with IV If R and S are leaf pages then add (E,F) to result-set Else Join2(E,F,CommonEF)
Until all pairs are examined
In general, number of comparisons equals
size(R) + size(S) + relevant(R)* relevant(S)
Reduce the product term
1 5
Join1: 7 of R * 7 of S Now: 3 of R * 2 of S
1 3 5 1 1
= 49 comparisons Now: 3 of R 2 of S Plus Scanning: 7 of R + 7 of S =6 comp = 14 comp
Compute a read schedule of the pages to minimize Compute a read schedule of the pages to minimize
Local optimization policy based on spatial locality
Three methods
Local plane sweep Local plane sweep with pinning
L l d
Local z-order
Plane sweep again:
Read schedule r1, s1, s2, r3 Every subtree examined only once Consider a slightly different layout
After examining a pair (E F) compute the degree After examining a pair (E,F), compute the degree
degree(E) is the number of intersections between E and
If the degrees are non-zero, pin the pages of the
Perform spatial joins for this page Continue with plane sweep
1.
Compute the intersections between each rectangle of the
2.
Sort the rectangles according to the Z-ordering of their centers
3.
Use this ordering to fetch pages
r3 III III s1 r1 r2 s2 r4 IV II I IV II I III
How many disk (= node) accesses we’ll need for
range nn spatial joins
Worst Case vs. Average Case
In the worst case, we need to perform
We need to show a family of datasets We need to show a family of datasets
2 4 6 8 10
y axis
2 4 6 8 10
x axis
12 14 16 18 20
How many disk accesses (expected value) for range
query distribution wrt location?
How many disk accesses for range queries?
query distribution wrt location? uniform; (biased)
easier case: we know the positions of data nodes and
How many times will P1 be retrieved (unif. queries)?
How many times will P1 be retrieved (unif. POINT
How many times will P1 be retrieved (unif. POINT
How many times will P1 be retrieved (unif. queries of
Minkowski sum
How many times will P1 be retrieved (unif. queries of
Thus given a tree with n nodes (i= 1
Thus, given a tree with n nodes (i 1, ... n) we expect
2 2 , 1 1 , 2 1
i n i i
2 , 1 , i n i i
i 1 , 2 2 , 1 i n i i n i
2 1
Thus, given a tree with n nodes (i= 1, ... n) we expect
2 2 , 1 1 , 2 1
i n i i
2 , 1 , i n i i
i 1 , 2 2 , 1 i n i i n i
2 1
for point queries: only volume matters for horizontal-line queries: (q2= 0): vertical length
for large queries (q1, q2 > > 0): the count N matters
formula: easily extendible to n dimensions
splits should try to minimize area and perimeter ie., we want few, small, square-like parent MBRs rule of thumb: shoot for queries with q1= q2 = 0.1 (or
What if we have only the dataset D and the set of
We should “predict” the structures of a “good” R-tree
For point dataset, we can use the Fractal Dimension
(More in the [FK94] paper)
uniformly distributed in space.
MBRs that contain a given point in space. OR the total area covered by the MBRs over the area of the work space.
2
g g p g capacity is f, then we have Nln = N/f leaf nodes.
area of each leaf MBR is s2, then: S ti t f N f D
1 1 2 1 1
We need to estimate D1 from the dataset’s
density…
f
f
N 1
the density at the leaf level, from the density of the dataset: W l h id i l h h
2 1
levels of the tree.
Assuming Uniform distribution:
2 1 1 j h j j
+ =
j
− 2 1
Christos Faloutsos and Ibrahim Kamel. Beyond Uniformity and Independence: Analysis of R-trees Using the Concept of Fractal Dimension”. Proc. ACM PODS, 1994.
tree Performance”. Proc. ACM PODS, 1996.