Outline The Niagara Internet Query System 1:45 - 2:30 - - PowerPoint PPT Presentation

1

East Coast Systems

Larissa Smelkov Wayne Wight Joshua Sunshine Andrew See Swetha Shankar

Outline

• The Niagara Internet Query System

– 1:45 - 2:30

• Punctuating Continuous Data Streams

– 2:30 – 3:00

• Break

– 3:00 – 3:10

• Monitoring Streams (Aurora)

– 3:10 – 4:00

• Discussion

– 4:00 – 4:30

The Niagara Internet Query System

• Demo: Querying XML Documents
• Motivation
• Partial Results and Differential Operators
• Synchronization and Streams
• NiagaraCQ (Continuous Queries)

Demo: Querying XML Documents

Sample XML:

<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE W4F_DOC SYSTEM "movies.dtd"> <W4F_DOC> <Movie> <Title>Body Shots</Title> <Year>1999</Year> <Directed_By> <Director>Michael Cristofer</Director> </Directed_By> <Genres> <Genre>Drama</Genre> </Genres> </Movie> </W4F_DOC>

2

XML

• Q

L and SEQL

Sample XML-QL:

“Titles of movies directed by Terry Gilliam” Used by the Query Processing Engine. Converted to SEQL for the Search Engine.

WHERE <Movie> <Title>$v68</> <Directed_by> <Directory>$v71</> </> </> IN “*” conform_to http://www.cs.wisc.edu/niagara/data /xml-movies/dtd $v71 = “Terry Gilliam” CONSTRUCT <result> <Title>$v68</> </>

XML

• Q

L and SEQL

Sample SEQL:

Used to locate relevant URLs for the Query Processing Engine.

Movie CONTAINS (Title and Director IS "Terry Gilliam")

Next: Partial Results

On To Demo…

Next: Motivation and Partial Results

Issues with Querying the Internet

• Distributed over a lot of space
• Network connections not stable
• Sometimes sites not available
• Some data might be an infinite stream

These factors are a bottleneck in query evaluation. Blocking operators (nest, average) add to the problem The solution???

3

Blocking Operators

Operators that produce output only after receiving complete input.

• Examples:

– Nest – Average – Set Difference

Partial Results

Useful for infinite stream of data and also for slow connections. Traditional QE architecture unsuitable because: Unsuitable assumptions about properties of

• perators

Lack of synchronization support for producing consistent partial results

Properties Operators Should Have to Produce Partial Results.

• 1. Flexible input property
• 2. Anytime property
• 3. Non-monotonic input-output property

Flexible Input Property

• If some input streams are available,
• perators should not block.
• Example:

– One website is down, others are not. Partial Results should be available.

4

Anytime Property

• Anytime you ask for a result, the result

should be available.

• Example:

– An average is being computed, and the user wants the average so far.

Non

• m
• notonic I/O Property
• Operators should support input that is not

steadily increasing.

• Example:

– Set Difference:

• Computing {A} – {B}, a new value arrives in {B}

that needs to be removed from the output.

Differential Operators

• Example:

– Join gets updates from Average and Replicate – Does not need to re-compute the join for the entire result set.

Next: Synchronization

• Operators work on differences between old and

new input, rather than re-evaluating the query.

• Each tuple contains old value and new value.

Synchronization

• In Order to get

partial results,

• perators need to

be synchronized starting with the sources.

(More on this with the next paper.)

5

Querying Streams

• Sample Data:

<NEWSDOC> <Article> <Company> <Name>Intel </Name> <Symbol>DELL</Symbol> </Company> went bankrupt today. </Article> … <Quotes> <Quote> <Symbol>DELL</Symbol> <Price>120.35</Price> <Change>-13</Change> </Quote> …

Querying Streams

• Sample Query

“Tell me when a company is mentioned in a news stream and has fast falling stock prices.”

WHERE <NEWSDOC> <Article> <Company> <Symbol>$v1</> </> </> </> IN “http://127.0.0.1/test.php”, <Quotes> <Quote> <Symbol>$v2</> <Change>$c</> </> </> IN “http://127.0.0.1/test2.php”, $v1=$v2, $c < “-10” CONSTRUCT <result><Symbol>$v1</></> Next: NiagaraCQ

NiagaraCQ

• Sub-system of Niagara for Continuous Queries.
• Made to support thousands of queries.
• Uses timers and changes in source files to

trigger queries.

• Groups queries based on expression signature

to improve performance.

NiagaraCQ

6

NiagaraCQ (cont.)

Example XML-QL Queries and their expression signature.

Grouping Queries

Plan for Queries vs. Group Plan for Queries.

Grouping Queries (cont.)

• Selection Operators are added incrementally,

most selective first

– For example, “Symbol=INTC” before “Price < 100”

• Join Operators accompanied by Selection

Operators are added incrementally based on a cost model

– Either split with selection first (few very selective splits), or – Join first, then split (many splits sharing same join)

Next: Summary

Summary

(What you should walk away with)

• Niagara supports XML queries over the

Internet.

• Gives partial results and supports

streaming input.

• Allows continuous queries (NiagaraCQ).

7

References

• “NiagaraCQ: A Scalable Continuous Query System for Internet

Databases”, Jianjun Chen, David J. DeWitt, Feng Tian, Yuan Wang. SIGMOD 2000 , p379-390.

• “The Niagara Internet Query System”, Jeffrey Naughton, David

DeWitt, David Maier, and many others.

• “Differential Evaluation of Continual Queries”, L. Liu, C. Pu, R.

Barga, T. Zhou, Proceedings of the 16th International Conference on Distributed Computing Systems. May, 1996.

• “XML-QL: A Query Language for XML”, A. Deutsch, M. Fernandez,
• D. Florescu, A. Levy, D. Suciu. Proceedings of the Eighth

International World Wide Web Conference, Toronto, May, 1999.

• ✁

• ✟

• ✟

• ✟

• ✬

• ✬

8

• ☎

• ✽

• ✟

∅

∈

≥

≤

• ✭

• ✽

9

• ✟

• ✖

• ✽

10

• ✟

• ✽

• ✄

• ✟

• ✽

• ✽

• ✽

• ❈

• ☎

• ✽

11

The Aurora System The Aurora System

Larissa Smelkov Larissa Smelkov Wayne Wight Wayne Wight

Main Points Main Points

• Why was Aurora designed?

Why was Aurora designed?

• Who will find the system useful?

Who will find the system useful?

• What can Aurora do?

What can Aurora do?

• How does Aurora work?

How does Aurora work?

• When can I get a copy?

When can I get a copy?

Traditional DBMS vs. Monitoring Systems Traditional DBMS vs. Monitoring Systems

• HADP

HADP

• Only current

Only current

• Not so good

Not so good

• Synchronized

Synchronized

• No

No

• DAHP

DAHP

• Streams

Streams

• Good

Good

• Not stable

Not stable

• Yes

Yes Model Model Data storage Data storage Trigger support Trigger support Data Data Real time Real time

Who Needs Aurora? Who Needs Aurora?

• Military Surveillance / Tracking

Military Surveillance / Tracking

• Financial Applications

Financial Applications

• Power Plant Monitoring

Power Plant Monitoring

• Traffic Measurement and Rerouting

Traffic Measurement and Rerouting

• Lending Services

Lending Services

12

What Can Aurora Do? What Can Aurora Do?

• The Basic Operators

The Basic Operators

Filter and Drop (Projection) Filter and Drop (Projection)

Map (Transformation) Map (Transformation)

Group By Group By

Union Union

What Else Can Aurora Do? What Else Can Aurora Do?

• Windowed Operators

Windowed Operators

Slide Slide

Tumble Tumble

Sort Sort

Resample Resample

Query Model Query Model

• Continual queries

Continual queries

• Views

Views

• Ad

Ad-

• hoc queries

hoc queries

Query Model Query Model

13

How Does Aurora Work? How Does Aurora Work? Aurora’s GUI Aurora’s GUI

• Optimization Tactics

Optimization Tactics

• Inserting Projections

Inserting Projections

• Combining Boxes

Combining Boxes

• Reordering Boxes

Reordering Boxes

Box Cost = Execution Time per Box Cost = Execution Time per Tuple Tuple * Number of * Number of Tuples Tuples

Real Real-

• Time Optimization

Time Optimization

14

QoS Data Structures QoS Data Structures

• Response times

Response times

• Tuple drops

Tuple drops

• Values produced

Values produced

Storage Management Storage Management

• Queue Management

Queue Management

Windowed operators Windowed operators

Accumulation due to high traffic Accumulation due to high traffic

Stored on Stored on-

• disk to increase scalability

disk to increase scalability

Keeps high priority queues in main memory Keeps high priority queues in main memory

• Connection Point Management

Connection Point Management

Stores stream history Stores stream history

Uses B Uses B-

• Tree for excess storage

Tree for excess storage

Deletes stale value in batches Deletes stale value in batches

Scheduling Scheduling

• Generate long tuple trains

Generate long tuple trains

• Process complete train at once

Process complete train at once

• Pass trains to subsequent boxes

Pass trains to subsequent boxes

Introspection Introspection

• Static Analysis

Static Analysis

Checks for sufficient Checks for sufficient hardware and system hardware and system resources resources

• Dynamic Analysis

Dynamic Analysis

Watches for excessive Watches for excessive delays according to delays according to QoS QoS information information

15

Load Shedding Load Shedding

• Dropping Tuples

Dropping Tuples

• Filtering Tuples

Filtering Tuples

So, when can I get my copy? So, when can I get my copy?

Conclusion Conclusion

• Traditional DBMS vs. Monitoring Systems

Traditional DBMS vs. Monitoring Systems

• Aurora’s Potential Users

Aurora’s Potential Users

• Features

Features

• Query Model

Query Model

• Implementation

Implementation

• GUI

GUI

Conclusion Conclusion

• Optimization

Optimization

• QoS

QoS

• Storage Management

Storage Management

• Introspection

Introspection

• Load Shedding

Load Shedding

• Stage of Project

Stage of Project

16

If you remember nothing else, If you remember nothing else, remember this: remember this:

http://www.cs.brown.edu/research/aurora/ http://www.cs.brown.edu/research/aurora/

Fin Fin

Discussion Applications

• What kinds of queries could be run in Niagara
• vs. Aurora?

Note:

– Aurora uses window operators and stores history. – Niagara supports set operations with punctuated streams (but no apparent history). – Both support continuous queries.

17

Operators

• How could you apply traditional set
• perators to streams?
• What do Niagara or Aurora supply for this?

Partial Results?

• Niagara:

– Uses punctuated streams

• User can get partial results on request.
• Aurora:

– Continuously computes results.

• What happens when data comes too fast or too

slow?

– How does the system deal with this? – How will this affect what the user sees?

• Aurora supports streaming sources, such

as sensors.

• Niagara searches the web, with support

for streaming data.

How did this influence design?

• Suppose in the future sensors are web
• accessible. How will this affect Niagara?

Optimization?

Niagara:

• Groups queries for better

performance

• Uses indexes to reduce

number of URLs processed. Aurora:

• QoS based load-

shedding.

• Train and super-box

scheduling.

Are these systems scalable? What are the limits to scalability? How could the strategies of one be applied to the

• ther?