Outline The Niagara Internet Query System 1:45 - 2:30 - - PowerPoint PPT Presentation

outline
SMART_READER_LITE
LIVE PREVIEW

Outline The Niagara Internet Query System 1:45 - 2:30 - - PowerPoint PPT Presentation

Outline The Niagara Internet Query System 1:45 - 2:30 Punctuating Continuous Data Streams East Coast Systems 2:30 3:00 Break Larissa Smelkov 3:00 3:10 Wayne Wight Monitoring Streams (Aurora) Joshua


slide-1
SLIDE 1

1

East Coast Systems

Larissa Smelkov Wayne Wight Joshua Sunshine Andrew See Swetha Shankar

Outline

  • The Niagara Internet Query System

– 1:45 - 2:30

  • Punctuating Continuous Data Streams

– 2:30 – 3:00

  • Break

– 3:00 – 3:10

  • Monitoring Streams (Aurora)

– 3:10 – 4:00

  • Discussion

– 4:00 – 4:30

The Niagara Internet Query System

  • Demo: Querying XML Documents
  • Motivation
  • Partial Results and Differential Operators
  • Synchronization and Streams
  • NiagaraCQ (Continuous Queries)

Demo: Querying XML Documents

Sample XML:

<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE W4F_DOC SYSTEM "movies.dtd"> <W4F_DOC> <Movie> <Title>Body Shots</Title> <Year>1999</Year> <Directed_By> <Director>Michael Cristofer</Director> </Directed_By> <Genres> <Genre>Drama</Genre> </Genres> </Movie> </W4F_DOC>

slide-2
SLIDE 2

2

XML

  • Q

L and SEQL

Sample XML-QL:

“Titles of movies directed by Terry Gilliam” Used by the Query Processing Engine. Converted to SEQL for the Search Engine.

WHERE <Movie> <Title>$v68</> <Directed_by> <Directory>$v71</> </> </> IN “*” conform_to http://www.cs.wisc.edu/niagara/data /xml-movies/dtd $v71 = “Terry Gilliam” CONSTRUCT <result> <Title>$v68</> </>

XML

  • Q

L and SEQL

Sample SEQL:

Used to locate relevant URLs for the Query Processing Engine.

Movie CONTAINS (Title and Director IS "Terry Gilliam")

Next: Partial Results

On To Demo…

Next: Motivation and Partial Results

Issues with Querying the Internet

  • Distributed over a lot of space
  • Network connections not stable
  • Sometimes sites not available
  • Some data might be an infinite stream

These factors are a bottleneck in query evaluation. Blocking operators (nest, average) add to the problem The solution???

slide-3
SLIDE 3

3

Blocking Operators

Operators that produce output only after receiving complete input.

  • Examples:

– Nest – Average – Set Difference

Partial Results

Useful for infinite stream of data and also for slow connections. Traditional QE architecture unsuitable because: Unsuitable assumptions about properties of

  • perators

Lack of synchronization support for producing consistent partial results

Properties Operators Should Have to Produce Partial Results.

  • 1. Flexible input property
  • 2. Anytime property
  • 3. Non-monotonic input-output property

Flexible Input Property

  • If some input streams are available,
  • perators should not block.
  • Example:

– One website is down, others are not. Partial Results should be available.

slide-4
SLIDE 4

4

Anytime Property

  • Anytime you ask for a result, the result

should be available.

  • Example:

– An average is being computed, and the user wants the average so far.

Non

  • m
  • notonic I/O Property
  • Operators should support input that is not

steadily increasing.

  • Example:

– Set Difference:

  • Computing {A} – {B}, a new value arrives in {B}

that needs to be removed from the output.

Differential Operators

  • Example:

– Join gets updates from Average and Replicate – Does not need to re-compute the join for the entire result set.

Next: Synchronization

  • Operators work on differences between old and

new input, rather than re-evaluating the query.

  • Each tuple contains old value and new value.

Synchronization

  • In Order to get

partial results,

  • perators need to

be synchronized starting with the sources.

(More on this with the next paper.)

slide-5
SLIDE 5

5

Querying Streams

  • Sample Data:

<NEWSDOC> <Article> <Company> <Name>Intel </Name> <Symbol>DELL</Symbol> </Company> went bankrupt today. </Article> … <Quotes> <Quote> <Symbol>DELL</Symbol> <Price>120.35</Price> <Change>-13</Change> </Quote> …

Querying Streams

  • Sample Query

“Tell me when a company is mentioned in a news stream and has fast falling stock prices.”

WHERE <NEWSDOC> <Article> <Company> <Symbol>$v1</> </> </> </> IN “http://127.0.0.1/test.php”, <Quotes> <Quote> <Symbol>$v2</> <Change>$c</> </> </> IN “http://127.0.0.1/test2.php”, $v1=$v2, $c < “-10” CONSTRUCT <result><Symbol>$v1</></> Next: NiagaraCQ

NiagaraCQ

  • Sub-system of Niagara for Continuous Queries.
  • Made to support thousands of queries.
  • Uses timers and changes in source files to

trigger queries.

  • Groups queries based on expression signature

to improve performance.

NiagaraCQ

slide-6
SLIDE 6

6

NiagaraCQ (cont.)

Example XML-QL Queries and their expression signature.

Grouping Queries

Plan for Queries vs. Group Plan for Queries.

Grouping Queries (cont.)

  • Selection Operators are added incrementally,

most selective first

– For example, “Symbol=INTC” before “Price < 100”

  • Join Operators accompanied by Selection

Operators are added incrementally based on a cost model

– Either split with selection first (few very selective splits), or – Join first, then split (many splits sharing same join)

Next: Summary

Summary

(What you should walk away with)

  • Niagara supports XML queries over the

Internet.

  • Gives partial results and supports

streaming input.

  • Allows continuous queries (NiagaraCQ).
slide-7
SLIDE 7

7

References

  • “NiagaraCQ: A Scalable Continuous Query System for Internet

Databases”, Jianjun Chen, David J. DeWitt, Feng Tian, Yuan Wang. SIGMOD 2000 , p379-390.

  • “The Niagara Internet Query System”, Jeffrey Naughton, David

DeWitt, David Maier, and many others.

  • “Differential Evaluation of Continual Queries”, L. Liu, C. Pu, R.

Barga, T. Zhou, Proceedings of the 16th International Conference on Distributed Computing Systems. May, 1996.

  • “XML-QL: A Query Language for XML”, A. Deutsch, M. Fernandez,
  • D. Florescu, A. Levy, D. Suciu. Proceedings of the Eighth

International World Wide Web Conference, Toronto, May, 1999.

✂ ✁ ✄✆☎ ✝ ✞ ✁ ✟ ✠☛✡ ☞ ✌ ✍ ✎ ✏ ✑ ✌ ✎ ✁ ✟ ✠☛✡ ✄ ✎ ✒ ✓ ✔ ✁ ✌ ✟ ✏ ✠✕✡ ✌ ✖ ✏ ✗ ✁ ✘ ✖ ✎ ✏ ✌ ✙ ✚ ✁ ✛ ✌ ✟ ✌ ✙ ✜ ✢ ✣ ✤✥✦✆✧ ★✆✩ ✩ ✪ ✩ ✫ ✦ ✧ ✬✆✭ ✮ ✧ ✭ ✯✰ ✮✱ ✬ ✮✭ ✭✲ ✬ ✮ ✱ ✳ ✴ ✣ ✵ ✦✶ ✷ ✶ ✮ ✬✆✸ ✭✲ ✹ ✬ ✴ ✺ ✩ ✫✻ ✭ ✳ ✹ ✳ ✴ ✼ ✲ ✪ ✬ ✮✱ ✴ ✩ ✮ ✽
✘ ✾✿ ✁ ✒ ✌ ✖ ✏ ✓ ✘ ✿ ☎ ✂ ✎ ✘ ✖ ❀ ✘ ✟ ✓ ✂ ✟ ✁ ✌ ✒ ✙ ✽ ✚ ✘ ✟ ✒ ✌ ✿ ✑ ✌ ✂ ✔ ✁ ✒ ✌ ✂ ✎ ✝ ✙ ✽ ☞ ✁ ✙ ✎ ✟ ✌ ✾✿ ✁
✘❁ ✁ ✟ ✂ ✎ ✁ ✙ ✽
✘ ✘ ❀ ✽ ❂ ✘ ✖ ✝ ✿ ☎ ✙ ✎ ✘ ✖ ✽ ✓ ✘ ✒ ✁ ❃ ☎ ✁ ✟❄ ✘ ❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✘ ☎ ✂ ❁ ☎ ✂ ✘ ✖ ✿ ❄ ❅ ✔ ✁ ✖ ✂ ✔ ✁ ✎ ✖ ❁ ☎ ✂ ✎ ✙ ❀ ✎ ✖ ✎ ✙ ✔ ✁ ✏❆ ✾ ✿ ✘ ✝ ✞ ✎ ✖ ✛ ❇ ❈ ✰ ❉❊ ❋ ★ ✭ ✮ ✴ ✳
✮ ✱ ✴ ★ ✭ ❍ ✬ ✫ ✫ ✭✲ ✭ ✮ ✧ ✭ ✼❏■ ❑▼▲ ❑ ◆❖ ✹ ✴ P ✭ ✧ ✩ ◆◗ ✪ ✭ ✴ ✭ ✽ ❘ ✂ ✔ ✁ ✟ ❃ ☎ ✁ ✟❄ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✒ ✌ ✎ ✖ ✂ ✌ ✎ ✖ ✿ ✎ ✒ ✎ ✂ ✿ ✁ ✙ ✙ ✙ ✂ ✌ ✂ ✁ ❅ ✔ ✎ ✿ ✁ ✂ ✔ ✁ ❄ ✝ ✘ ✒ ❁ ☎ ✂ ✁ ✟ ✁ ✙ ☎ ✿ ✂ ✙ ❈ ✰ ❉❊ ❋ ★ ✭ ✮ ✴ ✳
✮ ✱ ✴ ★ ✭ ✶ ✮ ✬ ✩ ✮ ✩ ✫ ✼ ✳ ✮ ❍ ❑ ▲ ✴ ★ ✭ ✩ ◗ ✭✲ ✳ ✴ ✩ ✲ ◆ ✳ ✬ ✮ ✴ ✳ ✬ ✮ ✹ ✳ ★ ✬ ✹ ✴ ✩ ✲ ✺ ✩ ✫ ✳ ✪ ✪ ✩ ❖ ✴ ◗ ❖ ✴ ✴ ✩ ✭ ✪ ✬ ◆ ✬ ✮ ✳ ✴ ✭ ❍ ❖ ◗ ✪ ✬ ✧ ✳ ✴ ✭✹❏❙
slide-8
SLIDE 8

8

✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖ ✏ ✁ ✙ ✝ ✟ ✎ ✾ ✁ ✙ ✌ ✙ ☎ ✾ ✙ ✁ ✂ ✘ ❀ ✂ ✔ ✁ ✘ ☎ ✂ ❁ ☎ ✂✁ ✽ ✂ ✘ ✒ ✘ ✟ ✁ ✂ ☎ ❁ ✿ ✁ ✙ ✌ ✟ ✁ ✁ ✒ ✎ ✂ ✂ ✁ ✏ ✂ ✔ ✌ ✂ ✒ ✌ ✂ ✝ ✔ ✂ ✔ ✁ ❁ ☎ ✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖
✄✆☎ ✌ ✒ ❁ ✿ ✁ ✝ ✞ ✙ ✂ ✘ ✝ ✞ ✂ ✎ ✝ ✞ ✁ ✟ ✒ ✎ ✛ ✔ ✂ ✁ ✒ ✎ ✂ ❁ ☎ ✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖ ❀ ✘ ✟ ✌ ✾ ✌ ✖ ✞ ✟ ☎ ❁ ✂ ✝ ✘ ✒ ❁ ✌ ✖ ❄
✠ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ★ ✳ ✹ ✴ ★ ✭ ✹ ✳ ◆✭ ✹ ✧ ★ ✭ ◆ ✳ ✳ ✹ ❍ ✳ ✴ ✳ ✟ ✡ ✳ ✴ ✧ ★ ☛ ★ ✭ ✮ ✭ ✳ ✧ ★ ✴ ❖ ◗ ✪ ✭ ✳ ✴ ✴ ✲ ✬ P ❖ ✴ ✭ ◆ ✳ ✴ ✧ ★ ✭ ✹ ▲ ◗ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ✳ ✴ ✴ ✲ ✬ P ❖ ✴ ✭ ❙ ☞✆✌ ✍✆✎ ✏

✏✑ ✒ ✓ ✔ ✕

✖✆✗ ✘✚✙ ✗ ✛ ✙ ✜ ✢ ✖✆✗ ✘✚✙ ✗ ✛ ✜ ✢ ✍ ✣ ✎ ✓ ✣

✗ ✘ ✤ ✣

✗ ✛ ✥ ✗ ✘✚✙ ✗ ✛ ✦ ✧ ✌ ★✩ ✏ ✣ ✪ ✗ ✗ ✗ ✫ ★✎ ✓ ✌ ★ ✓ ✓ ✧✬ ✏ ✭ ✮ ✣ ✍✯ ✗ ✌ ✧ ✯ ✑ ✌ ✓ ✗ ✰ ✗ ✫ ★ ✯ ✣ ✓ ✣ ✫ ★ ✒ ✌ ✓ ✓ ✏ ✧ ★ ★ ✌ ✑ ✏ ✽ ✗ ✎ ✒ ✎ ✂ ✿ ✁ ✙ ✙ ✙ ✂ ✌ ✂ ✁ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✝ ✌ ✖ ❁ ☎ ✟ ✛ ✁ ✒ ✌ ✂ ✝ ✔ ✎ ✖ ✛ ✏ ✌ ✂ ✌ ❈ ✰ ❉ ❊ ✶ ✮ ✬ ✩ ✮ ☛ ✬ ✪ ✪ ✩ ✮ ✪ ✺
◗ ✴ ✴ ❖ ◗ ✪✆✭ ✹ ✴ ★ ✳ ✴ ❍✆✩ ✮ ✩ ✴ ◆ ✳ ✴ ✧ ★ ◗ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ✽ ✱ ✿ ✘ ✝ ✞ ✎ ✖ ✛ ✘ ❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✒ ✌ ❄ ✘ ☎ ✂ ❁ ☎ ✂ ✌ ✙ ☎ ✾ ✙ ✁ ✂ ✘ ❀ ✂ ✔ ✁ ✟ ✁ ✙ ☎ ✿ ✂ ❈ ✰ ❉ ❊ ✤ ✲ ✩ ❖ ◗ ❑ ✺ ✧ ✳ ✮ ✩ ❖ ✴ ◗ ❖ ✴ ❍ ✳ ✴ ✳ ✧ ✩ ✲ ✲ ✭ ✹ ◗ ✩ ✮ ❍ ✬ ✮ ✱ ✴ ✩ ✱ ✲ ✩ ❖ ◗ ✹ ✴ ★ ✳ ✴ ◆ ✳ ✴ ✧ ★ ◗ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ✹ ✽ ✓ ✁ ✖ ✙ ✘ ✟ ✂ ✔ ✌ ✂ ✁ ✒ ✎ ✂ ✙ ✂ ☎ ❁ ✿ ✁ ✙ ✡ ❅ ✎ ✂ ✔ ✲ ✔ ✘ ☎ ✟ ✡ ✒ ✎ ✖ ☎ ✂ ✁ ✡ ❁ ☎ ✿ ✙ ✁ ✳ ✽ ✲ ✴ ✡ ✵ ✡ ✵ ✳ ❅ ✘ ☎ ✿ ✏ ✒ ✁ ✌ ✖ ✂ ✔ ✌ ✂ ✌ ✿ ✿ ✂ ☎ ❁ ✿ ✁ ✙ ✌ ✟ ✁ ✏ ✘ ✖ ✁ ❀ ✘ ✟ ✔ ✘ ☎ ✟ ✴
✲ ✴ ✡ ✶ ✷ ✡ ✸ ✷ ✹ ✡ ✵ ✳ ❅ ✘ ☎ ✿ ✏ ✒ ✁ ✌ ✖ ✌ ✿ ✿ ✂ ☎ ❁ ✿ ✁ ✙ ✌ ✟ ✁ ✏ ✘ ✖ ✁ ❀ ✘ ✟ ✂ ✔ ✁ ❀ ✎ ✟ ✙ ✂ ✔ ✘ ☎ ✟ ✌ ✖ ✏ ✸ ✷ ✒ ✎ ✖ ☎ ✂ ✁ ✙
slide-9
SLIDE 9

9

✽ ✚ ☎ ✖ ✝ ✂ ✎ ✘ ✖ ✌ ✿
✘ ✛ ✟ ✌ ✒ ✒ ✎ ✖ ✛✁
✘ ✙ ✎ ✏ ✁ ✌ ❀ ❀ ✁ ✝ ✂ ✙ ✽ ✗ ✌ ✂ ❄ ✄ ✍ ✌ ✿ ☎ ✌ ✂ ✎ ✘ ✖ ✄ ✘ ✖ ✿ ❄ ✁ ✍ ✌ ✿ ☎ ✌ ✂ ✁ ✁ ☎ ❁ ✟ ✁ ✙ ✙ ✎ ✘ ✖ ✙ ❅ ✔ ✁ ✖ ✟ ✁ ❃ ☎ ✎ ✟ ✁ ✏ ❈ ✰ ❉ ❊ ☎ ❍ ✭ ✫ ✬ ✮ ✭ ☎ ✬ ✮ ✴ ✭ ✱ ✭✲ ✹ ✮ ✆ ☎ ✧ ✩ ✮ ✹ ✮ ✬ ✮ ✴ ✭ ✱ ✭✲ ✹ ☎ ✝ ✮ ✞ ✆ ✆ ✽ ✞ ✿ ✿ ✘ ❅ ✙ ✏ ✌ ✂ ✌ ✎ ✂ ✁ ✒ ✙ ✂ ✘ ✾ ✁ ❁ ✟ ✘ ✝ ✁ ✙ ✙ ✁ ✏ ✌ ✙ ✂ ✔ ✁ ❄ ✌ ✟ ✟ ✎ ✍ ✁ ✎ ✖ ✂ ✔ ✁ ✙ ✂ ✟ ✁ ✌ ✒ ✽ ✓ ✂ ✟ ✁ ✌ ✒ ✂ ✘ ✙ ✂ ✟ ✁ ✌ ✒ ❀ ☎ ✖ ✝ ✂ ✎ ✘ ✖ ✽ ❘ ☎ ✂ ❁ ☎ ✂ ✎ ✙ ✙ ✁ ❃ ☎ ✁ ✖ ✝ ✁ ✘ ❀ ✝ ✘ ✒ ❁ ☎ ✂ ✌ ✂ ✎ ✘ ✖ ✙ ✘ ✍ ✁ ✟ ✙ ✂ ✟ ✁ ✌ ✒ ✙ ✁ ✛ ✒ ✁ ✖ ✂ ✙ ✽ ✓ ✁ ✁ ✙ ✘ ✖ ✿ ❄ ❀ ✎ ✖ ✎ ✂ ✁ ❁ ✘ ✟ ✂ ✎ ✘ ✖ ✙ ✘ ❀ ✂ ✔ ✁ ✙ ✂ ✟ ✁ ✌ ✒ ✌ ✂ ✌ ✂ ✎ ✒ ✁
✚ ☎ ✖ ✝ ✂ ✎ ✘ ✖ ✟ ❀ ✟ ✘ ✒ ✗ ✎ ✙ ✂ ✂ ✘ ✗ ✎ ✙ ✂ ✽ ✓ ✂ ✟ ✁ ✌ ✒ ✠ ✂ ✁ ✟ ✌ ✂ ✘ ✟ ✚ ✘ ✍ ✁ ✟ ✓ ✂ ✟ ✁ ✌ ✒ ✓ ✝ ❈ ✫ ☎ ✦ ✆ ✡ ☛ ☎ ✦☞ ✞✌ ✆ ✝ ☛ ☎ ✦☞ ✍ ✌ ✆ ✝ ✎ ✽ ✄✆☎ ✌ ✒ ❁ ✿ ✁ ✝ ✓ ✁ ✿ ✁ ✝ ✂ ✡ ❁ ✟ ✁ ✏ ✎ ✝ ✌ ✂ ✁ ❁ ❈ ✏ ☎✑ ✝ ☞ ✳ ✌ ✆ ✡ ☞ ✳ ✌ ✬ ✫ ◗ ☎ ✳ ✆ ▲ ✏ ☎✑ ✝ ☞ ✳ ✌ ✆ ✡ ☞ ✌ ✩ ✴ ★ ✭✲ ☛ ✬ ✹ ✭ ✽ ✓ ✌ ❀ ✁ ✂ ❄ ✄ ✒ ✁ ✘ ✖ ✿ ❄ ✁ ✒ ✎ ✂ ✘ ☎ ✂ ❁ ☎ ✂ ❅ ✔ ✁ ✖ ❅ ✁ ✌ ✟ ✁ ✙ ☎ ✟ ✁ ✎ ✂ ❅ ✎ ✿ ✿ ✖ ✘ ✂ ✝ ✘ ✖ ❀ ✿ ✎ ✝ ✂ ❅ ✎ ✂ ✔ ✿ ✌ ✂ ✁ ✟ ✎ ✖ ❁ ☎ ✂✁ ✽ ❂ ✘ ✒ ❁ ✿ ✁ ✂ ✁ ✖ ✁ ✙ ✙ ✄ ✒ ✁ ✌ ✿ ❅ ✌ ❄ ✙ ✁ ✒ ✎ ✂ ✘ ☎ ✂ ❁ ☎ ✂ ❅ ✔ ✁ ✖ ❅ ✁ ✌ ✟ ✁ ✝ ✁ ✟ ✂ ✌ ✎ ✖ ✎ ✂ ❅ ✎ ✿ ✿ ✖ ✘ ✂ ✾ ✁ ✌ ❀ ❀ ✁ ✝ ✂ ✁ ✏ ✾ ❄ ✿ ✌ ✂ ✁ ✟ ✎ ✖ ❁ ☎ ✂✁
slide-10
SLIDE 10

10

✘ ❁ ✟ ✎ ✁ ✂ ❄ ✒ ✁ ✌ ✖ ✙ ✂ ✔ ✌ ✂ ❁ ☎ ✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖ ✛ ☎ ✌ ✟ ✌ ✖ ✂ ✁ ✁ ✙ ✂ ✔ ✌ ✂ ✖ ✘ ✒ ✌ ✂ ✝ ✔ ✎ ✖ ✛ ✂ ☎ ❁ ✿ ✁ ✙ ❅ ✎ ✿ ✿ ❀ ✘ ✿ ✿ ✘ ❅
✓ ✂ ✟ ✘ ✖ ✛ ✁ ✟ ✝ ✘ ✖ ✏ ✎ ✂ ✎ ✘ ✖ ✙ ✝ ❈ ✡ ✳ ❉ ✬ ◆ ✳ ✪✁ ✳ ✹ ◆ ❖ ✧ ★ ◗ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ✬ ✹ ✩ ❖ ✴ ◗ ❖ ✴ ✴ ✭ ❍ ✳ ✹ ◗ ✩ ✹ ✹ ✬ P✪✆✭ ❙ ❈ ✂ ✭ ✴ ✭✲ ◆ ✬ ✮ ✬ ✹ ✴ ✬ ✧
✩ ✲ ✳ ✮ ✺ ✩ ❖ ✴ ◗ ❖ ✴ ☛ ✭ ✳ ✲ ✭ ✱ ❖ ✳ ✲ ✳ ✮ ✴ ✭ ✭ ❍ ✩ ✫ ✭ ✬ ✴ ★ ✭✲ ✹ ✭ ✭ ✬ ✮✱ ✴ ★ ✳ ✴ ✩ ❖ ✴ ◗ ❖ ✴ ✩ ✲ ◗ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ✧ ✩ ✸ ✭✲ ✬ ✮ ✱ ✴ ★ ✳ ✴ ✩ ❖ ✴ ◗ ❖ ✴ ❙ ✽
✘ ✍ ✁ ✂ ✔ ✁ ❀ ✌ ✎ ✂ ✔ ❀ ☎ ✿ ✖ ✁ ✙ ✙ ✌ ✖ ✏ ❁ ✟ ✘ ❁ ✟ ✎ ✁ ✂ ❄ ✘ ❀ ✓ ✂ ✟ ✁ ✌ ✒ ✠ ✂ ✁ ✟ ✌ ✂ ✘ ✟ ✙
☞ ✁ ❀ ✎ ✖ ✁ ✂ ✔ ✁ ❀ ✌ ✎ ✂ ✔ ❀ ☎ ✿ ✖ ✁ ✙ ✙ ✌ ✖ ✏ ❁ ✟ ✘❁ ✟ ✎ ✁ ✂ ❄ ✎ ✖ ✟ ✁ ✿ ✌ ✂ ✎ ✘ ✖ ✂ ✘ ✂ ✌ ✾✿ ✁ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙
☞ ✁ ❀ ✎ ✖ ✁ ✛ ✿ ✘ ✾ ✌ ✿ ✟ ☎ ✿ ✁ ✙ ✘ ❀ ✘❁ ✁ ✟ ✌ ✂ ✎ ✘ ✖ ❀ ✘ ✟ ✌ ✿ ✿ ✓ ✂ ✟ ✁ ✌ ✒ ✠ ✂ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✂ ✔ ✌ ✂ ❀ ☎ ✿ ❀ ✎ ✿ ✿ ❀ ✌ ✎ ✂ ✔ ❀ ☎ ✿ ✖ ✁ ✙ ✙ ✌ ✖ ✏ ❁ ✟ ✘❁ ✟ ✎ ✁ ✂ ❄ ❁ ✟ ✘❁ ✁ ✟ ✂ ✎ ✁ ✙
✑ ✘ ✟ ✁ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✂ ✘ ✾ ✁ ✒ ✘ ✏ ✁ ✿ ✁ ✏ ✽ ✞ ✟ ✁ ✙ ✘ ✒ ✁ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✎ ✒ ❁ ✘ ✙ ✙ ✎ ✾✿ ✁ ✂ ✘ ☎ ✖ ✾ ✿ ✘ ✝ ✞ ❀ ✘ ✟ ✙ ✂ ✟ ✁ ✌ ✒ ✙ ☎ ✽ ✟ ☎ ✁ ✟❄ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✝ ✘ ☎ ✿ ✏ ✝ ✟ ✁ ✌ ✂ ✁ ❁ ☎ ✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖
✰ ❉ ❊ ✦ ✭ ✪✆✭ ✧ ✴ ✩ ◗ ✭✲ ✳ ✴ ✩ ✲ ☛ ✬ ✴ ★ ◗ ✲ ✭ ❍ ✬ ✧ ✳ ✴ ✭ ★ ✩ ❖ ✲ ✡ ✆ ✧ ✩ ❖ ✪ ❍ ✬ ◆ ◆ ✭ ❍ ✬ ✳ ✴ ✭ ✪ ✺ ✬ ✹ ✹ ❖ ✭ ✴ ☛ ✩ ◗ ❖ ✮ ✧ ✴ ❖ ✳ ✴ ✬ ✩ ✮ ✹ ✫ ✩ ✲ ★ ✩ ❖ ✲ ✝ ✆ ✳ ✮ ❍ ★ ✩ ❖ ✲ ✞ ✆ ❙ ✽
✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖ ✌ ✿ ✿ ✘ ❅ ✙ ✙ ✂ ✟ ✁ ✌ ✒ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✂ ✘ ❁ ✟ ✁ ✏ ✎ ✝ ✂ ✂ ✔ ✁ ❀ ☎ ✂ ☎ ✟ ✁
✒ ✎ ✂ ✔ ✂ ✔ ✁ ✔ ✁ ✿ ❁ ✘ ❀ ❁ ☎ ✖ ✝ ✂ ☎ ✌ ✂ ✎ ✘ ✖ ✡ ✂ ✟ ✘ ☎ ✾✿ ✁ ✙ ✘ ✒ ✁ ✘❁ ✁ ✟ ✌ ✂ ✘ ✟ ✙ ✝ ✌ ✖ ✘ ❁ ✁ ✟ ✌ ✂ ✁ ✁ ❀ ❀ ✁ ✝ ✂ ✎ ✍ ✁ ✿ ❄ ✘ ✖ ✙ ✂ ✟ ✁ ✌ ✒ ✎ ✖ ✛ ✏ ✌ ✂ ✌
slide-11
SLIDE 11

11

The Aurora System The Aurora System

Larissa Smelkov Larissa Smelkov Wayne Wight Wayne Wight

Main Points Main Points

  • Why was Aurora designed?

Why was Aurora designed?

  • Who will find the system useful?

Who will find the system useful?

  • What can Aurora do?

What can Aurora do?

  • How does Aurora work?

How does Aurora work?

  • When can I get a copy?

When can I get a copy?

Traditional DBMS vs. Monitoring Systems Traditional DBMS vs. Monitoring Systems

  • HADP

HADP

  • Only current

Only current

  • Not so good

Not so good

  • Synchronized

Synchronized

  • No

No

  • DAHP

DAHP

  • Streams

Streams

  • Good

Good

  • Not stable

Not stable

  • Yes

Yes Model Model Data storage Data storage Trigger support Trigger support Data Data Real time Real time

Who Needs Aurora? Who Needs Aurora?

  • Military Surveillance / Tracking

Military Surveillance / Tracking

  • Financial Applications

Financial Applications

  • Power Plant Monitoring

Power Plant Monitoring

  • Traffic Measurement and Rerouting

Traffic Measurement and Rerouting

  • Lending Services

Lending Services

slide-12
SLIDE 12

12

What Can Aurora Do? What Can Aurora Do?

  • The Basic Operators

The Basic Operators

✁ ✁

Filter and Drop (Projection) Filter and Drop (Projection)

✁ ✁

Map (Transformation) Map (Transformation)

✁ ✁

Group By Group By

✁ ✁

Union Union

What Else Can Aurora Do? What Else Can Aurora Do?

  • Windowed Operators

Windowed Operators

✁ ✁

Slide Slide

✁ ✁

Tumble Tumble

✁ ✁

Sort Sort

✁ ✁

Resample Resample

Query Model Query Model

  • Continual queries

Continual queries

  • Views

Views

  • Ad

Ad-

  • hoc queries

hoc queries

Query Model Query Model

slide-13
SLIDE 13

13

How Does Aurora Work? How Does Aurora Work? Aurora’s GUI Aurora’s GUI

  • Optimization Tactics

Optimization Tactics

  • Inserting Projections

Inserting Projections

  • Combining Boxes

Combining Boxes

  • Reordering Boxes

Reordering Boxes

Box Cost = Execution Time per Box Cost = Execution Time per Tuple Tuple * Number of * Number of Tuples Tuples

Real Real-

  • Time Optimization

Time Optimization

slide-14
SLIDE 14

14

QoS Data Structures QoS Data Structures

  • Response times

Response times

  • Tuple drops

Tuple drops

  • Values produced

Values produced

Storage Management Storage Management

  • Queue Management

Queue Management

✁ ✁

Windowed operators Windowed operators

✁ ✁

Accumulation due to high traffic Accumulation due to high traffic

✁ ✁

Stored on Stored on-

  • disk to increase scalability

disk to increase scalability

✁ ✁

Keeps high priority queues in main memory Keeps high priority queues in main memory

  • Connection Point Management

Connection Point Management

✁ ✁

Stores stream history Stores stream history

✁ ✁

Uses B Uses B-

  • Tree for excess storage

Tree for excess storage

✁ ✁

Deletes stale value in batches Deletes stale value in batches

Scheduling Scheduling

  • Generate long tuple trains

Generate long tuple trains

  • Process complete train at once

Process complete train at once

  • Pass trains to subsequent boxes

Pass trains to subsequent boxes

Introspection Introspection

  • Static Analysis

Static Analysis

✁ ✁

Checks for sufficient Checks for sufficient hardware and system hardware and system resources resources

  • Dynamic Analysis

Dynamic Analysis

✁ ✁

Watches for excessive Watches for excessive delays according to delays according to QoS QoS information information

slide-15
SLIDE 15

15

Load Shedding Load Shedding

  • Dropping Tuples

Dropping Tuples

  • Filtering Tuples

Filtering Tuples

So, when can I get my copy? So, when can I get my copy?

Conclusion Conclusion

  • Traditional DBMS vs. Monitoring Systems

Traditional DBMS vs. Monitoring Systems

  • Aurora’s Potential Users

Aurora’s Potential Users

  • Features

Features

  • Query Model

Query Model

  • Implementation

Implementation

  • GUI

GUI

Conclusion Conclusion

  • Optimization

Optimization

  • QoS

QoS

  • Storage Management

Storage Management

  • Introspection

Introspection

  • Load Shedding

Load Shedding

  • Stage of Project

Stage of Project

slide-16
SLIDE 16

16

If you remember nothing else, If you remember nothing else, remember this: remember this:

http://www.cs.brown.edu/research/aurora/ http://www.cs.brown.edu/research/aurora/

Fin Fin

Discussion Applications

  • What kinds of queries could be run in Niagara
  • vs. Aurora?

Note:

– Aurora uses window operators and stores history. – Niagara supports set operations with punctuated streams (but no apparent history). – Both support continuous queries.

slide-17
SLIDE 17

17

Operators

  • How could you apply traditional set
  • perators to streams?
  • What do Niagara or Aurora supply for this?

Partial Results?

  • Niagara:

– Uses punctuated streams

  • User can get partial results on request.
  • Aurora:

– Continuously computes results.

  • What happens when data comes too fast or too

slow?

– How does the system deal with this? – How will this affect what the user sees?

  • Aurora supports streaming sources, such

as sensors.

  • Niagara searches the web, with support

for streaming data.

How did this influence design?

  • Suppose in the future sensors are web
  • accessible. How will this affect Niagara?

Optimization?

Niagara:

  • Groups queries for better

performance

  • Uses indexes to reduce

number of URLs processed. Aurora:

  • QoS based load-

shedding.

  • Train and super-box

scheduling.

Are these systems scalable? What are the limits to scalability? How could the strategies of one be applied to the

  • ther?