using naiad to analyze twitter data in batch and real time
play

Using Naiad to Analyze Twitter Data in Batch and Real-time George - PowerPoint PPT Presentation

Feb 01, 2024 •118 likes •179 views

Using Naiad to Analyze Twitter Data in Batch and Real-time George Wort University of Cambridge 2017 Naiad Timely Dataflow System. Batch Processing. Stream Processing. Graph Processing. Supports iterative and incremental

  1. Using Naiad to Analyze Twitter Data in Batch and Real-time George Wort University of Cambridge 2017

  2. Naiad • Timely Dataflow System. • Batch Processing. • Stream Processing. • Graph Processing. • Supports iterative and incremental data analysis. • Low latency. • High throughput.

  3. Naiad • Complex system offering a lot of options. • Too complex for most applications? • Overheads and ease of use? • Additions: • Differential Dataflow. • GraphLINQ.

  4. Twitter Data Processing • Implement real-time and batch processing of tweet stream. • Geographically categorise word frequencies. • Allow selection of different levels of granularity. • Query geographical data. • Extend to allow similarity comparison between areas or cluster areas in batch. • Extend to view frequency of spelling mistakes in English.

  5. Assessment • Implement on a single machine and distributed environment. • Using: • The base Naiad system. • Differential dataflow. • GraphLINQ. • Assessing: • Ease of use. • Flexibility. • Latency. • Throughput.

  6. Questions?

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Big Data in Real-Time at Twitter by Nick Kallen (@nk) What is Real-Time Data? On-line

Big Data in Real-Time at Twitter by Nick Kallen (@nk) What is Real-Time Data? On-line

Big Data in Real-Time at Twitter by Nick Kallen (@nk) What is Real-Time Data? On-line queries for a single web request Off-line computations with very low latency Latency and throughput are equally important Not talking about

710 views • 46 slides
Naiad James Thomas Goals High-throughput batch processing Low-latency processing

Naiad James Thomas Goals High-throughput batch processing Low-latency processing

Naiad James Thomas Goals High-throughput batch processing Low-latency processing Iterative computation with streaming updates (novel contribution) For 100% in-memory workloads Novel Application, CIDR 2013 paper

423 views • 13 slides
Big Data in Real-Time at Twitter by Nick Kallen (@nk) Friday, November 5, 2010 What is

Big Data in Real-Time at Twitter by Nick Kallen (@nk) Friday, November 5, 2010 What is

Big Data in Real-Time at Twitter by Nick Kallen (@nk) Friday, November 5, 2010 What is Real-Time Data? On-line queries for a single web request Off-line computations with very low latency Latency and throughput are equally important

746 views • 46 slides
Naiad: A Timely Dataflow System Derek G. Murray Frank McSherry Rebecca Isaacs Michael Isard

Naiad: A Timely Dataflow System Derek G. Murray Frank McSherry Rebecca Isaacs Michael Isard

Naiad: A Timely Dataflow System Derek G. Murray Frank McSherry Rebecca Isaacs Michael Isard Paul Barham Martn Abadi MSR Silicon Valley Presented by Jesse Mu (jlm95) Background: dataflow programming Batch processing Batch processing

964 views • 74 slides
Agenda 1. Capital One 2. Traditional Batch Analytics 3. The Great Paradigm Shift Real-Time

Agenda 1. Capital One 2. Traditional Batch Analytics 3. The Great Paradigm Shift Real-Time

Agenda 1. Capital One 2. Traditional Batch Analytics 3. The Great Paradigm Shift Real-Time Analytics 4. What are the Drivers? 5. Apache Flink Next Generation Big Data Analytics Framework 6. Business Use Case: Customer Activity Event Logs 7.

1.05k views • 47 slides
In-mem DB Performance, Flash Cost Enabling Real-time AI June 2018 The Data-Driven Business

In-mem DB Performance, Flash Cost Enabling Real-time AI June 2018 The Data-Driven Business

In-mem DB Performance, Flash Cost Enabling Real-time AI June 2018 The Data-Driven Business Challenge From Reactive to Proactive AI Event-Driven Value of Data Interactive Batch Real-time Minutes Days Time to Action 2 Big and S low or

450 views • 22 slides
Is this normal? Finding anomalies in real-time data. Who am I? Im Theo (@postwait on Twitter)

Is this normal? Finding anomalies in real-time data. Who am I? Im Theo (@postwait on Twitter)

Is this normal? Finding anomalies in real-time data. Who am I? Im Theo (@postwait on Twitter) I write a lot of code 50+ open source projects several commercial code bases I wrote Scalable Internet Architectures I sit on the ACM

327 views • 18 slides
Rainbird: Real-time Analytics @Twitter Kevin Weil -- @kevinweil Product Lead for Revenue,

Rainbird: Real-time Analytics @Twitter Kevin Weil -- @kevinweil Product Lead for Revenue,

Rainbird: Real-time Analytics @Twitter Kevin Weil -- @kevinweil Product Lead for Revenue, Twitter TM Thursday, February 3, 2011 Agenda Why Real-time Analytics? Rainbird and Cassandra Production Uses at Twitter Open Source

717 views • 60 slides
Konstantin Ignatyev konstantin@inspur.com Data Acquisition Data Storage OLAP OLTP ETL (batch)

Konstantin Ignatyev konstantin@inspur.com Data Acquisition Data Storage OLAP OLTP ETL (batch)

Deploying Java Microservices on Kubernetes Konstantin Ignatyev konstantin@inspur.com Data Acquisition Data Storage OLAP OLTP ETL (batch) Raw (Near) Real Time Customer facing Slow Demand Analytics Exploration Streaming Internal

387 views • 12 slides
VSched: Mixing Batch And Interactive Virtual Machines Using Periodic Real-time Scheduling Bin

VSched: Mixing Batch And Interactive Virtual Machines Using Periodic Real-time Scheduling Bin

VSched: Mixing Batch And Interactive Virtual Machines Using Periodic Real-time Scheduling Bin Lin Peter A. Dinda Prescience Lab Department of Electrical Engineering and Computer Science Northwestern University http://www.presciencelab.org

467 views • 44 slides
Analyzing twitter data AN ALYZ IN G S OCIAL MEDIA DATA IN R Sowmya Vivek Data Science Coach

Analyzing twitter data AN ALYZ IN G S OCIAL MEDIA DATA IN R Sowmya Vivek Data Science Coach

Analyzing twitter data AN ALYZ IN G S OCIAL MEDIA DATA IN R Sowmya Vivek Data Science Coach Course Overview Extract and visualize twitter data Analyze tweet text Perform network analysis View tweets on the map Explore tweets on

662 views • 52 slides
Real-time PCR Data Markup Language A new standard for archiving and exchanging real-time PCR data

Real-time PCR Data Markup Language A new standard for archiving and exchanging real-time PCR data

Real-time PCR Data Markup Language A new standard for archiving and exchanging real-time PCR data Steve Lefever On behalf of the RDML consortium qPCR = golden standard for nucleic acids quantification Dedicated and precise instruments needed

780 views • 21 slides
Modelling of the London Bus Network The Data London Open Transport API Real-time bus

Modelling of the London Bus Network The Data London Open Transport API Real-time bus

Alex Gubbay Real Time Graph Modelling of the London Bus Network The Data London Open Transport API Real-time bus arrivals at every stop 130,000 data points a minute Also provided: Real-time traffic incidents Real-time

173 views • 5 slides
Infrastructure for Distributed Analysis Matev z Tadel PROBLEM: Provide real-time access to

Infrastructure for Distributed Analysis Matev z Tadel PROBLEM: Provide real-time access to

Infrastructure for Distributed Analysis Matev z Tadel PROBLEM: Provide real-time access to distributed data-storage and CPU resources In contrast to batch jobs, DA requires immediate response (few minutes): 1. Only staged data really

375 views • 4 slides
Naiad: A Timely Dataflow System Derek G. Murray Frank McSherry Rebecca Isaacs Michael Isard

Naiad: A Timely Dataflow System Derek G. Murray Frank McSherry Rebecca Isaacs Michael Isard

Naiad: A Timely Dataflow System Derek G. Murray Frank McSherry Rebecca Isaacs Michael Isard Paul Barham Martin Abadi Microsoft Research Silicon Valley Presented by Braden Ehrat Batch Stream Graph processing processing processing

791 views • 21 slides
Building Real-Time Visualizations at Scale Mike Barry @msb5014 Kevin Robinson @krob Hello!

Building Real-Time Visualizations at Scale Mike Barry @msb5014 Kevin Robinson @krob Hello!

Building Real-Time Visualizations at Scale Mike Barry @msb5014 Kevin Robinson @krob Hello! Hello! analytics.twitter.com Hello! Answers Building Real-time Visualizations Real-time Actionable User-focused Analytics at Twitter

1.08k views • 76 slides
FDOE DATABASE BASICS 2017 FAMIS Conference TERESA SANCHO Daytona Beach, FL Presenter June 28,

FDOE DATABASE BASICS 2017 FAMIS Conference TERESA SANCHO Daytona Beach, FL Presenter June 28,

FDOE DATABASE BASICS 2017 FAMIS Conference TERESA SANCHO Daytona Beach, FL Presenter June 28, 2017 1 www.FLDOE.org Purpose The purpose of this session is to provide a high-level overview of school district reporting to the Student and

655 views • 32 slides
Simplifying ML Workflows with Apache Beam & TensorFlow Extended Tyler Akidau @takidau

Simplifying ML Workflows with Apache Beam & TensorFlow Extended Tyler Akidau @takidau

Simplifying ML Workflows with Apache Beam & TensorFlow Extended Tyler Akidau @takidau Software Engineer at Google Apache Beam PMC + Apache Beam Portable data-processing pipelines + Example pipelines Python Java + Cross-language

680 views • 40 slides
Industrial Industrial Volume Volume Proprietary Proprietary Technology Technology Low-Cost

Industrial Industrial Volume Volume Proprietary Proprietary Technology Technology Low-Cost

Industrial Industrial Volume Volume Proprietary Proprietary Technology Technology Low-Cost Low-Cost 25 Boul. Montpellier, Montreal, Qc H4N 2G3 2019-10-03 1 TSXV: GRA | OTCQX: NNXPF TSXV: GRA | OTCQX: NNXPF www.nanoxplore.ca

566 views • 24 slides
Lake Chelan Coordinated Cost-Reimbursement Vanessa Brinkhuis Water Resources Program Contracts

Lake Chelan Coordinated Cost-Reimbursement Vanessa Brinkhuis Water Resources Program Contracts

Lake Chelan Coordinated Cost-Reimbursement Vanessa Brinkhuis Water Resources Program Contracts and Grant Specialist Ecology Watershed Planning Contact Central Regional Office Lake Chelan Reserve Chelan PUD reservoir right 1925 water

201 views • 9 slides
Life Cycle of a Batch Dina Kamen, Deputy Director of Accounting Joseph Walker, Professional

Life Cycle of a Batch Dina Kamen, Deputy Director of Accounting Joseph Walker, Professional

Life Cycle of a Batch Dina Kamen, Deputy Director of Accounting Joseph Walker, Professional Accountant I Clyde Woodbury, Professional Accountant I Hanz Cadet, Professional Accountant I May 3, 2016 Life Cycle of a Batch JAC Accounting Overview

915 views • 53 slides
A"endance Areas 2018-19+ Listening & Input Sessions September 14, 2017 & October

A"endance Areas 2018-19+ Listening & Input Sessions September 14, 2017 & October

A"endance Areas 2018-19+ Listening & Input Sessions September 14, 2017 & October 23, 2017 Welcome Thanks Purpose Provide feedback to help the superintendent make recommendaAons to the school board about aBendance areas

1.01k views • 54 slides
Hosted Mainframe Suzan Tasvibi-Tanha, Chief of Managed Services In-House Mainframe Challenges

Hosted Mainframe Suzan Tasvibi-Tanha, Chief of Managed Services In-House Mainframe Challenges

Hosted Mainframe Suzan Tasvibi-Tanha, Chief of Managed Services In-House Mainframe Challenges Funding Requires large capital investment and ongoing operational expenses Hosting facility, equipment leases / purchases and software

586 views • 11 slides
Mechanisation to Automation Mechanisation to Automation Title Sponsor Platinum Sponsor The

Mechanisation to Automation Mechanisation to Automation Title Sponsor Platinum Sponsor The

Organised By Mechanisation to Automation Mechanisation to Automation Title Sponsor Platinum Sponsor The Future of Cashew Processing JC Reddy Manufacturing & Technical Head VINK CORPORATION DMCC 25 th Jan 2019 24-26 January, 2019

482 views • 11 slides

More recommend