NTTS 2015 - Session 6A Big data sources: web scraping and smart - PowerPoint PPT Presentation

Feb 18, 2023 •290 likes •396 views

Automatic price collection on Ingolf Boettcher the internet (web scraping) Brussels 10. March 2015 NTTS 2015 - Session 6A Big data sources: web scraping and smart meters www.statistik.at Wir bewegen Informationen Web scraping There is a

Automatic price collection on Ingolf Boettcher the internet (web scraping) Brussels 10. March 2015 NTTS 2015 - Session 6A – Big data sources: web scraping and smart meters www.statistik.at Wir bewegen Informationen
Web scraping There is a huge amount of data on the internet <HTML> How can we best collect/scrape/harvest <HEAD> data from there for <TITLE> DATA statistical purposes? </Title> </HEAD> </HTML> www.statistik.at Folie 2 | 10.03.2015
Web scraping Internet data collection – Minimum goal for (Price) Statistics: Turn website content into a spreadsheet www.statistik.at Folie 3 | 10.03.2015
Web scraping Internet data collection Options: 1. Manual price collection 2. Develop an API /Web scraper 2.1 by writing custom computer code 2.2 by using point and click web tools www.statistik.at Folie 4 | 10.03.2015
Web scraping Reasons for not writing an own web scraper IT-developer needed, therefore: • Expensive • Inflexible • Even maintenance cannot be handled by CPI staff www.statistik.at Folie 5 | 10.03.2015
Web scraping Reasons to use click and point webtools for web scraping: No IT-developer needed, therefore: • Cheap • Flexible • No programming skill required www.statistik.at Folie 6 | 10.03.2015
Web scraping How web scraping with click and point using import.io looks like: • web-platform that allows to structure and extract data from websites www.statistik.at Folie 7 | 10.03.2015
Webscraping with import.io www.statistik.at Folie 8 | 10.03.2015
Webscraping Web scraping with click and point on web- based platform offers solutions to: • extract data by point-and-click • record actions on a website • crawl all the data of a webpage More issues to be considered: • Legality to crawl on websites • Internal IT Security • Training of staff www.statistik.at Folie 9 | 10.03.2015
Automatic price Contact: Ingolf Boettcher collection on the internet Guglgasse 13, 1110 Wien Tel: +43 (1) 71128-7917 (web scraping) Fax: +43 (1) 7180718 Ingolf.boettcher@statistik.gv.at www.statistik.at Folie 10 | 10.03.2015

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Session 6A - Big data sources: web scraping and smart meters Using Internet as a Data Source for

NTTS 2015 Session 6A - Big data sources: web scraping and smart meters Using Internet as a Data Source for Official Statistics: a Comparative Analysis of Web Scraping Technologies Giulio Barcaroli(*) (barcarol@istat.it), Monica Scannapieco (*)

436 views • 12 slides

Web Scraping 1 / 9 Web Scraping Two ways to mine data from the web The hard way, by web

Web Scraping 1 / 9 Web Scraping Two ways to mine data from the web The hard way, by web scraping The easy way, using web service APIs Well see examples of both. 2 / 9 Web Scraping Web scraping, a.k.a. screen scraping, means getting

485 views • 9 slides

NTTS 2015 9 13 March 2015, Brussels Session 13C - Survey integration, coordination and

NTTS 2015 9 13 March 2015, Brussels Session 13C - Survey integration, coordination and alignment Avoiding duplicate collection of flow data: estimating intra-EU inbound tourism using partner data Christophe Demunter, Krista Dimitrakopoulou

697 views • 16 slides

Analysis of the Potential of Selected Big Data Repositories as Data Sources for Official

Analysis of the Potential of Selected Big Data Repositories as Data Sources for Official Statistics NTTS 2015 New Techniques and Technologies for Statistics Petrakos M., Santourian A., Farmakis G., Stavropoulos P., Oikonomopoulou G., Ntakou E.

221 views • 8 slides

A more visual dissemination for attracting new users NTTS 2015 Brussels Philippe BAUTIER,

A more visual dissemination for attracting new users NTTS 2015 Brussels Philippe BAUTIER, Bernard LE GOFF Session 5A The role of visualisation in data dissemination All the same challenge! How to build a dissemination and communication

491 views • 23 slides

video demo End-User Web Scraping: Google Scholar Edition Sarah Chasins data scraping tool

video demo End-User Web Scraping: Google Scholar Edition Sarah Chasins data scraping tool input demonstration of how to collect the first row of a relational dataset F r o m h i g h l y s t r u c t u r e d w e b p a

837 views • 63 slides

Katy Stokes Demographic Methods Centre, ONS NTTS Conference, March 2015 Uncertainty measures

Measuring uncertainty in ONS population estimates: capturing variability in statistics from combinations of census, administrative and survey sources Katy Stokes Demographic Methods Centre, ONS NTTS Conference, March 2015 Uncertainty measures

198 views • 8 slides

Using Web Scraped Data to Construct Consumer Price Indices Nigel Swier NTTS Conference, 10-12

Using Web Scraped Data to Construct Consumer Price Indices Nigel Swier NTTS Conference, 10-12 March 2015, Brussels Background One of 4 big data pilots in ONS Prices collection manually based Difficulties accessing retail

455 views • 11 slides

Mauro Michielon ICT Development NTTS conference, Brussels, 12 th March 2015 EEAs context EEA

Web tools for accessing and disseminating data of different formats Mauro Michielon ICT Development NTTS conference, Brussels, 12 th March 2015 EEAs context EEA needs to deal with a vast number of very heterogeneous datasets, coming from

478 views • 5 slides

from an integrated system of administrative microdata to an infrastructure for the users NTTS

Reversing the flow: from an integrated system of administrative microdata to an infrastructure for the users NTTS 2015 Session 6C - Enterprise architecture and integration Simone Ambroselli Istat ambrosel@istat.it Giuseppe Garofalo Istat

446 views • 10 slides

The Office for National Statistics Big Data Project Nigel Swier NTTS Conference, 10-12 March

The Office for National Statistics Big Data Project Nigel Swier NTTS Conference, 10-12 March 2015, Brussels Background Set up as a12 month project Jan-Dec 2014 Aims: Investigate the potential of big data for official statistics and

612 views • 12 slides

Modelling sample data from smart-type meter electricity usage Susan Williams NTTS Conference,

Modelling sample data from smart-type meter electricity usage Susan Williams NTTS Conference, 10-12 March 2015, Brussels Introduction Big Data Project in the UKs Office For National Statistics Practical pilot on the potential of

312 views • 12 slides

Web Scraping With P y thon W E B SC R AP IN G IN P YTH ON Thomas Laetsch Data Scientist , NYU B

Web Scraping With P y thon W E B SC R AP IN G IN P YTH ON Thomas Laetsch Data Scientist , NYU B u siness Sa vvy What are b u sinesses looking for ? Comparing prices Satisfaction of c u stomers Generating potential leads ... and m u ch more !

854 views • 29 slides

STATS 507 Data Analysis in Python Lecture 27: APIs Previously: Scraping Data from the Web We

STATS 507 Data Analysis in Python Lecture 27: APIs Previously: Scraping Data from the Web We used BeautifulSoup to process HTML that we read directly We had to figure out where to find the data in the HTML This was okay for simple things like

638 views • 26 slides

Scraping Distributed, Hierarchical Web Data with Programming by Demonstration! Sarah E.

Scraping Distributed, Hierarchical Web Data with Programming by Demonstration! Sarah E. Chasins 1 Maria Mueller 2 Rastislav Bodik 2 1 University of California, Berkeley 2 University of Washington The web: a rich source of data! 2008:

256 views • 24 slides

Data Sources; SCNL Data Sources Data sources producing waveform data can come from a remote

Data Sources; SCNL Data Sources Data sources producing waveform data can come from a remote source, via an import or a connection to a Seed Link server (ie: slink2ew), or from a local source, for example: a Quanterra digitizer (q3302ew) or

252 views • 22 slides

Digital Success for NonProfits Ellen Didier NONPROFIT UNIVERSITY President Sponsored by the

Digital Success for NonProfits Ellen Didier NONPROFIT UNIVERSITY President Sponsored by the Community Foundation June 22, 2017 www.RedSageOnline.com Your Digital Toolbox E-Newsletter Website Tools & Apps Social Media Your Website

581 views • 30 slides

Social Media What is Social Media? forms of electronic communication (such as Web sites) through

EDUCATE HONOR CHALLENGE Social Media What is Social Media? forms of electronic communication (such as Web sites) through which people create online communities to share information, ideas, personal messages, etc. Social Media Platforms

410 views • 14 slides

Albert Leung, Kaitao Wu, Hengjia Zhang, Richard Chen Elevator Pitch For social media managers who

Albert Leung, Kaitao Wu, Hengjia Zhang, Richard Chen Elevator Pitch For social media managers who are dissatisfied with the inconvenience of updating their followers across many platforms, Our product is a social media posting web application,

757 views • 13 slides

S P H E R A Controlling networks while maintaining resilience Baruch Barzel 1 Challenges 5.5

S P H E R A Controlling networks while maintaining resilience Baruch Barzel 1 Challenges 5.5 10 7 People affected 10 2 Fatalities 6 10 9 USD in damages 2 Structure vs. dynamics Structural perturbation (component failure) Dynamic

375 views • 20 slides

ANGLO AMERICAN PLATINUM CORPORATION FACILITY VISIT: Eastern Limb 4 March 2003 A member of the

World Leader in Platinum Facility Visit March 2003 ANGLO AMERICAN PLATINUM CORPORATION FACILITY VISIT: Eastern Limb 4 March 2003 A member of the Anglo American plc group World Leader in Platinum Facility Visit March 2003 TWICKENHAM

999 views • 40 slides

Project Plan Customer App Review Dashboard The Capstone Experience Team Ford Paul Friesen Noah

Project Plan Customer App Review Dashboard The Capstone Experience Team Ford Paul Friesen Noah Keppers Drew Morgan Jordan Schroeder Mingkai Yang Department of Computer Science and Engineering Michigan State University From Students

397 views • 13 slides

A Presentation A Presentation from from Caldervale Technology Caldervale Technology Ltd Ltd

A Presentation A Presentation from from Caldervale Technology Caldervale Technology Ltd Ltd www.caldertech.com www.caldertech.com www.bestpracticetooling.com 1 29/10/2012 The need for PE pipe The need for PE pipe surface preparation

956 views • 21 slides

simplifying the customer experience through account aggregation Sim Sangha Business Development

simplifying the customer experience through account aggregation Sim Sangha Business Development Director total account aggregation James Born 29th June 2007 What is account aggregation? Why is account aggregation important to your

605 views • 11 slides