VAEX: 1 BILLION ROWS, 1 LAPTOP, SERIOUS DATA SCIENCE JOVAN - PowerPoint PPT Presentation

Sep 19, 2022 •312 likes •365 views

VAEX: 1 BILLION ROWS, 1 LAPTOP, SERIOUS DATA SCIENCE JOVAN VELJANOSKI Sr. Data Scientist @ XebiaLabs UNCOMFORTABLY LARGE DATA Working with %i samples without going to the cloud: < 1_000_000 samples ~10_000_000 samples

VAEX: 1 BILLION ROWS, 1 LAPTOP, SERIOUS DATA SCIENCE JOVAN VELJANOSKI Sr. Data Scientist @ XebiaLabs
UNCOMFORTABLY LARGE DATA Working with %i samples without going to the cloud: ➡ < 1_000_000 samples ➡ ~10_000_000 samples ➡ ~100_000_000 samples ➡ ~1_000_000_000 samples ➡ larger datasets
VAEX.IO: WHO ARE WE? Maarten Breddels Jovan Veljanoski Yonatan Alexander Former astrophysicist Former astrophysicist Head of Data Science at BuiltOn � jonathan@xdss.io Freelancer / consultant / data scientist Sr. Data Scientist @ XebiaLabs Core Jupyter-Widgets developer � https://www.linkedin.com/in/xdssio/ Co-founder of vaex.io � jovan.veljanoski@gmail.com Founder of vaex.io Principal author of vaex � https://www.linkedin.com/in/jovanvel/ � maartenbreddels@gmail.com � www.maartenbreddels.com � @maartenbreddels � github.com/maartenbreddels Mario Buikhuizen Freelancer / consultant Front-end / dashboards / widgets specialist � mbuikhuizen@gmail.com
THE NEED FOR VAEX The Gaia satellite: More than 1 billion observations of stars in our Galaxy! How do we work (explore, filter, visualize, analyze) with such data?
LIVE DEMO The Jupyter notebooks presented at the live demo can be found at: https://github.com/vaexio/vaex-talks

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Vaex: Out of core dataframes for Python Maarten A. Breddels & Jovan Veljanoski Article:

Vaex: Out of core dataframes for Python Maarten A. Breddels & Jovan Veljanoski Article: A&A 618, 2017 / Arxiv 1801.02638 PyParis - Nov 13/2018 Maarten Breddels Ex: astronomer (working on software for big data and visualization: vaex)

1.15k views • 54 slides

The First Billion Rows Alexander Zaitsev and Robert Hodges About Us Robert Hodges - Altinity CEO

ClickHouse Data Warehouse 101 The First Billion Rows Alexander Zaitsev and Robert Hodges About Us Robert Hodges - Altinity CEO Alexander Zaitsev - Altinity CTO 30+ years on DBMS plus Expert in data warehouse with virtualization and security.

578 views • 47 slides

1 Tabulation means putting data into tables. A table is a matrix of data in rows and

1 Tabulation means putting data into tables. A table is a matrix of data in rows and columns, with the rows and the columns having titles. 2 converting the set of numbers into the form of a grouped frequency table. This involves

833 views • 39 slides

Working with tidy data in R: dplyr Fundamental actions on data tables: choose rows

Working with tidy data in R: dplyr Fundamental actions on data tables: choose rows filter() choose columns select() make new columns mutate() arrange rows arrange() calculate summary statistics summarize()

341 views • 14 slides

Working with tidy data in R: tidyverse Fundamental actions on data tables: choose rows

Working with tidy data in R: tidyverse Fundamental actions on data tables: choose rows filter() choose columns select() make new columns mutate() arrange rows arrange() calculate summary statistics

599 views • 20 slides

Welcome to BASH! 1:1 Laptop Deployment Log Into Your Laptop Reaching IT The Student Helpdesk at

Welcome to BASH! 1:1 Laptop Deployment Log Into Your Laptop Reaching IT The Student Helpdesk at BASH is your first resource for any issues that you may have with your 1:1 laptop. We can be reached in a myriad of ways: Stopping by FLEX

391 views • 14 slides

Variable Shara Auty Problem Insurance data contains multiple rows of transactions per policy

Application of the INTO: Host - Variable Shara Auty Problem Insurance data contains multiple rows of transactions per policy holder. It is necessary to summarize the data or to view the data in a different manner. I needed to summarize

226 views • 18 slides

Research Data Managem ent: life beyond the laptop Library and Learning Services conference

Research Data Managem ent: life beyond the laptop Library and Learning Services conference Sunley Conference Centre 15 th June 2012 Miggie Pickton Outline Research data management drivers at Northampton Current RDM practice

510 views • 15 slides

One Laptop per Child One Laptop per Child USENIX: June 22, 2007 Mary Lou Jepsen This works are

One Laptop per Child One Laptop per Child USENIX: June 22, 2007 Mary Lou Jepsen This works are licensed under a Creative Commons Attribution 2.5 License. Acknowledgement Carla Gomez Monroy + all at OLPC + thousands of other organizations and

445 views • 43 slides

Fast and Secure H T Y T O H Laptop Backups F G R E U D B I N with Encrypted

I V N E U R S E I Fast and Secure H T Y T O H Laptop Backups F G R E U D B I N with Encrypted De-duplication Le Zhang <zhang.le@ed.ac.uk> Paul Anderson <dcspaul@ed.ac.uk> LISA 2010 Laptop Backup Options

390 views • 18 slides

Why IoT IoT Domain IoT Data Characteristics Massive data: 20.4 Billion connected Growing

Why IoT IoT Domain IoT Data Characteristics Massive data: 20.4 Billion connected Growing fast, impact on IoT Visualization our life thing by 2020 (Data Volume) Industries are putting e ff ort: Real-time integration of devices

146 views • 3 slides

About My New Laptop I ntroducing: HP OMEN 15-dc0051nr 15.6" Gaming Laptop Computer The Reason

About My New Laptop I ntroducing: HP OMEN 15-dc0051nr 15.6" Gaming Laptop Computer The Reason for Purchasing Performance Price/Value From Microcenter Website Overview of Specs From Microcenter Website More Spec Details: HP Omen 15

349 views • 30 slides

USING OBS STUDIO RECORD YOUR PRESENTATION WITH YOUR VIDEO IN LAPTOP USING OBS STUDIO What do you

RECORD YOUR PRESENTATION WITH YOUR VIDEO IN LAPTOP USING OBS STUDIO RECORD YOUR PRESENTATION WITH YOUR VIDEO IN LAPTOP USING OBS STUDIO What do you need PowerPoint Slide Laptop with Builtin Microphone & Camera You can either use the

452 views • 12 slides

One Laptop Per Child Exploring, Sharing, Reflecting, Learning Presentation at: Mona

One Laptop Per Child Exploring, Sharing, Reflecting, Learning Presentation at: Mona School of Business The University of the West Indies Sameer Verma, Ph.D. Mona, Jamaica sverma@sfsu.edu Outline Introduction Education Laptop

560 views • 36 slides

Modeling Highly Heterogeneous (Large) Data Sets: Towards a Billion Models Robert Grossman

Modeling Highly Heterogeneous (Large) Data Sets: Towards a Billion Models Robert Grossman University of Illinois at Chicago & Open Data Group Traditional Statistics One small data set A few attributes Vector-valued data Data

453 views • 18 slides

Understanding the CARES Act Programs for Employers The CARES Act $250 billion in $260 billion

April 1, 2020 Understanding the CARES Act Programs for Employers The CARES Act $250 billion in $260 billion in cash payments unemployment to families and insurance individuals benefits $377 billion in small business loans $500 billion

599 views • 29 slides

Raising AI: Tutoring Matters Jordi Bieger 1 (jbieger@gmail.com), Kristinn R. Thrisson 1,2 &

Raising AI: Tutoring Matters Jordi Bieger 1 (jbieger@gmail.com), Kristinn R. Thrisson 1,2 & Deon Garrett 2 1 Reykjavik University | Center for Analysis & Design of Intelligent Agents 2 Icelandic Institute for Intelligent Machines Raising

342 views • 19 slides

ZIVD, LLC 1 Laboratory Optimized patient care Clinician Optimized patient care 2

Marcia L. Zucker, Ph.D. ZIVD, LLC 1 Laboratory Optimized patient care Clinician Optimized patient care 2 Laboratory Clinician Accuracy Speed High Quality Precision Minimal disruption of Test results routine Compliance

316 views • 20 slides

GR committee J. Orduna, Chair Louise Suter, Deputy Chair Important links and announcements DC

GR committee J. Orduna, Chair Louise Suter, Deputy Chair Important links and announcements DC Trip dates March 16-18, 2016 (Wed. to Fri.) Planning meeting: January 22, 2016. UEC Meeting during the morning followed by planning meeting. The

612 views • 10 slides

Transactional and Experiential Law Teaching Trends and Challenges in the Asia Pacific Region

Transactional and Experiential Law Teaching Trends and Challenges in the Asia Pacific Region Law and Society Conference 2016 Andrew Godwin Director, Transactional Law Melbourne Law School www.law.unimelb.edu.au Outline of themes and

256 views • 6 slides

adequate medical care to asylum seekers in PNG and Nauru Our Work We use the law to eradicate

The Commonwealths duty to provide adequate medical care to asylum seekers in PNG and Nauru Our Work We use the law to eradicate institutional racism and we empower vulnerable communities to create a fairer society Where Can You Follow Us?

406 views • 36 slides

Reproducibility as a Community Effort Lessons from the Madagascar Project Sergey Fomel Jackson

Reproducibility as a Community Effort Lessons from the Madagascar Project Sergey Fomel Jackson School of Geosciences The University of Texas at Austin 12/13/2012 ICERM Reproducibility in Computational and Experimental Mathematics What is

314 views • 16 slides

Reproducible research in practice M ADAGASCAR software package Sergey Fomel Jackson School of

Reproducible Research M ADAGASCAR Project Reproducible research in practice M ADAGASCAR software package Sergey Fomel Jackson School of Geosciences The University of Texas at Austin July 1, 2010 S. Fomel SciPy 2010 Reproducible Research M

431 views • 16 slides

Service Authority Janice R. Carr, Member Joyce P. Eagles, Member Ernie Gonzales, Member David A.

4 County Complex Court Phone(703)335-7900 P. 0. Box 2266 Fax (703) 335-7905 www.pwcsa.org Woodbridge, VA 22195-2266 Paul E. Ruecker, Chaim,an Dr. K. Jack Kooyoomjian, Vice Chairman Kurt E. Voggenreiter II, Secretary-Treasurer Mark Allen, Member

349 views • 6 slides