big data analytics
play

Big Data Analytics 3 rd NESUS Winter School on Data Science & - PowerPoint PPT Presentation

Feb 11, 2024 •167 likes •2.07k views

http://nesusws.irb.hr/ Big Data Analytics 3 rd NESUS Winter School on Data Science & Heterogeneous Computing Sbastien Varrette, PhD Parallel Computing and Optimization Group (PCOG), University of Luxembourg (UL), Luxembourg

  1. Introduction Pulling and Running a Vagrant Box $> vagrant up # boot the box(es) set in the Vagrantfile Base box is downloaded and stored locally ~/.vagrant.d/boxes/ A new VM is created and configured with the base box as template → The VM is booted and (eventually) provisioned ֒ → Once within the box: /vagrant = directory hosting Vagrantfile ֒ $> vagrant status # State of the vagrant box(es) $> vagrant ssh # connect inside it, CTRL-D to exit Sebastien Varrette (University of Luxembourg) Big Data Analytics 15 / 133 �

  2. Introduction Stopping Vagrant Box $> vagrant { destroy | halt } # destroy / halt Once you have finished your work within a running box → save the state for later with vagrant halt ֒ → reset changes / tests / errors with vagrant destroy ֒ → commit changes by generating a new version of the box ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 16 / 133 �

  3. Introduction Hands-on 0: Vagrant This tutorial heavily relies on Vagrant → you will need to familiarize with the tool if not yet done ֒ Your Turn! Hands-on 0 http://nesusws-tutorials-BD-DL.rtfd.io/en/latest/hands-on/vagrant/ Clone the tutorial repository Step 1 Basic Usage of Vagrant Step 2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 17 / 133 �

  4. Introduction Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 18 / 133 �

  5. Introduction Why HPC and BD ? HPC : H igh P erformance C omputing BD : B ig D ata Andy Grant, Head of Big Data and HPC, Atos UK&I To out-compete you must out-compute Increasing competition, heightened customer expectations and shortening product development cycles are forcing the pace of acceleration across all industries. Sebastien Varrette (University of Luxembourg) Big Data Analytics 19 / 133 �

  6. Introduction Why HPC and BD ? HPC : H igh P erformance C omputing BD : B ig D ata Essential tools for Science, Society and Industry → All scientific disciplines are becoming computational today ֒ � requires very high computing power, handles huge volumes of data Industry, SMEs increasingly relying on HPC → to invent innovative solutions ֒ → . . . while reducing cost & decreasing time to market ֒ Andy Grant, Head of Big Data and HPC, Atos UK&I To out-compete you must out-compute Increasing competition, heightened customer expectations and shortening product development cycles are forcing the pace of acceleration across all industries. Sebastien Varrette (University of Luxembourg) Big Data Analytics 19 / 133 �

  7. Introduction Why HPC and BD ? HPC : H igh P erformance C omputing BD : B ig D ata Essential tools for Science, Society and Industry → All scientific disciplines are becoming computational today ֒ � requires very high computing power, handles huge volumes of data Industry, SMEs increasingly relying on HPC → to invent innovative solutions ֒ → . . . while reducing cost & decreasing time to market ֒ HPC = global race (strategic priority) - EU takes up the challenge: → EuroHPC / IPCEI on HPC and Big Data (BD) Applications ֒ Andy Grant, Head of Big Data and HPC, Atos UK&I To out-compete you must out-compute Increasing competition, heightened customer expectations and shortening product development cycles are forcing the pace of acceleration across all industries. Sebastien Varrette (University of Luxembourg) Big Data Analytics 19 / 133 �

  8. Introduction New Trends in HPC Continued scaling of scientific, industrial & financial applications → . . . well beyond Exascale ֒ F ��������� ��� � C ����� �� E ��������� �� H ��� -P ���������� C �������� S ������ New trends changing the landscape for HPC → Emergence of Big Data analytics ֒ → Emergence of ( Hyperscale ) Cloud Computing ֒ → Data intensive Internet of Things (IoT) applications ֒ → Deep learning & cognitive computing paradigms ֒ Eurolab-4-HPC Long-Term Vision on High-Performance Computing This study was carried out for RIKEN by Editors: Theo Ungerer, Paul Carpenter Funded by the European Union Horizon 2020 Framework Programme (H2020-EU.1.2.2. - FET Proactive) [Source : EuroLab-4-HPC] Special Study Analysis of the Characteristics and Development Trends of the Next-Generation of Supercomputers in Foreign Countries Earl C. Joseph, Ph.D. Robert Sorensen Steve Conway Kevin Monroe [Source : IDC RIKEN report, 2016] Sebastien Varrette (University of Luxembourg) Big Data Analytics 20 / 133 � � � �

  9. Introduction Toward Modular Computing Aiming at scalable , flexible HPC infrastructures → Primary processing on CPUs and accelerators ֒ � HPC & Extreme Scale Booster modules → Specialized modules for: ֒ � HTC & I/O intensive workloads; � [Big] Data Analytics & AI [Source : "Towards Modular Supercomputing: The DEEP and DEEP-ER projects", 2016] Sebastien Varrette (University of Luxembourg) Big Data Analytics 21 / 133 �

  10. Introduction Prerequisites: Metrics HPC : H igh P erformance C omputing BD : B ig D ata Main HPC/BD Performance Metrics Computing Capacity : often measured in flops (or flop/s ) → Fl oating p oint o perations per s econds ֒ (often in DP) → GFlops = 10 9 TFlops = 10 12 PFlops = 10 15 EFlops = 10 18 ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 22 / 133 �

  11. Introduction Prerequisites: Metrics HPC : H igh P erformance C omputing BD : B ig D ata Main HPC/BD Performance Metrics Computing Capacity : often measured in flops (or flop/s ) → Fl oating p oint o perations per s econds ֒ (often in DP) → GFlops = 10 9 TFlops = 10 12 PFlops = 10 15 EFlops = 10 18 ֒ Storage Capacity : measured in multiples of bytes = 8 bits → GB = 10 9 bytes PB = 10 15 EB = 10 18 TB = 10 12 ֒ → GiB = 1024 3 bytes PiB = 1024 5 EiB = 1024 6 TiB = 1024 4 ֒ Transfer rate on a medium measured in Mb/s or MB/s Other metrics: Sequential vs Random R/W speed , IOPS . . . Sebastien Varrette (University of Luxembourg) Big Data Analytics 22 / 133 �

  12. Introduction Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 23 / 133 �

  13. Introduction HPC Components: [GP]CPU CPU Always multi-core Ex: Intel Core i7-7700K (Jan 2017) R peak ≃ 268.8 GFlops (DP) → 4 cores @ 4.2GHz (14nm, 91W, 1.75 billion transistors) ֒ → + integrated graphics (24 EUs) R peak ≃ +441.6 GFlops ֒ GPU / GPGPU Always multi-core, optimized for vector processing Ex: Nvidia Tesla V100 (Jun 2017) R peak ≃ 7 TFlops (DP) → 5120 cores @ 1.3GHz (12nm, 250W, 21 billion transistors) ֒ → focus on Deep Learning workloads R peak ≃ 112 TFLOPS (HP) ֒ ≃ 100 Gflops for 130$ (CPU), 214$? (GPU) Sebastien Varrette (University of Luxembourg) Big Data Analytics 24 / 133 �

  14. Introduction HPC Components: Local Memory Larger, slower and cheaper L1 L2 L3 - - - CPU Memory Bus I/O Bus C C C a a a Memory c c c h h h Registers e e e L1-cache L2-cache L3-cache register (SRAM) (SRAM) (DRAM) Memory (DRAM) reference Disk memory reference reference reference reference reference Level: 1 4 2 3 Size: 500 bytes 64 KB to 8 MB 1 GB 1 TB Speed: sub ns 1-2 cycles 10 cycles 20 cycles hundreds cycles ten of thousands cycles SSD (SATA3) R/W: 550 MB/s; 100000 IOPS 450 e /TB HDD (SATA3 @ 7,2 krpm) R/W: 227 MB/s; 85 IOPS 54 e /TB Sebastien Varrette (University of Luxembourg) Big Data Analytics 25 / 133 �

  15. Introduction HPC Components: Interconnect latency : time to send a minimal (0 byte) message from A to B bandwidth : max amount of data communicated per unit of time Technology Effective Bandwidth Latency Gigabit Ethernet 1 Gb/s 125 MB/s 40 µ s to 300 µ s 10 Gigabit Ethernet 10 Gb/s 1.25 GB/s 4 µ s to 5 µ s Infiniband QDR 40 Gb/s 5 GB/s 1 . 29 µ s to 2 . 6 µ s Infiniband EDR 100 Gb/s 12.5 GB/s 0 . 61 µ s to 1 . 3 µ s 100 Gigabit Ethernet 100 Gb/s 1.25 GB/s 30 µ s Intel Omnipath 100 Gb/s 12.5 GB/s 0 . 9 µ s Infiniband 32.6 % [Source : www.top500.org , Nov. 2017] 1.4 % 40.8 % 4.8 % Proprietary 10G 7 % Gigabit Ethernet 13.4 % Omnipath Custom Sebastien Varrette (University of Luxembourg) Big Data Analytics 26 / 133 �

  16. Introduction HPC Components: Interconnect latency : time to send a minimal (0 byte) message from A to B bandwidth : max amount of data communicated per unit of time Technology Effective Bandwidth Latency Gigabit Ethernet 1 Gb/s 125 MB/s 40 µ s to 300 µ s 10 Gigabit Ethernet 10 Gb/s 1.25 GB/s 4 µ s to 5 µ s Infiniband QDR 40 Gb/s 5 GB/s 1 . 29 µ s to 2 . 6 µ s Infiniband EDR 100 Gb/s 12.5 GB/s 0 . 61 µ s to 1 . 3 µ s 100 Gigabit Ethernet 100 Gb/s 1.25 GB/s 30 µ s Intel Omnipath 100 Gb/s 12.5 GB/s 0 . 9 µ s Infiniband 32.6 % [Source : www.top500.org , Nov. 2017] 1.4 % 40.8 % 4.8 % Proprietary 10G 7 % Gigabit Ethernet 13.4 % Omnipath Custom Sebastien Varrette (University of Luxembourg) Big Data Analytics 26 / 133 �

  17. Introduction Network Topologies Direct vs. Indirect interconnect → direct : each network node attaches to at least one compute node ֒ → indirect : compute nodes attached at the edge of the network only ֒ � many routers only connect to other routers. Sebastien Varrette (University of Luxembourg) Big Data Analytics 27 / 133 �

  18. Introduction Network Topologies Direct vs. Indirect interconnect → direct : each network node attaches to at least one compute node ֒ → indirect : compute nodes attached at the edge of the network only ֒ � many routers only connect to other routers. Main HPC Topologies CLOS Network / Fat-Trees [Indirect] → can be fully non-blocking (1:1) or blocking (x:1) ֒ → typically enables best performance ֒ � Non blocking bandwidth, lowest network latency Sebastien Varrette (University of Luxembourg) Big Data Analytics 27 / 133 �

  19. Introduction Network Topologies Direct vs. Indirect interconnect → direct : each network node attaches to at least one compute node ֒ → indirect : compute nodes attached at the edge of the network only ֒ � many routers only connect to other routers. Main HPC Topologies CLOS Network / Fat-Trees [Indirect] → can be fully non-blocking (1:1) or blocking (x:1) ֒ → typically enables best performance ֒ � Non blocking bandwidth, lowest network latency Mesh or 3D-torus [Direct] → Blocking network, cost-effective for systems at scale ֒ → Great performance solutions for applications with locality ֒ → Simple expansion for future growth ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 27 / 133 �

  20. Introduction HPC Components: Operating System Exclusively Linux-based ( really 100%) Reasons: → stability ֒ → prone to devels ֒ [Source : www.top500.org , Nov 2017] Linux 100 % Sebastien Varrette (University of Luxembourg) Big Data Analytics 28 / 133 �

  21. Introduction [Big]Data Management Storage architectural classes & I/O layers Application [Distributed] File system Network Network SATA NFS SAS iSCSI CIFS FC ... AFP ... ... DAS Interface SAN Interface NAS Interface Fiber Ethernet/ Fiber Ethernet/ Channel Network DAS Channel Network SATA SAN SAS File System NAS Fiber Channel SATA SAS SATA Fiber Channel SAS Fiber Channel Sebastien Varrette (University of Luxembourg) Big Data Analytics 29 / 133 �

  22. Introduction [Big]Data Management: Disk Encl. ≃ 120 K e - enclosure - 48-60 disks (4U) → incl. redundant ( i.e. 2) RAID controllers (master/slave) ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 30 / 133 �

  23. Introduction [Big]Data Management: File Systems File System (FS) Logical manner to store , organize , manipulate & access data Sebastien Varrette (University of Luxembourg) Big Data Analytics 31 / 133 �

  24. Introduction [Big]Data Management: File Systems File System (FS) Logical manner to store , organize , manipulate & access data (local) Disk FS : FAT32 , NTFS , HFS+ , ext{3,4} , {x,z,btr}fs . . . → manage data on permanent storage devices ֒ → poor perf. read : 100 → 400 MB/s | write : 10 → 200 MB/s ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 31 / 133 �

  25. Introduction [Big]Data Management: File Systems Networked FS : NFS , CIFS / SMB , AFP → disk access from remote nodes via network access ֒ → poorer performance for HPC jobs especially parallel I/O ֒ � read : only 381 MB/s on a system capable of 740MB/s (16 tasks) � write : only 90MB/s on system capable of 400MB/s (4 tasks) [Source : LISA’09] Ray Paden: How to Build a Petabyte Sized Storage System Sebastien Varrette (University of Luxembourg) Big Data Analytics 32 / 133 �

  26. Introduction [Big]Data Management: File Systems Networked FS : NFS , CIFS / SMB , AFP → disk access from remote nodes via network access ֒ → poorer performance for HPC jobs especially parallel I/O ֒ � read : only 381 MB/s on a system capable of 740MB/s (16 tasks) � write : only 90MB/s on system capable of 400MB/s (4 tasks) [Source : LISA’09] Ray Paden: How to Build a Petabyte Sized Storage System [scale-out] NAS → aka Appliances OneFS . . . ֒ → Focus on CIFS, NFS ֒ → Integrated HW/SW ֒ → Ex : EMC (Isilon) , IBM ֒ (SONAS), DDN . . . Sebastien Varrette (University of Luxembourg) Big Data Analytics 32 / 133 �

  27. Introduction [Big]Data Management: File Systems Basic Clustered FS GPFS → File access is parallel ֒ → File System overhead operations is distributed and done in parallel ֒ � no metadata servers → File clients access file data through file servers via the LAN ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 33 / 133 �

  28. Introduction [Big]Data Management: File Systems Multi-Component Clustered FS Lustre, Panasas → File access is parallel ֒ → File System overhead operations on dedicated components ֒ � metadata server (Lustre) or director blades (Panasas) → Multi-component architecture ֒ → File clients access file data through file servers via the LAN ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 34 / 133 �

  29. Introduction [Big]Data Management: FS Summary File System (FS) : Logical manner to store, organize & access data → (local) Disk FS : FAT32 , NTFS , HFS+ , ext4 , {x,z,btr}fs . . . ֒ → Networked FS : NFS , CIFS / SMB , AFP ֒ → Parallel/Distributed FS : SpectrumScale/GPFS , Lustre ֒ � typical FS for HPC / HTC (High Throughput Computing) Sebastien Varrette (University of Luxembourg) Big Data Analytics 35 / 133 �

  30. Introduction [Big]Data Management: FS Summary File System (FS) : Logical manner to store, organize & access data → (local) Disk FS : FAT32 , NTFS , HFS+ , ext4 , {x,z,btr}fs . . . ֒ → Networked FS : NFS , CIFS / SMB , AFP ֒ → Parallel/Distributed FS : SpectrumScale/GPFS , Lustre ֒ � typical FS for HPC / HTC (High Throughput Computing) Main Characteristic of Parallel/Distributed File Systems Capacity and Performance increase with #servers Sebastien Varrette (University of Luxembourg) Big Data Analytics 35 / 133 �

  31. Introduction [Big]Data Management: FS Summary File System (FS) : Logical manner to store, organize & access data → (local) Disk FS : FAT32 , NTFS , HFS+ , ext4 , {x,z,btr}fs . . . ֒ → Networked FS : NFS , CIFS / SMB , AFP ֒ → Parallel/Distributed FS : SpectrumScale/GPFS , Lustre ֒ � typical FS for HPC / HTC (High Throughput Computing) Main Characteristic of Parallel/Distributed File Systems Capacity and Performance increase with #servers Name Type Read* [GB/s] Write* [GB/s] ext4 Disk FS 0.426 0.212 nfs Networked FS 0.381 0.090 gpfs (iris) Parallel/Distributed FS 10.14 8,41 gpfs (gaia) Parallel/Distributed FS 7.74 6.524 lustre Parallel/Distributed FS 4.5 2.956 ∗ maximum random read/write, per IOZone or IOR measures, using 15 concurrent nodes for networked FS. Sebastien Varrette (University of Luxembourg) Big Data Analytics 35 / 133 �

  32. Introduction HPC Components: Data Center Definition (Data Center) Facility to house computer systems and associated components → Basic storage component: rack (height: 42 RU) ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 36 / 133 �

  33. Introduction HPC Components: Data Center Definition (Data Center) Facility to house computer systems and associated components → Basic storage component: rack (height: 42 RU) ֒ Challenges: Power (UPS, battery) , Cooling, Fire protection, Security Power/Heat dissipation per rack: Power Usage Effectiveness → HPC computing racks: 30-120 kW ֒ → Storage racks: 15 kW PUE = Total facility power ֒ Interconnect racks: → 5 kW IT equipment power ֒ Various Cooling Technology → Airflow ֒ → Direct-Liquid Cooling, Immersion... ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 36 / 133 �

  34. Interlude: Software Management in HPC systems Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 37 / 133 �

  35. Interlude: Software Management in HPC systems Software/Modules Management https://hpc.uni.lu/users/software/ Based on Environment Modules / LMod → convenient way to dynamically change the users environment $PATH ֒ → permits to easily load software through module command ֒ Currently on UL HPC: → > 163 software packages , in multiple versions, within 18 categ. ֒ → reworked software set for iris cluster and now deployed everywhere ֒ � RESIF v2.0, allowing [real] semantic versioning of released builds → hierarchical organization Ex : toolchain/{foss,intel} ֒ $> module avail # List available modules $> module load <category>/<software>[/<version>] Sebastien Varrette (University of Luxembourg) Big Data Analytics 38 / 133 �

  36. Interlude: Software Management in HPC systems Software/Modules Management Key module variable: $MODULEPATH / where to look for modules → altered with module use <path> . Ex : ֒ export EASYBUILD_PREFIX=$HOME/.local/easybuild export LOCAL_MODULES=$EASYBUILD_PREFIX/modules/all module use $LOCAL_MODULES Sebastien Varrette (University of Luxembourg) Big Data Analytics 39 / 133 �

  37. Interlude: Software Management in HPC systems Software/Modules Management Key module variable: $MODULEPATH / where to look for modules → altered with module use <path> . Ex : ֒ export EASYBUILD_PREFIX=$HOME/.local/easybuild export LOCAL_MODULES=$EASYBUILD_PREFIX/modules/all module use $LOCAL_MODULES Main modules commands : Command Description module avail Lists all the modules which are available to be loaded module spider <pattern> Search for among available modules (Lmod only) module load <mod1> [mod2...] Load a module module unload <module> Unload a module module list List loaded modules module purge Unload all modules (purge) module display <module> Display what a module does module use <path> Prepend the directory to the MODULEPATH environment variable module unuse <path> Remove the directory from the MODULEPATH environment variable Sebastien Varrette (University of Luxembourg) Big Data Analytics 39 / 133 �

  38. Interlude: Software Management in HPC systems Software/Modules Management http://hpcugent.github.io/easybuild/ Easybuild: open-source framework to (automatically) build scientific software Why? : "Could you please install this software on the cluster?" → Scientific software is often difficult to build ֒ � non-standard build tools / incomplete build procedures � hardcoded parameters and/or poor/outdated documentation → EasyBuild helps to facilitate this task ֒ � consistent software build and installation framework � includes testing step that helps validate builds � automatically generates LMod modulefiles $ > module use $LOCAL_MODULES $ > module load tools/EasyBuild $ > eb -S HPL # Search for recipes for HPL software $ > eb HPL-2.2-intel-2017a.eb # Install HPL 2.2 w. Intel toolchain Sebastien Varrette (University of Luxembourg) Big Data Analytics 40 / 133 �

  39. Interlude: Software Management in HPC systems Hands-on 1: Modules & Easybuild Your Turn! Hands-on 1 http://nesusws-tutorials-BD-DL.rtfd.io/en/latest/hands-on/easybuild/ Discover Environment Modules and Lmod Part 1 Installation of EasyBuild Part 2 (a) Local vs. Global Usage Part 2 (b) → local installation of zlib ֒ → global installation of snappy and protobuf, needed later ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 41 / 133 �

  40. Interlude: Software Management in HPC systems Hands-on 2: Building Hadoop We will need to install the Hadoop MapReduce by Cloudera using EasyBuild. → this build is quite long ( ~30 minutes on 4 cores ) ֒ → Obj : make it build while the keynote continues ;) ֒ Hands-on 2 http://nesusws-tutorials-BD-DL.rtfd.io/en/latest/hands-on/hadoop/install/ Pre-requisites Step 1 → Installing Java 1.7.0 (7u80) and 1.8.0 (8u152) Step 1.a ֒ → Installing Maven 3.5.2 Step 1.b ֒ Installing Hadoop 2.6.0-cdh5.12.0 Step 2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 42 / 133 �

  41. [Big] Data Management in HPC Environment: Overview and Challenges Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 43 / 133 �

  42. [Big] Data Management in HPC Environment: Overview and Challenges Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 44 / 133 �

  43. [Big] Data Management in HPC Environment: Overview and Challenges Data Intensive Computing Data volumes increasing massively → Clusters, storage capacity increasing massively ֒ Disk speeds are not keeping pace. Seek speeds even worse than read/write Sebastien Varrette (University of Luxembourg) Big Data Analytics 45 / 133 �

  44. [Big] Data Management in HPC Environment: Overview and Challenges Data Intensive Computing Data volumes increasing massively → Clusters, storage capacity increasing massively ֒ Disk speeds are not keeping pace. Seek speeds even worse than read/write Sebastien Varrette (University of Luxembourg) Big Data Analytics 45 / 133 �

  45. [Big] Data Management in HPC Environment: Overview and Challenges Speed Expectation on Data Transfer http://fasterdata.es.net/ How long to transfer 1 TB of data across various speed networks? Network Time 10 Mbps 300 hrs (12.5 days) 100 Mbps 30 hrs 1 Gbps 3 hrs 10 Gbps 20 minutes (Again) small I/Os really kill performances → Ex : transferring 80 TB for the backup of ecosystem_biology ֒ → same rack, 10Gb/s. 4 weeks − → 63TB transfer. . . ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 46 / 133 �

  46. [Big] Data Management in HPC Environment: Overview and Challenges Speed Expectation on Data Transfer http://fasterdata.es.net/ Sebastien Varrette (University of Luxembourg) Big Data Analytics 47 / 133 �

  47. [Big] Data Management in HPC Environment: Overview and Challenges Speed Expectation on Data Transfer http://fasterdata.es.net/ Sebastien Varrette (University of Luxembourg) Big Data Analytics 47 / 133 �

  48. [Big] Data Management in HPC Environment: Overview and Challenges Storage Performances: GPFS Sebastien Varrette (University of Luxembourg) Big Data Analytics 48 / 133 �

  49. [Big] Data Management in HPC Environment: Overview and Challenges Storage Performances: Lustre Sebastien Varrette (University of Luxembourg) Big Data Analytics 49 / 133 �

  50. [Big] Data Management in HPC Environment: Overview and Challenges Storage Performances Based on IOR or IOZone, reference I/O benchmarks Read → tests performed in 2013 ֒ 65536 32768 16384 I/O bandwidth (MiB/s) 8192 4096 2048 1024 512 SHM / Bigmem Lustre / Gaia 256 NFS / Gaia SSD / Gaia 128 Hard Disk / Chaos 64 0 5 10 15 Number of threads Sebastien Varrette (University of Luxembourg) Big Data Analytics 50 / 133 �

  51. [Big] Data Management in HPC Environment: Overview and Challenges Storage Performances Based on IOR or IOZone, reference I/O benchmarks Write → tests performed in 2013 ֒ 32768 16384 8192 I/O bandwidth (MiB/s) 4096 2048 1024 512 SHM / Bigmem Lustre / Gaia 256 NFS / Gaia 128 SSD / Gaia Hard Disk / Chaos 64 0 5 10 15 Number of threads Sebastien Varrette (University of Luxembourg) Big Data Analytics 50 / 133 �

  52. [Big] Data Management in HPC Environment: Overview and Challenges Understanding Your Storage Options Where can I store and manipulate my data? Shared storage → NFS - not scalable ~ ≃ 1.5 GB/s (R) O (100 TB) ֒ → GPFS - scalable ~~ ≃ 10 GB/s (R) O (1 PB) ֒ → Lustre - scalable ~~ ≃ 5 GB/s (R) O (0.5 PB) ֒ Local storage → local file system ( /tmp ) O (200 GB) ֒ � over HDD ≃ 100 MB/s, over SDD ≃ 400 MB/s → RAM ( /dev/shm ) ≃ 30 GB/s (R) O (20 GB) ֒ Distributed storage → HDFS, Ceph, GlusterFS - scalable ~~ ≃ 1 GB/s ֒ ⇒ In all cases: small I/Os really kill storage performances Sebastien Varrette (University of Luxembourg) Big Data Analytics 51 / 133 �

  53. [Big] Data Management in HPC Environment: Overview and Challenges Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 52 / 133 �

  54. [Big] Data Management in HPC Environment: Overview and Challenges Data Transfer in Practice $> wget [-O <output>] <url> # download file from <url> $> curl [-o <output>] <url> # download file from <url> Transfer from FTP/HTTP[S] wget or (better) curl → can also serve to send HTTP POST requests ֒ → support HTTP cookies (useful for JDK download) ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 53 / 133 �

  55. [Big] Data Management in HPC Environment: Overview and Challenges Data Transfer in Practice $> scp [-P <port>] <src> <user>@<host>:<path> $> rsync -avzu [-e ’ssh -p <port>’] <src> <user>@<host>:<path> [Secure] Transfer from/to two remote machines over SSH → scp or (better) rsync (transfer only what is required) ֒ Assumes you have understood and configured appropriately SSH! Sebastien Varrette (University of Luxembourg) Big Data Analytics 54 / 133 �

  56. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Secure Shell Ensure secure connection to remote (UL) server → establish encrypted tunnel using asymmetric keys ֒ � Public id_rsa.pub vs. Private id_rsa ( without .pub ) � typically on a non-standard port ( Ex : 8022) limits kiddie script � Basic rule: 1 machine = 1 key pair → the private key is SECRET : never send it to anybody ֒ � Can be protected with a passphrase Sebastien Varrette (University of Luxembourg) Big Data Analytics 55 / 133 �

  57. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Secure Shell Ensure secure connection to remote (UL) server → establish encrypted tunnel using asymmetric keys ֒ � Public id_rsa.pub vs. Private id_rsa ( without .pub ) � typically on a non-standard port ( Ex : 8022) limits kiddie script � Basic rule: 1 machine = 1 key pair → the private key is SECRET : never send it to anybody ֒ � Can be protected with a passphrase SSH is used as a secure backbone channel for many tools → Remote shell i.e remote command line ֒ → File transfer: rsync , scp , sftp ֒ → versionning synchronization ( svn , git ), github, gitlab etc. ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 55 / 133 �

  58. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Secure Shell Ensure secure connection to remote (UL) server → establish encrypted tunnel using asymmetric keys ֒ � Public id_rsa.pub vs. Private id_rsa ( without .pub ) � typically on a non-standard port ( Ex : 8022) limits kiddie script � Basic rule: 1 machine = 1 key pair → the private key is SECRET : never send it to anybody ֒ � Can be protected with a passphrase SSH is used as a secure backbone channel for many tools → Remote shell i.e remote command line ֒ → File transfer: rsync , scp , sftp ֒ → versionning synchronization ( svn , git ), github, gitlab etc. ֒ Authentication: → password (disable if possible) ֒ → ( better ) public key authentication ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 55 / 133 �

  59. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Public Key Authentication Client Local Machine local homedir ~/.ssh/ owns local private key id_rsa id_rsa .pub logs known servers known_hosts Sebastien Varrette (University of Luxembourg) Big Data Analytics 56 / 133 �

  60. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Public Key Authentication Client Server Local Machine Remote Machine local homedir remote homedir ~/.ssh/ ~/.ssh/ knows granted owns local private key authorized_keys id_rsa (public) key id_rsa .pub logs known servers known_hosts Sebastien Varrette (University of Luxembourg) Big Data Analytics 56 / 133 �

  61. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Public Key Authentication Client Server Local Machine Remote Machine local homedir remote homedir ~/.ssh/ ~/.ssh/ knows granted owns local private key authorized_keys id_rsa (public) key id_rsa .pub SSH server config /etc/ssh/ sshd_config logs known servers known_hosts ssh_host_rsa_key ssh_host_rsa_key .pub Sebastien Varrette (University of Luxembourg) Big Data Analytics 56 / 133 �

  62. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Public Key Authentication Client Server Local Machine Remote Machine local homedir remote homedir ~/.ssh/ ~/.ssh/ knows granted owns local private key authorized_keys id_rsa (public) key id_rsa .pub Sebastien Varrette (University of Luxembourg) Big Data Analytics 56 / 133 �

  63. [Big] Data Management in HPC Environment: Overview and Challenges SSH: Public Key Authentication Client Server Local Machine Remote Machine local homedir remote homedir ~/.ssh/ ~/.ssh/ 1. Initiate connection knows granted owns local private key authorized_keys id_rsa (public) key 2. create random challenge, “encrypt” using public key id_rsa .pub 3. solve challenge using private key return response 4. allow connection iff response == challenge Restrict to public key authentication: /etc/ssh/sshd_config : PermitRootLogin no # Enable Public key auth. # Disable Passwords RSAAuthentication yes PubkeyAuthentication yes PasswordAuthentication no ChallengeResponseAuthentication no Sebastien Varrette (University of Luxembourg) Big Data Analytics 56 / 133 �

  64. [Big] Data Management in HPC Environment: Overview and Challenges Hands-on 3: Data transfer over SSH Before doing Big Data, learn how to transfer data between 2 hosts → do it securely over SSH ֒ # Quickly generate a 10GB file $ > dd if=/dev/zero of=/tmp/bigfile.txt bs=100M count=100 # Now try to transfert it between the 2 Vagrant boxes ;) Hands-on 3 http://nesusws-tutorials-BD-DL.rtfd.io/en/latest/hands-on/data-transfer/ Generate SSH Key Pair and authorize the public part Step 1 Data transfer over SSH with scp Step 2.a Data transfer over SSH with rsync Step 2.b Sebastien Varrette (University of Luxembourg) Big Data Analytics 57 / 133 �

  65. [Big] Data Management in HPC Environment: Overview and Challenges Summary 1 Introduction Before we start... Overview of HPC & BD Trends Main HPC and DB Components 2 Interlude: Software Management in HPC systems 3 [Big] Data Management in HPC Environment: Overview and Challenges Performance Overview in Data transfer Data transfer in practice Sharing Data 4 Big Data Analytics with Hadoop & Spark Apache Hadoop Apache Spark 5 Deep Learning Analytics with Tensorflow Sebastien Varrette (University of Luxembourg) Big Data Analytics 58 / 133 �

  66. [Big] Data Management in HPC Environment: Overview and Challenges Sharing Code and Data Before doing Big Data, manage and version correctly normal data What kinds of systems are available? Good : NAS, Cloud Dropbox, Google Drive, Figshare. . . Better - Version Control systems (VCS) → SVN, Git and Mercurial ֒ Best - Version Control Systems on the Public/Private Cloud → GitHub, Bitbucket, Gitlab ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 59 / 133 �

  67. [Big] Data Management in HPC Environment: Overview and Challenges Sharing Code and Data Before doing Big Data, manage and version correctly normal data What kinds of systems are available? Good : NAS, Cloud Dropbox, Google Drive, Figshare. . . Better - Version Control systems (VCS) → SVN, Git and Mercurial ֒ Best - Version Control Systems on the Public/Private Cloud → GitHub, Bitbucket, Gitlab ֒ Which one? → Depends on the level of privacy you expect ֒ � . . . but you probably already know these tools � → Few handle GB files. . . ֒ Sebastien Varrette (University of Luxembourg) Big Data Analytics 59 / 133 �

  68. [Big] Data Management in HPC Environment: Overview and Challenges Centralized VCS - CVS, SVN Computer A Central VCS Server Checkout Version Database File Version 3 Version 2 Version 1 Sebastien Varrette (University of Luxembourg) Big Data Analytics 60 / 133 �

  69. [Big] Data Management in HPC Environment: Overview and Challenges Centralized VCS - CVS, SVN Computer A Central VCS Server Checkout Version Database File Version 3 Version 2 Computer B Checkout Version 1 File Sebastien Varrette (University of Luxembourg) Big Data Analytics 60 / 133 �

  70. [Big] Data Management in HPC Environment: Overview and Challenges Distributed VCS - Git Server Computer Version Database Version 3 Computer A Computer B Version 2 File File Version 1 Version Database Version Database Version 3 Version 3 Version 2 Version 2 Version 1 Version 1 Everybody has the full history of commits Sebastien Varrette (University of Luxembourg) Big Data Analytics 61 / 133 �

  71. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (most VCS) Checkins over Time C1 file A file B file C Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  72. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (most VCS) Checkins over Time C1 C2 file A Δ 1 file B file C Δ 1 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  73. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (most VCS) Checkins over Time C1 C2 C3 file A Δ 1 file B file C Δ 1 Δ 2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  74. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (most VCS) Checkins over Time C1 C2 C3 C4 file A Δ 1 Δ 2 file B Δ 1 file C Δ 1 Δ 2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  75. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (most VCS) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  76. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (most VCS) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  77. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 snapshot (DAG) storage Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  78. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 snapshot A (DAG) storage B C Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  79. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 C2 snapshot A A1 (DAG) storage B B C C1 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  80. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 C2 snapshot A A1 (DAG) storage B B C C1 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  81. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 C2 C3 snapshot A A1 A1 (DAG) storage B B B C C1 C2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  82. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 C2 C3 snapshot A A1 A1 (DAG) storage B B B C C1 C2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  83. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 C2 C3 C4 snapshot A A1 A1 A2 (DAG) storage B B B B1 C C1 C2 C2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

  84. [Big] Data Management in HPC Environment: Overview and Challenges Tracking changes (Git) Checkins over Time C1 C2 C3 C4 C5 file A Δ 1 Δ 2 delta storage file B Δ 1 Δ 2 file C Δ 1 Δ 2 Δ 3 Checkins over Time C1 C2 C3 C4 snapshot A A1 A1 A2 (DAG) storage B B B B1 C C1 C2 C2 Sebastien Varrette (University of Luxembourg) Big Data Analytics 62 / 133 �

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Predictive Simulation &amp; Big Data Analytics ISD Analytics Predict a better future

Predictive Simulation &amp; Big Data Analytics ISD Analytics Predict a better future

Predictive Simulation &amp; Big Data Analytics ISD Analytics Predict a better future Overview Simulation can play a vital role in the emerging $billion field of Big Data analytics to support Government policy and business strategy

607 views • 31 slides
Analytics (9:55-10:15am) Break Research Opportunities in Location, Analytics, Big Data and GIS

Analytics (9:55-10:15am) Break Research Opportunities in Location, Analytics, Big Data and GIS

LOCATIONAL ANALYTICS AND BIG DATA: FOUNDATIONS, EMERGING APPLICATIONS AND RESEARCH Overview of Spatial Big Data and Analytics (9:10-9:55am) Applications and Examples of Spatial Big Data and Analytics (9:55-10:15am) Break Research

739 views • 24 slides
Data Mining &amp; Analytics Data Mining Reference Model Data Warehouse Legal and Ethical Issues

Data Mining &amp; Analytics Data Mining Reference Model Data Warehouse Legal and Ethical Issues

Data Mining &amp; Analytics Data Mining Reference Model Data Warehouse Legal and Ethical Issues Slides by Michael Hahsler Data Mining &amp; Analytics Analytics is the discovery and communication of meaningful patterns in data. Analytics

687 views • 19 slides
Google Analytics Overview Whats Google Analytics? The Google Analytics

Google Analytics Overview Whats Google Analytics? The Google Analytics

Google Analytics Overview Whats Google Analytics? The Google Analytics implementation formula. Steps involved in installing Google Analytics. Tracking events, campaigns and ecommerce data. Creating goals and funnels

652 views • 40 slides
Big Data Analytics Armistead Boyd SVP, Product &amp; Data Partnerships October 25, 2016 What is

Big Data Analytics Armistead Boyd SVP, Product &amp; Data Partnerships October 25, 2016 What is

Big Data Analytics Armistead Boyd SVP, Product &amp; Data Partnerships October 25, 2016 What is Big Data Analytics? Data Future Data Past Data Present 1 Big Data Trends Data Driven Organization Other Executives 5% 8% Operations Data

620 views • 22 slides
Maximizing the Value of Data Analytics for Operational Risk Intelligence Don't just do data

Maximizing the Value of Data Analytics for Operational Risk Intelligence Don't just do data

Maximizing the Value of Data Analytics for Operational Risk Intelligence Don't just do data analytics. Transform data analysis activities into ORM programs. Presented by: Sergiu Cernautan, CPA, CISA; Director of GRC Strategy, ACL Services, Ltd.

740 views • 36 slides
Data Analytics in Healthcare Health Data Analytics Conference October 2017 Dr Richard Ashby

Data Analytics in Healthcare Health Data Analytics Conference October 2017 Dr Richard Ashby

Data Analytics in Healthcare Health Data Analytics Conference October 2017 Dr Richard Ashby Chief Executive / Chief Information Officer eHealth Queensland Follow us on Twitter @eHealthQLD Data analytics in healthcare Advanced health

462 views • 28 slides
Richey Mays Data Analytics TYLER HOUSE TYLER@RICHEYMAY.COM Data Analytics Dashboards HMDA

Richey Mays Data Analytics TYLER HOUSE TYLER@RICHEYMAY.COM Data Analytics Dashboards HMDA

Richey Mays Data Analytics TYLER HOUSE TYLER@RICHEYMAY.COM Data Analytics Dashboards HMDA Market Share Dashboard Loan Origination Trends (from NMLS aggregated data) MBAs Mortgage Finance Forecast Accessible on our website:

134 views • 12 slides
Data Analytics CS301 Introduction to Data Analytics Week 1: 1 st Sept Fall 2020 Oliver

Data Analytics CS301 Introduction to Data Analytics Week 1: 1 st Sept Fall 2020 Oliver

Data Analytics CS301 Introduction to Data Analytics Week 1: 1 st Sept Fall 2020 Oliver BONHAM-CARTER Data Analytics CMPSC*301 Lect: T/TH: 11:10 am 12:25 pm Lab: F: 3:00 pm 4:50 pm Alden Hall 101, Questions? Contact Dr. BONHAM-CARTER

470 views • 30 slides
Modified and Shortened Introduction to Analytics Tools Data Models

Modified and Shortened Introduction to Analytics Tools Data Models

Modified and Shortened Introduction to Analytics Tools Data Models Problem solving with analytics Analytics is the use of: data, information technology, statistical analysis, quantitative methods, and

779 views • 42 slides
A Course in Data Discovery and Predictive Analytics 16 Nov 2013 A definition of business

A Course in Data Discovery and Predictive Analytics 16 Nov 2013 A definition of business

A Course in Data Discovery and Predictive Analytics 16 Nov 2013 A definition of business analytics David M. Levine, Baruch CollegeCUNY A Course in Broad categories of business analytics Kathryn A. Szabat, La Salle University Data

1.08k views • 94 slides
Analytics@TP Pre resen ented ed by: : Michael Yap 2018-09-28 Agenda Our Analytics

Analytics@TP Pre resen ented ed by: : Michael Yap 2018-09-28 Agenda Our Analytics

Analytics@TP Pre resen ented ed by: : Michael Yap 2018-09-28 Agenda Our Analytics Journey Capability Development Challenges Sample of Data Products Student Analytics Learning Analytics Graduate

1.03k views • 44 slides
Document Name Solar Analytics - Rooftop PV energy analytics PREPARED BY: Your Name, Your Title

Document Name Solar Analytics - Rooftop PV energy analytics PREPARED BY: Your Name, Your Title

Document Name Solar Analytics - Rooftop PV energy analytics PREPARED BY: Your Name, Your Title and fault diagnosis DATE: 6 TH MARCH 2015 Jonathon Dore Head of Data Analytics Solar Analytics 1 Outline About Solar Analytics Faults

417 views • 39 slides
THINGWORX ANALYTICS Name Title KEY TAKEAWAYS IoT Analytics Analytics is a journey that

THINGWORX ANALYTICS Name Title KEY TAKEAWAYS IoT Analytics Analytics is a journey that

INTRODUCTION TO THINGWORX ANALYTICS Name Title KEY TAKEAWAYS IoT Analytics Analytics is a journey that relies on a spectrum of capabilities Start your journey now, with the data you have, and gain more value over time 2

639 views • 16 slides
Data Analytics Xiaoling Huang, Data Analytics Director for Medicaid and CHIP August 2016

Data Analytics Xiaoling Huang, Data Analytics Director for Medicaid and CHIP August 2016

Data Analytics Xiaoling Huang, Data Analytics Director for Medicaid and CHIP August 2016 Medicaid Overview Medicaid is a jointly funded state-federal program that provides medical coverage to eligible needy persons. 4.1M Programs: Fee for

159 views • 13 slides
Advanced Analytics in Business [D0S07a] Big Data Platforms &amp; Technologies [D0S06a] Data

Advanced Analytics in Business [D0S07a] Big Data Platforms &amp; Technologies [D0S06a] Data

Advanced Analytics in Business [D0S07a] Big Data Platforms &amp; Technologies [D0S06a] Data Science Tools Overview In-memory analytics Python and R More on visualization The road to big data Notebooks and development environments A word on

1.23k views • 100 slides
Pick up a handout on the front table 1 Welcome to DS504/CS586: Big Data Analytics --Review

Pick up a handout on the front table 1 Welcome to DS504/CS586: Big Data Analytics --Review

Pick up a handout on the front table 1 Welcome to DS504/CS586: Big Data Analytics --Review Prof. Yanhua Li Time: 6:00pm 8:50pm R Location: AK232 Fall 2016 Today 1. Review Key topics, techniques, discussed in the semester 2.

660 views • 24 slides
FIGHTING FOOD WASTE Caitrin OBrien Senior Manager, Corporate Sustainability HILTONS 2030

FIGHTING FOOD WASTE Caitrin OBrien Senior Manager, Corporate Sustainability HILTONS 2030

HILTONS COMMITMENT TO FIGHTING FOOD WASTE Caitrin OBrien Senior Manager, Corporate Sustainability HILTONS 2030 SUSTAINABLE F&amp;B GOALS By 2030, we will: Reduce food waste sent to landfill by 50% Participate in food donation

336 views • 14 slides
CBG Business Chain CBG Project in Remote Area Conclusion 2 Energy Outlook and Government Policy

CBG Business Chain CBG Project in Remote Area Conclusion 2 Energy Outlook and Government Policy

Energy Outlook and Government Policy for Alternative Fuel CBG as Alternative Fuel for NGV CBG Business Chain CBG Project in Remote Area Conclusion 2 Energy Outlook and Government Policy for Alternative Fuel 3 Oil &amp; Gas Balance

595 views • 47 slides
On detecting differences between groups Yi Yang Department of Computing Science University of

On detecting differences between groups Yi Yang Department of Computing Science University of

On detecting differences between groups Yi Yang Department of Computing Science University of Alberta Contrast-Set Mining Contrast-Set Mining Understanding the differences between contrasting Understanding the differences between

560 views • 25 slides
Universal Shape Formation for Programmable Matter (Thim Strothmann) Joint work with BDA 2016

Universal Shape Formation for Programmable Matter (Thim Strothmann) Joint work with BDA 2016

Theory of Distributed Systems Universal Shape Formation for Programmable Matter (Thim Strothmann) Joint work with BDA 2016 July 25, 2016 Theory of Distributed Systems Inspiration Video on this slide was deleted to decrease file size.

435 views • 20 slides
SPARQL Graph Pattern Processing with Apache Spark title ?P speaker Hubert Naacke University

SPARQL Graph Pattern Processing with Apache Spark title ?P speaker Hubert Naacke University

SPARQL Graph Pattern Processing with Apache Spark title ?P speaker Hubert Naacke University author P. et M. Curie Paris 6 Olivier Cur Paris Est Marne-la-Valle Bernd Amann GRADES 2017 1 Context Big RDF data Linked Open Data

704 views • 47 slides
732A54 Big Data Analytics Lecture 10: Machine Learning with MapReduce Jose M. Pe na IDA,

732A54 Big Data Analytics Lecture 10: Machine Learning with MapReduce Jose M. Pe na IDA,

732A54 Big Data Analytics Lecture 10: Machine Learning with MapReduce Jose M. Pe na IDA, Link oping University, Sweden 1/27 Contents MapReduce Framework Machine Learning with MapReduce Neural Networks Support Vector

527 views • 27 slides
Syllabus link 1 syllabus BDA17 Syllabus version C 2 Syllabus supplement about memos

Syllabus link 1 syllabus BDA17 Syllabus version C 2 Syllabus supplement about memos

Syllabus Syllabus link 1 syllabus BDA17 Syllabus version C 2 Syllabus supplement about memos Memorandum format for assignments 3 Overheads: Microsoft intro BDA day 1 Microsoft presentation 4 Overheads: some sample problems BDA

501 views • 32 slides

More recommend