Big Data Analytics in Development • Big Data Analytics is making an equally impressive impact on Development interventions – allowing decision-makers to reach and serve previously neglected populations. Busin Bu iness Nee eed Big Big Da Data Ana naly lytics Impact Imp Com ompany Bus usiness Ne Need Data and nd Analy alytic ics Imp mpact Mor ore tran ansparent, reli liable le, and nd low ow-cost t Web scraping of online price data used to Gov overnment statis tistic ical offic ices me meth thod to to trac ack k infla flatio ion in Argentina produce price indices, and econometric sh shift iftin ing to to accept t Big ig Data. analysis used to model disaggregated Central banks using Big Data to impacts of policies see day-to-day volatility. Unde nderstand how how mi migrants act t as s Iterative analysis of call detail records Informin ing labor po polic licy y de desig sign in n arbit itrageurs to bring labor markets into (CDRs) to track movement of migrants in low ow-in income countrie ies to equilibrium response to local shocks to labor demand incentivize or disincentivize (weather, economy, conflict, etc.) migratory behavior The he city ty of f Rio io de de Jan aneir iro wanted to to The city combines data from 30 city Rio io has has imp mproved eme mergency y imp mprove its eme mergency response by agencies – including weather, satellite, response tim ime by by 30 30%, %, better predicting heavy rainfall and video, GPS, historic rainfall, and catalogued 200+ flood points, subsequent severe landslides and topographic survey data – in a central and can now predict heavy rains flooding Operations Center 48 hours in advance on a half- km basis Create a be better ecos osystem for or mo mobile ile Remote crowdsourced data gathered via M-PESA is being used to low ower se servic ices in n the he agric icult ltural l se sectors of cell phones used to connect farmers to cos osts for or far armers to to receiv ive loan oans Kenya, Tanzania, and Mozambique markets, assess farmers’ credit and nd pe perform tran ansactio ions with worthiness, and incubate new mobile distributers and buyers, as well businesses with greater predictors of as to provide geography-specific success market information
Big Data Landscape
Creating a Big Data Organization Step 1: Be Yourself • Beginning with a clear understanding of the specific questions you intend to use Big Data Analytics to address can help guide where and which data solutions are deployed. Value enhancement Deliveri ering g fut uture ure va value ue • Data-driven decision-making in real time • Use analytics to develop new programs/opportunities • Relies heavily on data supplied by others • Often struggles to move away from exclusively intuitive decision-making Strategic Enabl abling ng strat rateg egy y and d imp mprov roving ng perfo rforman rmance • Use analytics to reduce political divergence and drive consensus • Real-time analytics to enable quick responses to events • Use data to develop personalized services • Need for more objective and higher quality data Tactical Value Day y to o day y op opera rations ons enablement • Struggle to move from narrow focus on reactive operations to more proactive, comprehensive management of daily operations • High value for digitization of operational processes across program units • Often already proficient in traditional business Operational intelligence
Creating a Big Data Organization Step 2: Secure People & Skills • The competencies required of “data scientists” within an analytics organization or project converge from multiple skill domains. Sub ubje ject ct Ar Area or or Do Domain Expertise in statistical Deep understanding of industry, subject Expertise Ex techniques, tools and area, or research domain to help languages used to run determine which questions need analyses that generate answering and on what frequency, insights to effectively specificity, or geography determine and communicate actionable insights Compute ter r Science ce & Sta tati tistical & Pr Progr gramming Mat athemat atica cal Comfort in programming across various languages, a thorough understanding of external and internal data sources, data gathering, storing, and retrieving methods which help combine Orga ganization-specific disparate data sources to Inf nformation Kno nowledge Organization-specific knowledge about data generate unique insights assets – including enterprise “metadata” – their location and appropriate business context for use in advanced analytics
Creating a Big Data Organization Step 3: Let objectives dictate structure, not vice versa • How analytics efforts or organizations are structured – whether reporting is vertically or horizontally aligned, how interconnected or autonomous separate units are, how resources and successes are shared – can influence efficiency and impact. Distri Dis ributed ed An Analyt ytics Feder erated ed An Analyt ytics Cen entr tralize zed An Analyt ytics Ana naly lyti tics Co Compet mpeten ency Ana naly lyti tics Co Compet mpeten ency Ana naly lyti tics Co Compet mpeten ency CENTRA TRAL CENTRA TRAL CENTRA TRAL Ce Cente nter Cente Ce nter Cente ter LOCAL LO Meta etadata ta Meta etadata ta Meta etadata ta Reposito itory ry Reposito itory ry Reposito itory ry ETL ETL ET ETL Data Wa Ware reho house Data Wa Ware reho house Data Wa Ware reho house LO LOCAL Data Mart rt Data Mart rt Data Mart rt BI App BI ppli licatio tions BI BI App ppli licatio tions BI App BI ppli licatio tions • Adopt previously proven practices • Subject area-specific innovations • Governance • Highly focused analytics support • Repeatable models • Aligning analytics to organization- Objectives wide strategy • Deployed locally • Deployed locally • Deployed and managed centrally Data Warehouses, • Some data and models shared Marts, etc. across groups • Managed locally • Managed locally, but connected to • Controlled centrally, with units Analytics Tools group framework having access to shared resources • Placed within individual units • Placed within individual units • Placed within central analytics team, Analytics Staff/ • Skills tailored to specific region or available as needed to support Competencies subject matter individual units
The ‘Hub - Spoke’ operating model often serves as a well - synchronized, connected system 4 3 2 1 Glob lobal l Centers of Competency Local Loc l Bus usiness Local Adoption Central Bus usiness Excellence Center Ope peratio ions of Practices Decision Hub Str trategy (Regional) (‘Standards’) 4 4 Local Local ‘Spoke’ ‘Spoke’ Local ‘Spoke’ 3 4 4 Local Sample Hub-Spoke Local Center of ‘Spoke’ Interaction Model ‘Spoke’ Competency Excellence Center (Regional) 3 2 4 Center of Central 4 Local Local Excellence Decision ‘Spoke’ ‘Spoke’ (Regional) Hub 1 4 ‘Standardization’ 4 Local Local ‘Spoke’ ‘Spoke’ Center of 4 Excellence 4 Local Local (Regional) ‘Spoke’ ‘Spoke’ 3
Creating a Big Data Organization Step 4: Invest in Appropriate Infrastructure • Big Data introduces challenges related to data volume and variety, processing constraints, and new data structures that traditional data infrastructure is not equipped to support Obj bjective Consi siderations Impact Dictates performance needs along with data structures Analysis Type Identify the type of and processing architecture analysis that will be Analytics Analysis Interface could restrict the ability to perform analysis conducted and define Capabilities Flexibility ad hoc and restrict ability to update which analytics capabilities will be Analysis Support for analysis specific data structures can employed Structures improve performance and reduce analysis effort Size of data sets introduce need for scalable Size infrastructure and performance Define the data set that will be used for the Variability of source data models and data set Data Variety analysis including its Structure structure require data model flexibility sources, size, and structure Diverse sources will require scalability, model Sources flexibility, and flexible interfaces Frequency of analysis will dictate the processing Frequency Define the timeliness architecture (batch or real time) and frequency of the The timeliness of the analysis will impact the need analysis results for Application Speed for scalability and performance reporting and downstream systems In and out bound interfaces are defined by the use Interfaces of data and required flexibility
Contents 2 Architecture Design
Emerging Infrastructure Options • To harness Big Data, storage solutions must be able to support targeted analytics capabilities, data diversity and performance needs Distributed Processing Hadoop and similar solutions that provide scalable distributed storage and distributed computation on commodity hardware NoSQL Embedded and persisted storage that implement data models through document, graph, and dictionary structures Cloud Computing Cloud computing can improve flexibility, scalability and cost management and enable a cohesive business strategy across a org Traditional challenges being addressed… • Scalability Issues • Data storage solutions need to provide flexible data models to better ingest • Big Data set information extraction and unstructured and semi structured data queries require large volumes of processing cycles that can quickly scale • Need to combine and link multiple data sources
Building an Analytics Organization: Critical Components Emerging Infrastructure – Computing/Storage Options Distributed Processing Hadoop and similar solutions that provide scalable distributed storage and distributed computation on commodity hardware Introduction to Hadoop • Hadoop is based on work done by Google in early 2000s (combination of Google File Faster and Lower Cost Analysis System (GFS) and MapReduce) • Useful for analyzing copious amounts of complex data across multiple data sources • Distributes data as it is initially stored in the system Linear Scalability • Applications are written in high-level code • Computation happens where data is stored, whenever possible Data is replicated multiple times on the • Greater flexibility system for increased availability and reliability
Building an Analytics Organization: Critical Components Emerging Infrastructure – Storage Options NoSQL Embedded and persisted storage that implement data models through document, graph, and dictionary structures NoSQL - Storage Types Key – Value Columnar Store Document Store Graph Store Store Increasing Data Complexity Pros: Simplicity & Pros: Scalability & Pros: Easy to Use Pros: Graph Joins Scalability Flexibility Cons: Scalability Cons: Flexibility Cons: Lack of advanced Cons: Complexity features/queries Solution Examples
Building an Analytics Organization: Critical Components Emerging Infrastructure – System Options Cloud Computing The model is compelling; cloud computing can improve flexibility, scalability and cost management. Businesses best able to realize the potential will establish a cohesive business strategy as cloud computing can transform your entire organization — people, processes, and systems Cloud transformation begins at the infrastructure level and leads to more agile applications, resulting in faster speed to market and more flexibility to meet client needs. The key benefits, beyond consolidation, include standardized application and development environments, resulting in better controlled and more efficient application lifecycles. Source: PwC, “Digital IQ Snapshot: Cloud,”; PwC, “FS Viewpoint: Clouds is the forecast”
Relational Reference Architecture
Extended Relational Reference Architecture
Non-Relational Reference Architecture
Data Discovery: Non-Relational Architecture
Business Reporting: Hybrid Architecture
Contents 3 Big Data Algorithms
Key components of Mahout in Hadoop (1)
Key components of Mahout in Hadoop (2)
Key Components of Spark MLlib
Spark ML Basic Statistics ◼ Correlation: Calculating the correlation between two series of data is a common operation in Statistics ➢ Pearson’s Correlation ➢ Spearman’s Correlation
Example of Popular Similarity Measurements ◆ Pearson Correlation Similarity ◆ Euclidean Distance Similarity ◆ Cosine Measure Similarity ◆ Spearman Correlation Similarity ◆ Tanimoto Coefficient Similarity (Jaccard coefficient) ◆ Log-Likelihood Similarity
Pearson Correlation Similarity Data: Missing Data
On Pearson Similarity Three problems with the Pearson Similarity: 1. Not take into account of the number of items in which two users’ preferences overlap. (e.g., 2 overlap items ==> 1, more items may not be better.) 2. If two users overlap on only one item, no correlation can be computed. 3. The correlation is undefined if either series of preference values are identical. Adding Weighting. WEIGHTED as 2nd parameter of the constructor can cause the resulting correlation to be pushed towards 1.0, or -1.0, depending on how many points are used.
Spearman Correlation Similarity Example for ties Pearson value on the relative ranks
Basic Spark Data Format Data: 1.0, 0.0, 3.0 // straightforward // number of parameters, location of non-zero indices, and non-zero values // number of parameters, Sequence of non-value values (index, value)
Correlation Example in Spark 1.0, 0.0, 0.0, -2.0 4.0, 5.0, 0.0, 3.0 6.0, 7.0, 0.0, 8.0 9.0, 0.0, 0.0, 1.0
Euclidean Distance Similarity Similarity = 1 / ( 1 + d )
Cosine Similarity Cosine similarity and Pearson similarity get the same results if data are normalized (mean == 0).
Caching User Similarity Spearman Correlation Similarity is time consuming. Need to use Caching ==> remember s user-user similarity which was previously computed.
Tanimoto (Jaccard) Coefficient Similarity Discard preference values
Log-LikeLihood Similarity Asses how unlikely it is that the overlap between the two users is just due to chance.
Performance Measurements Using GroupLens data (http://grouplens.org): 10 million rating MovieLens dataset. • Spearnman: 0.8 • Tanimoto: 0.82 • Log-Likelihood: 0.73 • Euclidean: 0.75 • Pearson (weighted): 0.77 • Pearson: 0.89
Spark ML Basic Statistics • Hypothesis testing: Hypothesis testing is a powerful tool in statistics to determine whether a result is statistically significant. Spark ML currently supports Pearson’s Chi- squared (χ2) tests for independence. • ChiSquareTest conducts Pearson’s independence test for every feature against the label.
Chi-Square Tests (1)
Chi-Square Tests (2)
Chi-Square Tests (3) We would reject the null hypothesis that there is no relationship between location and type of malaria. Our data tell us there is a relationship between type of malaria and location.
Chi-Square Tests in Spark
Spark ML Clustering
Example: Clustering Feature Space
Clustering
Clustering – on feature plane
Clustering example
Steps on Clustering
Making Initial Cluster Centers
K-means Clustering
HelloWorld Clustering Scenario Result
Testing difference distance measures
Manhattan and Cosine distances
Tanimoto distance and weighted distance
Results Comparison
Sample Code of K-Means Clustering in Spark
Vectorization Example 0: Weight 1: Color 2: Size
Canopy Clustering (estimate the number of clusters) Tell what size clusters to look for. The algorithm will find the number of clusters that have approximately that size. The algorithm uses two distance thresholds. This method prevents all points close to an already existing canopy from being the center of a new canopy.
Other Clustering Algorithms Hierarchical clustering
Different Clustering Algorithms https://github.com/HewlettPackard/cacti
Spark ML Classification
Spark ML Classification
Classification - definition
Classification example: using SVM to recognize a Toyota Camry
Classification example: using SVM to recognize a Toyota Camry
When to use Big Data System for Classification?
The advantage of using Big Data System for Classification
How does a classification systems work?
Key Terminology for Classification
Input and Output of a classification model
Four types of values for predictor variables
Sample data that illustrates all four values
Supervised vs. Unsupervised Learning
Work flow in a typical classification project
Classification Example – Color-Fill Position looks promising, especially the x-axis ==> predictor variable. Shape seems to be irrelevant. Target variable is “color - fill” label.
Classification Example – Color-Fill (another feature)
Fundamental classification algorithm Example of fundamental classification algorithms: • Naive Bayesian • Complementary Naive Bayesian • Stochastic Gradient Descent (SDG) • Random Forest • Support Vector Machines
Choose algorithm
Support Vector Machine (SVM) maximize boundary distances; remembering “support vectors” nonlinear kernels
Example SVM code in Spark
Contents 4 Tools Support
Data Mining, Text Mining, and Natural Language Processing Natural Language Processing NLP is a theoretically motivated range of computational Text Mining techniques for analyzing and representing Analysis of large naturally occurring texts quantities of natural at one or more levels of language text and linguistic analysis for the detecting lexical or Data Mining purpose of achieving linguistic usage Extraction of implicit , human-like patterns to extract previously unknown , language processing for a probably useful and potentially useful range of tasks or information information from data applications.
NLP Tools Too ool Desc Description Ana Analysis Typ ype • Tokenization • Named entity extraction A machine learning based toolkit for the processing of • sentence segmentation • Chunking, parsing Ope penNLP natural language text. Link • Part-of-speech tagging • Coreference resolution. • Information extraction • Tokenizer A Java suite of tools that can perform natural language GATE ATE • Part of speech tagging • Sentence splitter processing tasks for multiple languages. Link • Information extraction • Word categorization A suite of libraries and programs for symbolic and • Part of speech tagging, • Text classification NL NLTK statistical natural language processing Python. Link • Tokenizer • Including tokenization • Classification Statistical NLP toolkits for various computational • Part-of-speech tagging • Segmentation linguistics problems that can be incorporated into Sta tanford NLP • Named entity recognition • Coreference Resolution applications with human language technology needs. • Parsing Link • Sentiment analysis • Part of speech tagging A tool kit for processing text using computational • Entity recognition • Sentence detection linguistics. Link LingPipe • Clustering • Disambiguation • Topic classification • Information extraction • Text generation A suite of libraries and programs for symbolic and • Part of speech tagging • Stemming statistical natural language processing for both Python Mon ontyLingua • Tokenizer • Phrase chunking and Java. Link • Word categorization • Language Identification • name matching A suite of linguistic analysis components that integrate • Name, places, and key • name translation Ros Rosetta Linguistic into applications for mining unstructured data. Link Platform Pl concept extraction
Text Mining/Analytics Tools Too ool Des Descrip iption Ana nalysis is Type • Document classification • Data mining An open source environment for machine learning, • Sentiment analysis • Traditional analytics Rap apid idMin iner data mining, text mining, predictive analytics, and • Topic tracking business analytics. Link • Text Parsing • Feature Extraction A suite of text processing and analysis tools. Link, SAS S Text t Miner • Filtering • Topic Clustering • Information extractions • Data Mining Integrated development environment for building • Summarization • Document Filtering information extraction systems, natural language VisualT lText • Categorization • Natural Language processing systems, and text analyzers. Link Search • Customer sentiment • sentiment discovery SAS S Sentim timent t Commercial tool that is dedicated to customer Analy alysis is sentiment analysis. Link monitoring • Topic modeling, • Document analysis Tool for sorting large amounts of unstructured text • Information retrieval • Social media analysis Textif tifie ier with The Public Comment Analysis Toolkit (PCAT). Link • Term frequency • Customization of stop System for automatically preparing and • Term frequency inverse transforming unstructured text attributes into a words Infin nfinite • Document frequency • Stemming rules structured representation. Link Ins nsig ight • Root word coding • Concepts merging • synonym identification • Document clustering Software for grouping related documents into Clusti tify fy clusters, providing an overview of the document set and aiding with categorization. Link
Text Mining/Analytics Tools Cont. Too ool De Description Analysis Type • Unstructured • consumer profiling Customer analytics applications that help Attensi sity analyze high volumes of customer communication Ana Analyze conversations across multiple channels. Link analysis • sentiment analysis • Information • Topic Linking A program that automatically identifies and ReV eVerb extracts binary relationships from English extraction • Topic Identification sentences. Link • Document Open source tool for summarizing texts. Ope pen text xt Link summarization summarizer sum • Attribute/feature • Fact identification Web based API that is used to analyze Ope pen Cala alais content and extract topics or information. extraction Link • Semantic Analysis Knowledge Family of techniques tools for searching and Search Sea organizing large data collections. Link • Text Parsing • Network analysis A free software for Quantitative Content KH KH Coder • document search Analysis or Text Mining Link
Image Analytics Overview Ov Overv rview • The process of pulling relevant information from an image or sets of images for advanced classification and traditional analysis • Applies image capture, image processing, and machine learning techniques to extract, quantify, and structure, image information Adv dvantages • Provides a method to structure, organize, and search information that is stored within images • Offers an additional data set that can be applied to understanding consumer behavior, automating business processes, and discovering knowledge enterprise content
Image Analytics Tools Ima mage Co Computer Ma Machin ine Too ool Overv Ov rvie iew Processin ing Visio ion Le Learnin ing Open source library of computer vision functions that is X X X OpenCV Ope accessible via C, Java, and Python Integrated image analysis PAX AXit it Ima mage X X platform that provides basic Analysis Ana feature identification functions Java based image processing platform that can be accessed X Ima mageJ via an API and expanded with custom plugins Python image processing library X PIL A modular machine learning X PyBrain in library for Python
Audio Analytics Overview Over erview • The process of capturing audio and analyzing its features as to extract content and context of an event • Applies speech analysis and signal processing principles to structure audio information for analysis via NLP or traditional analytics techniques Advantages • Provides a method for identifying events or common patterns within sound bytes • Offers a way of capturing not only the content and topics within a conversation, but also the emotions and context
Audio Analytics Tools Aud Audio Infor ormatio ion Too ool Ov Overv rvie iew Processin ing Retr trie ieval A C++ library that provides varying level X X Clam Cla of audio processing and information retrieval capabilities A tool that is capable of translating calls to a more structured text data set and X Call CallMin iner combining with other communication forms Logs calls and structures audio for text X Nuance Nu based search and retrieval Aduio feature extraction toolkit with X yaa aafe wrappers for several languages Multiple platform audio analysis toolkit X PRA RAAT
Social Network → Applications (1) Analysis Ana Obj Objectives Evaluate team structures , • Identify team structures that are not effective information flows among team Co Colla labor oratio ion • Identify informal organizational structures members, and information Ana Analy lysis is • Identify individuals/roles or groups that are exchanges with other teams to influential to collaborative work environments improve working structures • Improve content and knowledge distribution Evaluate how knowledge or Co Content/ • content is diffused and Identify content bottlenecks, open Kno nowle ledge accessed within an communication flows, and establish channels Management Man organization • Explore impact of new communication methods • Improved structures for key organizational functions. Identify groups or informal • Improved information flows teams that share knowledge, Co Communit ity communicate frequently, solve • Identify potential bottlenecks for organizational Mining problems, or work together to functions perform specific tasks • Identify cultural patterns to build other communities Explore formal and informal • Improve hierarchy and structure of organization organization structures and Organiz Or izatio ion to better align with the informal practices how individuals work with one • Develo De lopment Identify team members that are effective leaders another to improve the design and would impact the organization if promoted of the organization
Social Network → Applications (2) Analysis Ana Obj bjectives • Identify communication improvements to disaster recovery Assess organizational structures and Disaster rec Disas ecovery communication patterns as they teams relate to the groups that play a role pla planning • Identify weak links among functional groups to improve in disaster recovery plans collaboration during recovery plan execution • Identify overlapping information sets and bottlenecks for Assess how data points or Data/ Da information sets originate or are information dissemination Informati tion distributed across the enterprise to • Assess how organization structures or information Dissemination Disse their intended targets architecture impact the flow of information to its targets • Identify network agents that collaborate with known Assess the organization or external Fraud De Detecti tion / network to identify communication fraudulent agents prevention pr or collaboration patterns that align • Identify activities that align with known fraudulent behavior with known fraudulent activity • Identify process improvements through discovery of hidden Analyze the organization structure Process ss and communication patterns to process steps, communication flows , and actors uncover process improvements or Disc Discovery / • Discover undocumented or informal processes that are identify new processes Improvement hidden within frequent collaboration and communication paths • Identify communication gaps that could impact dependent Evaluate the structure of a supply network and the interactions among process or operations Sup Supply Cha hain the entities that comprise the • Identify strategic relationships to optimize the supply Ana Analysis network to identify gaps, network bottlenecks and sourcing strategies • Identify supply nodes that create inefficiencies
Social Network → Applications (3) Analysis Ana Obj bjectives • Assess how target consumers/market will react to a piece Observe how a specific topic, news Novelty/ articles or sentiment diffuses of news or campaign Sen Sentiment Dif Diffusi sion through a consumer network • Evaluate how long news, data, or sentiment will be Analysis Ana retained within a system and how far it will spread • Identify individuals or groups that influence markets and Monitor and analyze connections within social media networks to adoption Mar arket Infl fluencer identify markets or consumers that • Identify untapped markets Ide dentifi fication are influential within communities • Identify market segments as targets for ad campaigns to improve product/service adoption • Improve product or service offerings based on attributes Analyze the connections and consumer attributes within the that connect the consumer market Consumer r target market to discover • Develop strategies to target new or existing consumers Seg Segmentation communities or groups with based on identified segmentation characteristics common characteristics • Identify segments or individuals that will be likely early Analyze the flow of communication Product or or Brand or ideas through a market segment adopters to evaluate how a product may • Identify incentives or campaigns that will improve Diffusi Dif sion Ana Analysi sis diffuse product/service adoption • Identify new feature sets for products and services Analyze consumer network Rec ecommendation connections and common features • Assess new markets for selling similar or new products Systems among consumers to develop • Target consumers with specific products or services recommendations
Recommend
More recommend
