Summary of Last Chapter Principles of Knowledge Discovery in - - PDF document

summary of last chapter principles of knowledge discovery
SMART_READER_LITE
LIVE PREVIEW

Summary of Last Chapter Principles of Knowledge Discovery in - - PDF document

Summary of Last Chapter Principles of Knowledge Discovery in Databases What kind of information are we collecting? What are Data Mining and Knowledge Discovery? Fall 1999 What kind of data can be mined? Chapter 2: Data Warehousing


slide-1
SLIDE 1

1

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

1

Principles of Knowledge Discovery in Databases

  • Dr. Osmar R. Zaïane

University of Alberta

Fall 1999

Chapter 2: Data Warehousing and OLAP

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

2

Summary of Last Chapter

  • What kind of information are we collecting?
  • What are Data Mining and Knowledge Discovery?
  • What kind of data can be mined?
  • What can be discovered?
  • Is all that is discovered interesting and useful?
  • How do we categorize data mining systems?
  • What are the issues in Data Mining?
  • Are there application examples?

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

3

  • Introduction to Data Mining
  • Data warehousing and OLAP
  • Data cleaning
  • Data mining operations
  • Data summarization
  • Association analysis
  • Classification and prediction
  • Clustering
  • Web Mining
  • Similarity Search
  • Other topics if time permits

Course Content

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

4

Chapter 2 Objectives

Realize the purpose of data warehousing. Comprehend the data structures behind data warehouses and understand the OLAP technology. Get an overview of the schemas used for multi-dimensional data.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

5

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

6

Incentive for a Data Warehouse

  • Businesses have a lot of data, operational data and facts.
  • This data is usually in different databases and in different

physical places.

  • Data is available (or archived), but in different formats and
  • locations. (heterogeneous and distributed).
  • Decision makers need to access information (data that has been

summarized) virtually on one single site.

  • This access needs to be fast regardless of the size of the data, and

how old the data is.

slide-2
SLIDE 2

2

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

7

Evolution of Decision Support Systems

1960s 1970s 1980s 1990s B a t c h a n d M a n u a l R e p

  • r

t i n g Terminal-based Decision Support Systems Desktop Data Analysis Tools Data Warehousing and On-Line Analytical Processing

  • Statistician
  • Computer scientist

Difficult and limited queries highly specific to some distinctive needs

  • Data Analyst

Inflexible and non-integrated tools

  • Executive

Integrated tools Data Mining

  • Data Analyst

Flexible integrated spreadsheets. Slow access to

  • perational data

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

8

What Is Data Warehouse?

  • A data warehouse consolidates different data sources.
  • A data warehouse is a database that is different and maintained

separately from an operational database.

  • A data warehouse combines and merges information in a consistent

database (not necessarily up-to-date) to help decision support.

Decision support systems access data warehouse and do not need to access operational databases do not unnecessarily over-load operational databases.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

9

Definitions

Data Warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management’s decision making process. (W.H. Inmon)

Subject oriented: oriented to the major subject areas of the corporation that have been defined in the data model. Integrated: data collected in a data warehouse originates from different heterogeneous data sources. Time-variant: The dimension “time” is all-pervading in a data warehouse. The data stored is not the current value, but an evolution of the value in time. Non-volatile: update of data does not occur frequently in the data

  • warehouse. The data is loaded and accessed.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

10

Definitions (con’t)

Data Warehousing is the process of constructing and using data warehouses. A corporate data warehouse collects data about subjects spanning the whole organization. Data Marts are specialized, single-line of business warehouses. They collect data for a department or a specific group of people.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

11

Building a Data Warehouse

Corporate data

Data Mart Data Mart Data Mart Data Mart

Corporate Data Warehouse Option 1: Consolidate Data Marts Option 2: Build from scratch

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

12

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?
slide-3
SLIDE 3

3

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

13

Describing the Organization

We sell products in various markets, and we measure our performance over time Business Manager We sell Products in various Markets, and we measure our performance over Time Data Warehouse Designer

Products Markets T i m e

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

14

Construction of Data Warehouse Based on Multi-dimensional Model

  • Think of it as a cube with labels
  • n each edge of the cube.
  • The cube doesn’t just have 3

dimensions, but may have many dimensions (N).

  • Any point inside the cube is at

the intersection of the coordinates defined by the edge of the cube.

  • A point in the cube may store

values (measurements) relative to the combination of the labeled dimensions.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

15

Concept-Hierarchies

Dimensions are hierarchical by nature: total orders or partial orders Example: Location(continent country province city) Time(yearquarter(month,week)day)

Dimensions: Product, Region, week Hierarchical summarization paths Industry Country Year Category Region Quarter Product City Month Week Office Day

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

16

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

17

On-Line Transaction Processing

  • Database management systems are typically used for on-line

transaction processing (OLTP)

  • OLTP applications normally automate clerical data

processing tasks of an organization, like data entry and enquiry, transaction handling, etc. (access, read, update)

  • Database is current, and consistency and recoverability are
  • critical. Records are accessed one at a time.

OLTP operations are structured and repetitive OLTP operations require detailed and up-to-date data OLTP operations are short, atomic and isolated transactions Databases tend to be hundreds of Mb to Gb.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

18

On-Line Analytical Processing

  • On-line analytical processing (OLAP) is essential for

decision support.

  • OLAP is supported by data warehouses.
  • Data warehouse consolidation of operational databases.
  • The key structure of the data warehouse always contains

some element of time.

  • Owing to the hierarchical nature of the dimensions, OLAP
  • perations view the data flexibly from different perspectives

(different levels of abstractions).

  • OLAP operations:
  • roll-up (increase the level of abstraction)
  • drill-down (decrease the level of abstraction)
  • slice and dice (selection and projection)
  • pivot (re-orient the multi-dimensional view)
  • drill-through (links to the raw data)

DW tend to be in the order of Tb

slide-4
SLIDE 4

4

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

19

Red Deer Q1 Q2 Q4 Drama Horror

  • Sci. Fi..

Comedy

Time (Quarters) Location (city, AB)

Q3 Edmonton Calgary Lethbridge

Category

Red Deer Jul Sep Drama Horror

  • Sci. Fi..

Comedy

Time (Months, Q3) Location (city, AB) Category

Aug Edmonton Calgary Lethbridge

Drill down on Q3 Roll-up on Location

Prairies Q1 Q2 Q4 Drama Horror

  • Sci. Fi..

Comedy

Time (Quarters) Location (province, Canada) Category

Q3 Maritimes Quebec Ontario Western Pr

OurVideoStore Data Warehouse

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

20

January

Slice on January

Edmonton Electronics January

Dice on

Electronics and Edmonton

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

21

OLTP vs OLAP

OLTP OLAP users Clerk, IT professional Knowledge worker function day to day operations decision support DB design application-oriented subject-oriented data current, up-to-date detailed, flat relational isolated historical, summarized, multidimensional integrated, consolidated usage repetitive ad-hoc access read/write index/hash on prim. key lots of scans unit of work short, simple transaction complex query # records accessed tens millions #users thousands hundreds DB size 100MB-GB 100GB-TB metric transaction throughput query throughput, response

(Source: JH)

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

22

Why Do We Separate DW From DB?

  • Performance reasons:

– OLAP necessitates special data organization that supports multidimensional views. – OLAP queries would degrade operational DB. – OLAP is read only. – No concurrency control and recovery.

  • Decision support requires historical data.
  • Decision support requires consolidated data.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

23

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

24

Three-tier Architecture

Data Warehouse

Extract Transform Load Refresh

metadata

OLAP Server

Analysis Query Reports Data mining

Client Tools Serve

Data Marts

External sources

Data sources

Operational DBs

Monitor & Integrator

slide-5
SLIDE 5

5

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

25

Data Sources

  • Data sources are often the operational systems,

providing the lowest level of data.

  • Data sources are designed for operational use, not for

decision support, and the data reflect this fact.

  • Multiple data sources are often from different systems

run on a wide range of hardware and much of the software is built in-house or highly customized.

  • Multiple data sources introduce a large number of

issues -- semantic conflicts.

Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

26

Data Cleaning

  • Data cleaning is important to warehouse.

– Operational data from multiple sources are often noisy (may contain data that is unnecessary for DS).

  • Three classes of tools.

– Data migration: allows simple data transformation. – Data scrubbing: uses domain-specific knowledge to scrub data. – Data auditing: discovers rules and relationships by scanning data (detect outliers).

Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

27

Load and Refresh

  • Loading the warehouse includes some other processing tasks:

– Checking integrity constraints, sorting, summarizing, build indices, etc.

  • Refreshing a warehouse means propagating updates on source data

to the data stored in the warehouse. – When to refresh.

  • Determined by usage, types of data source, etc.

– How to refresh.

  • Data shipping: using triggers to update snapshot log table and propagate

the updated data to the warehouse.

  • Transaction shipping: shipping the updates in the transaction log.
Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

28

Monitor

  • Detect changes to an information source that are of interest

to the warehouse. – Define triggers in a full-functionality DBMS. – Examine the updates in the log file. – Write programs for legacy systems.

  • Propagate the change in a generic form to the integrator.
Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

29

Integrator

  • Receive changes from the monitors

– Make the data conform to the conceptual schema used by the warehouse

  • Integrate the changes into the warehouse

– Merge the data with existing data already present – Resolve possible update anomalies

Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

30

Metadata Repository

  • Administrative metadata

– Source database and their contents – Gateway descriptions – Warehouse schema, view and derived data definitions – Dimensions and hierarchies – Pre-defined queries and reports – Data mart locations and contents – Data partitions – Data extraction, cleansing, transformation rules, defaults – Data refresh and purge rules – User profiles, user groups – Security: user authorization, access control

Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

slide-6
SLIDE 6

6

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

31

Metadata Repository

  • Business data

– business terms and definitions – ownership of data – charging policies

  • Operational metadata

– data lineage: history of migrated data and sequence of transformations applied – currency of data: active, archived, purged – monitoring information: warehouse usage statistics, error reports, audit trails

Prin c i p les o f Kn
  • wled g
e Disco v e ry in D atab ase s Un iv e rsity
  • f Alb erta
 Dr. Osmar R . Z aïan e , 1 9 9 9 23

Three-tier Architecture

Data Wareho use Extract Tra nsform Load Refresh metadata OLAP Server Ana lysis Que ry Re ports Data mining Client Tools Serve Data Marts External sources Data sources Operational DBs Monitor & Integra tor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

32

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

33

Data Warehouse Design

Most data warehouses use a star schema to represent the multi- dimensional model. Each dimension is represented by a dimension-table that describes it. A fact-table connects to all dimension-tables with a multiple

  • join. Each tuple in the fact-table consists of a pointer to each of

the dimension-tables that provide its multi-dimensional coordinates and stores measures for those coordinates. The links between the fact-table in the centre and the dimension- tables in the extremities form a shape like a star. (Star Schema)

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

34

Example of Star Schema

Date Month Year Date CustId CustName CustCity CustCountry Cust Sales Fact Table Date Product Store Customer unit_sales dollar_sales ProductNo ProdName ProdDesc Category

Product

StoreID City State Country Region

Store

(Source: JH)

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

35

Data Warehouses Design (con’t)

  • Modeling data warehouses: dimensions & measurements

Star schema: A single object (fact table) in the middle connected to a number of objects (dimension tables) Each dimension is represented by one table Un-normalized (introduces redundancy). Ex: (Edmonton, Alberta, Canada, North America) (Calgary, Alberta, Canada, North America) Normalize dimension tables Snowflake schema

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

36

Data Warehouses Design (con’t)

  • Snowflake schema: A refinement of star schema where the

dimensional hierarchy is represented explicitly by normalizing the dimension tables.

  • Fact constellations: Multiple fact tables share dimension tables.
slide-7
SLIDE 7

7

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

37

Example of Snowflake Schema

Date Month Date CustId CustName CustCity CustCountry Cust Sales Fact Table Date Product Store Customer unit_sales dollar_sales ProductNo ProdName ProdDesc Category

Product

Month Year Month Year Year

City State City Country Region Country State Country State StoreID City Store

(Source: JH)

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

38

What Is the Best Design?

Performance benchmarking can be used to determine what is the best design. Snowflake schema: Easier to maintain dimension tables when dimension table are very large (reduces overall space). Star schema: More effective for data cube browsing (less joins): can affect performance.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

39

Aggregation in Data Warehouses

Drama Comedy Horror

Category Sum Group By Sum Aggregate

Drama Comedy Horror Q4 Q1

By Time By Category Sum Cross Tab

Q3 Q2 Q1 Q2 Red Deer E d m

  • n

t

  • n

Drama Comedy Horror

By Category By Time & Category By Time & City By Category & City By Time By City Sum The Data Cube and The Sub-Space Aggregates

Lethbridge Calgary Q3 Q4

Multidimensional view of data in the warehouse: Stress on aggregation of measures by one or more dimensions

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

40

Construction of Multi-dimensional Data Cube

sum 0-20K20-40K 60K- sum Algorithms … ... sum Database

Amount Province Discipline

40-60K B.C. Prairies Ontario

All Amount Algorithms, B.C.

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

41

A Star-Net Query Model

Shipping Method Air-Express Truck Order Customer Orders Contracts Customer Product Product Group Product Line Product Item Sales Person District Division Organization Promotion City Region Country Geography Daily Quarterly Annually Time

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

42

Implementation of the OLAP Server

ROLAP: Relational OLAP - data is stored in tables in relational database or extended-relational databases. They use an RDBMS to manage the warehouse data and aggregations using often a star schema.

  • They support extensions to SQL
  • A cell in the multi-dimensional structure is represented by a tuple.

Advantage: Scalable (no empty cells for sparse cube). Disadvantage: no direct access to cells.

Ex: Microstrategy Metacube (Informix)

slide-8
SLIDE 8

8

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

43

Implementation of the OLAP Server

MOLAP: Multidimensional OLAP – implements the multidimensional view by storing data in special multidimensional data structures (MDDS) Advantage: Fast indexing to pre-computed aggregations. Only values are stored. Disadvantage: Not very scalable and sparse HOLAP: Hybrid OLAP - combines ROLAP and MOLAP

  • technology. (Scalability of ROLAP and faster computation of MOLAP)

Ex: Essbase of Arbor

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

44

View of Warehouses and Hierarchies with DBMiner

  • Importing data
  • Table Browsing
  • Dimension creation
  • Dimension browsing
  • Cube building
  • Cube browsing

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

45

DBMiner Cube Visualization

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

46

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

47

Issues

  • Scalability
  • Sparseness
  • Curse of dimensionality
  • Materialization of the multidimensional

data cube (total, virtual, partial)

  • Efficient computation of aggregations
  • Indexing

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

48

Data Warehouse and OLAP Outline

  • What is a data warehouse and what is it for?
  • What is the multi-dimensional data model?
  • What is the difference between OLAP and OLTP?
  • What is the general architecture of a data warehouse?
  • How can we implement a data warehouse?
  • Are there issues related to data cube technology?
  • Can we mine data warehouses?
slide-9
SLIDE 9

9

Principles of Knowledge Discovery in Databases University of Alberta

 Dr. Osmar R. Zaïane, 1999

49

Data Mining

Data mining requires integrated, consistent and cleaned data which data warehouses can provide. Data mining tools can interface with the OLAP engine to take advantage of the integrated and aggregated data, as well as the navigation power. Interactive and exploratory mining. OLAP-based mining is referred to as OLAP- mining or OLAM (on-line analytical mining).