MetaData Management 2005 MetaData Management 2005 Toronto IRMAC - - PowerPoint PPT Presentation
Gavilan Research Associates Gavilan Research Associates MetaData Management 2005 MetaData Management 2005 Toronto IRMAC April 19, 2005 April 19, 2005 Toronto IRMAC DAMA Wisconsin April 20, 2005 April 20, 2005 DAMA
Tuesday April 19, 2005 10:15:50 AM
1
Tuesday April 19, 2005 10:15:50 AM
2
Tuesday April 19, 2005 10:15:50 AM
3
source: Gartner Group Inc (October source: Gartner Group Inc (October 2004) 2004)
source: Miller Heiman, Inc. (March 2005) source: Miller Heiman, Inc. (March 2005)
Tuesday April 19, 2005 10:15:50 AM
4
– – limited audience (mostly technical/ developers) limited audience (mostly technical/ developers)
– – average metadata proj ect = $200 average metadata proj ect = $200-
500K+ per year
Tuesday April 19, 2005 10:15:50 AM
5
Tuesday April 19, 2005 10:15:50 AM
6
– – MetaData MetaData “ “ veteran veteran” ” , 20 years in the business. , 20 years in the business. – – has worked for notable metadata companies such as has worked for notable metadata companies such as Informatica, Data Advantage Group, R&O S
Informatica, Data Advantage Group, R&O S
Manager S
Manager S
Group. Group. – – has personally given over has personally given over one t housand present at ions
& t raining workshops & t raining workshops to Global 5000 companies on
to Global 5000 companies on enterprise meta enterprise meta-
data management www.gavilanresearch.com www.gavilanresearch.com
Tuesday April 19, 2005 10:15:50 AM
7
Tuesday April 19, 2005 10:15:50 AM
8
Typical Tools involved in DW development Typical Tools involved in DW development
Modeling Tools Extract / Transform / Load
Source Source Data Data Target Target DBMS DBMS
BI / Reporting Tools
1 1 2 3 4
Tuesday April 19, 2005 10:15:50 AM
9
(definit ions, descript ions, and int er (definit ions, descript ions, and int er-
relat ionships)
Modeling Tools Extract / Transform / Load
Sources Sources Targets Targets
BI / Reporting Tools
Tuesday April 19, 2005 10:15:50 AM
10
What data exists ?
Where is it being used ?
What is its business definition ?
What other names has it been called
How is it inter-
related to
Who is using it ?
Why do we need it ?
When was it last updated ?
Your Employees Can Your Employees Can’ ’ t Find Anyt hing t Find Anyt hing -
OR-
They have t o go t o mult iple places for Research
Tuesday April 19, 2005 10:15:50 AM
11
CASE/Modeling CASE/Modeling Tools Tools Any Other Any Other Meta Data Meta Data Source Source RDBMS / Source Code RDBMS / Source Code Stored Procedures, Stored Procedures, Files, Records, Files, Records, Programs Programs ETL Tools ETL Tools Business Rules, Business Rules, End End-
user requirements, Organizations, Organizations, Locations Locations BI / Reporting BI / Reporting Tools Tools Word Docs, Excel, Word Docs, Excel, Diagrams, etc. Diagrams, etc. People People’ ’ s s Heads Heads
Metadata Metadata Management Management Solution Solution
Tuesday April 19, 2005 10:15:50 AM
12
# of metadata source-types /
# of metadata solution US ERS few dozens many hundreds vocabularies / ontologies business metadata technical metadata ETL/ DW metadata process / source-code / reusable components Data-Quality metadata GIS met adata document metadata XML/ web/ internet metadata corporate data dict ionary Frameworks / Enterprise Architecture hardware / software assets
c
p l e x i t y
Tuesday April 19, 2005 10:15:50 AM
13
Analogy Analogy: :
In a given corporation, there could be 10,000 books there could be 10,000 books (information assets), in (information assets), in 3,000 different locations 3,000 different locations
These books are not useful to the corporation until to the corporation until they are inventoried, they are inventoried,
referenced, and made available to the rest of the available to the rest of the corporation corporation
Metadata Solution
Tuesday April 19, 2005 10:15:50 AM
14
Answering Two Key Questions Answering Two Key Questions: :
What Information exists ? ( (self self -
service “ “ discovery discovery” ” & & search) search)
How or Where is this information being used ? information being used ? (MapQuest (MapQuest – – focus on t he focus on t he int er int er-
relat ionships & “ “ mapping mapping” ” bet ween bet ween informat ion asset s) informat ion asset s)
Metadata Solution
Tuesday April 19, 2005 10:15:50 AM
15
DATARECORD COUNTS DATARECORD DESCR-FM-F85 DATARECORD SUPER-ZA-F97 DATARECORD CAUSE-MESSAGE DATARECORD MESSAGES DATARECORD RCR DATARECORD ACCUMULATORS DATARECORD GENINTRF-FMT2-SEGMENT DATARECORD TRIGGERS DATARECORD PREV-HOLD-AREAS DATARECORD ALPHA-TABLE DATARECORD TABLE-D-CONVERT DATARECORD SAVE-DOC-ID DATARECORD GENINTRF-FMT3-SEGMENT DATARECORD SAVE-SCHEDULE-CERT-DEP DATARECORD SCH-CERT-DEP-TABLE-RED DATARECORD ADA-DATE-TABLE DATARECORD ALPHA-TBL DATARECORD SERVED-SERVING-TBL DATARECORD SUPER-ZC-F80 DATARECORD GENINERR-RECORD DATARECORD GENINTRF-FMT14-SEGMENT DATARECORD SAVE-ICAN DATARECORD DATE-WORK-AREA DATARECORD GENERATED-FMT3-SEGMENT DATARECORD SAVE-INV-CUST-ACCT-NUM DATARECORD GENERATED-FMT2-SEGMENT DATARECORD GENERATED-FMT1-SEGMENT DATARECORD GENINTRF-RECORD DATARECORD GENINTRF-FMT1-SEGMENT DATARECORD SUPER-ZA-F84 DATARECORD WORK-SCHEDULE-CERT-DEP DATARECORD WORK-SCHEDULE-CERT-DEP-VPY DATARECORD ADAMINT-CONSTANTS DATARECORD GENERATED-REC DATARECORD GENERATED-FROM-INPUT DATARECORD TRANS-SEQ-NUMBER-VPY PROGRAM DF473 FIELD IN-COUNT FIELD OUT-COUNT FIELD DATA-STRING-CNT FIELD TOT-BATCHES-TRANS-CNT FIELD TOT-BATCHES-GEN-CNT FIELD TOT-ERR-BATCHES-CNT FIELD SUB FIELD SUB1 FIELD COMPLETE-RCD FIELD SUB2 FIELD SUB3 FIELD SUB4 FIELD SUB5 FIELD MLS-SEQ-NUM FIELD VARIABLE-W FIELD TOT-NET-AMOUNT FIELD CHK-PT FIELD RC FIELD JULIAN-DATE-3-FLAG FIELD JULIAN-DATE-2-FLAG FIELD JULIAN-DATE-1-FLAG FIELD SCHED-CERT-FLAG FIELD SCHED-CERT-BUILT FIELD TRAILER-REC FIELD DOC-NUMBER FIELD DOC-SUFFIX FIELD TRANS-CODE FIELD TC-STATUS-CODE FIELD AMT FIELD EFF-DATE FIELD EFF-DATE-MM FIELD EFF-DATE-DD FIELD EFF-DATE-YY FIELD FMT1-FIL-R1 FIELD DOC-REF-METER-NUM FIELD LINE-COUNT FIELD AMT-QTR-1 FIELD AMT-QTR-2 FIELD AMT-QTR-3 FIELD AMT-QTR-4 FIELD ALLOTTEE FIELD ALLOT-LVL-AFC FIELD FMT1-FIL-R2 FIELD INPUT-RGN-DIST FIELD INPUT-AGY FIELD INPUT-RGN FIELD FMT1-FIL-R3 FIELD FMT2-FIL-R1 FIELD ALPHA-7 FIELD ALPHA-8 FIELD ALPHA-9 FIELD ALPHA-10 FIELD ALPHA-11 FIELD ALPHA-12 FIELD ALPHA-13 FIELD ALPHA-14 FIELD ALPHA-15 FIELD ALPHA-16 FIELD ALPHA-17 FIELD ALPHA-18 FIELD ALPHA-19 FIELD ALPHA-20 FIELD ALPHA-21 FIELD ALPHA-22 FIELD ALPHA-23 FIELD ALPHA-24 FIELD ALPHA-25 FIELD ALPHA-26 FIELD ALPHA-27 FIELD ALPHA-28 FIELD ALPHA-29 FIELD ALPHA-30 FIELD HOLD-JUL-DATE FIELD HOLD-JUL-DATE-YY FIELD HOLD-JUL-DATE-DD FIELD HOLD-GREG-DATE FIELD HOLD-GREG-DATE-MO FIELD HOLD-GREG-DATE-DAY FIELD HOLD-GREG-DATE-YR FIELD HOLD-YEAR FIELD 00-YR FIELD HOLD-YR FIELD SAVE-JUL-DATE FIELD SAVE-JUL-YR FIELD SAVE-JUL-DAY FIELD SAVE-GREG-DATE FIELD SAVE-GREG-MO FIELD SAVE-GREG-DAY FIELD SAVE-GREG-YR FIELD END-MO-DATE-SAVE FIELD END-YY-SAVE FIELD END-DD-SAVE FIELD PREV-AGY-RGN-DIST FIELD PREV-INPUT-RGN-DIST FIELD PREV-VENDOR-TYPE FIELD PREV-VENDOR-SSN FIELD END-MO-DATE-YY FIELD END-MO-DATE-DD FIELD SCHED-CERT FIELD NUMERIC-CAUSE FIELD CAUSE-TEXT FIELD PROG-ID FIELD ADA-OPEN FIELD ADA-FD008001 FIELD ADA-LV008401 FIELD ADA-RV008401 FIELD ADA-FD008501 FIELD ADA-FD009701 FIELD ABEND-DF473 FIELD ABEND1-DF473 FIELD ABEND2-DF473 FIELD ABEND3-DF473 FIELD ABEND3-INTERFACE FIELD ABEND4-DF473 FIELD ABEND4-AGY-RGN-DIST FIELD ABEND5-DF473 FIELD ABEND5-TRANS-NUM FIELD ABEND6-DF473 FIELD MSG-INVALID-INTERFACE-3 FIELD INVALID-INTERFACE-2A FIELD MSG-INVALID-INTERFACE-2 FIELD INVALID-INTERFACE-CODE FIELD INVALID-INTERFACE-1A FIELD MSG-INVALID-INTERFACE-1 FIELD MSG-INT-TABLE FIELD MSG-EOF FIELD NO-RECORD-ON-DATE-TABLE FIELD MSG-STOP-DF473 FIELD MSG-ERROR-CNT FIELD MSG-GENERATED-CNT FIELD MSG-XMITTED-CNT FIELD MSG-COMPUTED-CNT FIELD MSG-ABNORM-EOJ FIELD MSG-NORM-EOJ FIELD MSG-DF473-PURPOSE FIELD MSG-START-DF473 FIELD MSG-AST FIELD SCHED-NUM FIELD JULIAN-DATE-3 FIELD JULIAN-DATE-2 FIELD JULIAN-DATE-1 FIELD WS-JULIAN-DATE FIELD SAVE-INPUT-RGN FIELD SCHED-TYPE FIELD SUFFIX FIELD FISCAL-YR FIELD DOC-ID-1-2 FIELD POS-6-9 FIELD POS-1-5 FIELD POS-1-9 FIELD PREV-INV-CUST-ACCT FIELD PREV-LINE-COUNT FIELD PREV-INTERFACE-CODE FIELD PREV-TRANS-NUM FIELD PREV-FIS-YR FIELD PREV-EFF-DATE FIELD DUP-SCHED-NUM-6 FIELD DUP-SCHED-NUM-6-7 FIELD DUP-SCHED-NUM-3-5 FIELD DUP-INP-RGN FIELD DUP-WORK-SCHED-TYPE FIELD DUPLICATE-SCHED-CERT-DEP FIELD ALPHA-D FIELD JULIAN-D FIELD TABD-DATA FIELD TABLE-D-ENTRY FIELD FILLER FIELD TABLE-D-DATA FIELD ALPHA-CHAR FIELD ALPHA-36 FIELD ALPHA-35 FIELD ALPHA-34 FIELD ALPHA-33 FIELD ALPHA-32 FIELD ALPHA-31 FIELD COMPUTATION-DATE FIELD STAT-DATA-2 FIELD STAT-DATA-1 FIELD ENERGY-CONV-CODE2 FIELD ENERGY-CONV-CODE1 FIELD SCHEDULE-TYPE FIELD TRAVEL-DATE FIELD VENDOR-SSN FIELD VEND-TYPE FIELD INV-CUST-ACCT-NUM FIELD FMT2-FIL-R3 FIELD CFWD-AMT FIELD FMT2-FIL-R2 FIELD PRIOR-FY-AMT FIELD AMOUNT-QTR4 FIELD AMOUNT-QTR3 FIELD AMOUNT-QTR2 FIELD AMOUNT-QTR1 FIELD LINE-CNT FIELD PREV-REC-TRAILER FIELD BAD-TRANS FIELD GOOD-TRANS FIELD CURRENT-MONTH-10 FIELD MONTH-EQUAL-10 FIELD ERROR-SWITCH-SET FIELD ERROR-SWITCH FIELD INPUT-EOF FIELD END-OF-FILE FIELD VALID-GREG-DATE FIELD INVALID-GREG-DATE FIELD GREG-DATE-VALIDATION FIELD VALID-JULIAN-DATE FIELD INVALID-JULIAN-DATE FIELD JULIAN-DATE-VALIDATION FIELD INVALID-AGENCY FIELD VALID-AGENCY FIELD AGENCY-ON-T01 FIELD TRAILER-RECORD FIELD ABEND6-SEQ-NUM FIELD TERMS-PERCENT FIELD EXCL-DISC-AMT FIELD REASON-CODE FIELD FACILITY-TYPE FIELD FACILITY-LOCATION FIELD IOTV-NUMBER FIELD ENERGY-CONV-CODE FIELD STAT-DATAS FIELD STAT-DATA FIELD COMPUTATION-DATE-YY FIELD COMPUTATION-DATE-MM FIELD COMPUTATION-DATE-DD FIELD INP-RGN FIELD POS-6-7 FIELD POSI-6 FIELD POSI-7 FIELD WORK-SCHED-TYPE-VPY FIELD INP-RGN-VPY FIELD MSG-PROGRAMMER-INFO-2 FIELD DF473 FIELD ADAOPEN FIELD ADACLOSE FIELD ADASNAP FIELD OPENMODE FIELD GOOD-TRANS-CODES-FPY-MLS FIELD FPY-MLS-TRANS-CODE FIELD BYTE-6-A-P FIELD A-P FIELD SS-MATCH FIELD SCD-SW FIELD FIRST-DETAIL FIELD LAST-REC-TRAILER FIELD INVALID-INTERFACE-TRIG FIELD 80-CHAR-PART FIELD INTERFACE-CODE FIELD POS-1 FIELD POS-2 FIELD POS-3 FIELD AGY-RGN-DIST FIELD AGY-CODE FIELD RGN-DIST FIELD TRANSMISSION-NUMBER FIELD JULIAN-DATE FIELD TRANS-SEQ-NO FIELD WAREHOUSE-DATE FIELD TERMS-DAY FIELD TERMS-NET FIELD TRANSFER-RGN-DIST FIELD STATUS-FLAG FIELD SCHEDULE-CERT-DEP FIELD CROSS-REGION-CODE FIELD ORIG-SCH-PREFIX FIELD PRIOR-YEAR-FLAG FIELD IMP-FUND-CASHIER FIELD FMT3-FILLER-R1 FIELD AGREE-NUM FIELD BILL-NUM FIELD COLL-CORRECT-FLAG FIELD COLL-ADV-MISC-FLAG FIELD FMT14-FIL-R1 FIELD DUP-SCHED-NUM-7 FIELD TRANSMISSION-NUM FIELD ENERGY-CONV-CODES FIELD WAREHOUSE-DATE-YY FIELD WAREHOUSE-DATE-MM FIELD WAREHOUSE-DATE-DD FIELD WK-DISPLAY-COUNT FIELD WK-CURRENT-DATE FIELD WK-C-DATE-MM FIELD WK-C-DATE-DD FIELD WK-C-DATE-YY FIELD WK-CURRENT-TIME FIELD WK-CURRENT-HHMM FIELD SCHED-NUM-6 FIELD SCHED-NUM-7 FIELD WORK-SCHED-TYPE FIELD SCHEDULE-TYPE-VPY FIELD WS-JULIAN-DATE-VPY FIELD SCHED-NUM-VPY FIELD TRANS-SEQ-NUM-1 FIELD TRANS-SEQ-NUM-2 FIELD ICAN-1-5 FIELD ICAN-1-THRU-5 FIELD ICAN-1 FIELD ICAN-2 FIELD ICAN-3 FIELD ICAN-4 FIELD ICAN-5 FIELD TBL-ID-ZC FIELD INVALID-INTERFACE-3A FIELD MSG-ERR-NO-SCHED FIELD MSG-PROGRAMMER-INFO-1 FIELD RCANAL FIELD FD008001 FIELD LV008401 FIELD RV008401 FIELD FD008501 FIELD FD009701 FIELD PREV-RECORD-TRAILER FIELD SEQ-NUM FIELD APPROP-CODE FIELD APPROP-CODE-LIM FIELD ALLOT-FUND-CNTL FIELD ALLOT-LEVEL-IND FIELD PROG-ELEM FIELD COST-CNTR FIELD OBJ-CLASS FIELD PUBLIC-GOVT-IND FIELD SYS-DATA FIELD DOC-ID FIELD DOC-TYPE FIELD FIS-YR FIELD FMT14-FIL-R2 FIELD SCHEDULE-NUM FIELD FMT14-FIL-R3 FIELD ERROR-CODE FIELD FMT14-FIL-R4 FIELD ALPHA-1 FIELD ALPHA-2 FIELD ALPHA-3 FIELD ALPHA-4 FIELD ALPHA-5 FIELD ALPHA- FIELD JULDATE FIELD JULIAN-YR FIELD JULIAN-DAY FIELD GREGDATE FIELD CURR-MO FIELD CURR-DAY FIELD CURR-YR FIELD HOLD-SCHEDULE-CERT-DEP FIELD HOLD-INPUT-RGN-DIST FIELD HOLD-SCHED-NUM-7 FIELD HOLD-JULIAN-DATE-3 FIELD HOLD-JULIAN-D FIELD HOLD-A FIELD AGY-RGN-DIST-ZC FIELD AGY-CODE-ZC FIELD RGN-DIST-ZC FIELD RCD-TYPE-ZA FIELD FIS-YEAR-ZA FIELD PROCESS-MO-ZA FIELD BEG-MO-DATE-ZA FIELD END-MO-DATE-ZA FIELD RCD-TYPE-ADA FIELD FIS-YEAR-ADA FIELD PROCESS-MO-ADA FIELD BEG-MO-DATE-A FIELD END-MO Tuesday April 19, 2005 10:15:50 AM
16
“ “Bucket Bucket” ” (traditional (traditional repository) repository)
(passive) copies and physically stores metadata
Hybrid Hybrid
real-time access with some copying / storage
Real Real-
Time
active (real-time) access to metadata (no storage)
Metadata Sources - Development Tool Repositories - DBMS Catalogs, etc …
Tuesday April 19, 2005 10:15:50 AM
17
Social Security Number ( Grouping Object) Emp_I D_Code ( ERw in) Soc- Sec- Num ( Oracle) Cust- I D- Code ( I nformatica) Employee_Number ( Bus. Objects)
t he classic t he classic “ “ met adat a reposit ory met adat a reposit ory” ”
good architecture for good architecture for physical metadata physical metadata integration integration + + definitions need to be definitions need to be continuously continuously refreshed and re refreshed and re-
synchronized
metadata definitions are copied and are copied and physically stored by physically stored by parsing, scanning, parsing, scanning, and/or importing and/or importing +/ +/ -
Tuesday April 19, 2005 10:15:50 AM
18
Social Security Number ( Grouping Object) Emp_I D_Code ( ERw in) Soc- Sec- Num ( Oracle) Cust- I D- Code ( I nformatica) Employee_Number ( Bus. Objects)
functional functional emphasis on emphasis on reporting reporting +/ +/ -
definitions do not need to be need to be continuously continuously refreshed or re refreshed or re-
synchronized + + metadata metadata definitions are definitions are accessed in real accessed in real-
time by active interfaces interfaces + +
This “ grouping obj ect” cannot exist in a pure (100% ) or real-time metadata system, because you cannot store obj ects.
Tuesday April 19, 2005 10:15:50 AM
19
Social Security Number ( Grouping Object) Emp_I D_Code ( ERw in) Soc- Sec- Num ( Oracle) Cust- I D- Code ( I nformatica) Employee_Number ( Bus. Objects)
bot h bot h “ “ bucket bucket ” ” and and “ “ real real -
t ime” ”
active definitions do active definitions do not need to be not need to be continuously re continuously re-
synchronized, but unique definitions do unique definitions do +/ +/ -
unique or extended definitions are definitions are physically stored in a physically stored in a separate metadata separate metadata repository repository +/ +/ -
development tool definitions are definitions are accessed in real accessed in real-
time by active (on by active (on-
demand) metadata interfaces interfaces + +
Tuesday April 19, 2005 10:15:50 AM
20
( (aka aka “ “ Repository Information Model Repository Information Model” ” ) ) “ “ t he heart & sole of your met adat a solut ion dat abase t he heart & sole of your met adat a solut ion dat abase” ”
PROC C-VARIABLE C-FUNCTION PLI-PROCEDURE AUTHOR MACRO DB2HOSTVAR SECTION CICS-TRANSACTION STANDARDS DB2CNSTR DB2DBS DB2INDEX DB2IPAR DB2SYN SYSTEM VSAMCAT VOLUME DB2STGRP DB2TPART DB2TSP DB2USER DB2TABLE DB2COL INCLUDE MAPSET MAP IMS-A IMS-TRANSACTION IMS-D DDNAME IMS-DBD IMS-PSB IMS-SEGMENT IMS-RECORD DOMAIN FIELD DATARECORD COPY PROGRAM PROCSTEP DSN JOBSTEP NET JOB
Tuesday April 19, 2005 10:15:50 AM
21
You should be able t o model & capt ure You should be able t o model & capt ure any any t ype of met adat a t ype of met adat a Add any obj ect Add any obj ect -
t ypes, at t ribut es, or relat ionships
Entity Type
Business Rule
Table
Program
Operating Platform
Critical Success Factor
Business Technical
ERwin Oracle COBOL
Tuesday April 19, 2005 10:15:50 AM
22
S
(DDL, S chemas) DBMS (DDL, S chemas)
Flat Files, Office Documents
Modeling & Development Tools Tools
ETL, BI, & Data Warehouse Tools Tools
Purchased Applications / Packages Packages
Industry Models / Templates Templates
Ascential Ascential DB2 DB2 Oracle Oracle Informatica “ “C C” ” Business Objects Business Objects Sybase Sybase ERwin ERwin COBOL COBOL
Tuesday April 19, 2005 10:15:50 AM
23
Development Test Production
Global Project 1 Project 2 Project 3 Project 4
Customer Number
G L L
Customer Name
G L L
Address
G L L
City
G L L
Phone
G L L
(Item-Type: FIELD)
Tuesday April 19, 2005 10:15:50 AM
24
Other DBMS Other Data structures SQL DBMS Language Data structures
tructure Generator for Data S tructure Generator for Languages and Languages and DBMS s DBMS s
Cross-
reference facility for Naming S tandards, Aliases, Homonyms, S tandards, Aliases, Homonyms, S ynonyms, Definitions S ynonyms, Definitions
Tuesday April 19, 2005 10:15:50 AM
25
Source Source-
code Databases Databases
“ “Data Lineage Data Lineage” ” Analysis Analysis Path Reports Path Reports Impact Analysis Impact Analysis & & “ “Where Used Where Used” ” Reporting Reporting Visual / Graphic Visual / Graphic Analysis Analysis
Tuesday April 19, 2005 10:15:50 AM
26
Syntactic Mapping Syntactic Mapping Semantic Mapping Semantic Mapping Individual MetaData Definitions Individual MetaData Definitions
Tool A Tool A Tool B Tool B Tool C Tool C Tool D Tool D
Em p_ I D_Code ( ERw in) Soc-Sec-Num ( Oracle) Cust-I D-Code ( I nform atica) Em ployee_ Num ber ( Bus. Objects)
Tuesday April 19, 2005 10:15:50 AM
27
Tuesday April 19, 2005 10:15:50 AM
28
inconsistency of data definitions -
low accuracy & reliability
data rationalization, grouping” ” & recognition of same data elements & recognition of same data elements – – e.g. does S
ecurity Number = e.g. does S
ecurity Number = Employee ID Code ? Employee ID Code ?
data lineage, understanding the “ “ true source of data true source of data” ”
understanding “ “ data consumption data consumption” ” (who uses specific data ? ) (who uses specific data ? )
usability of data for business users
maintenance of data -
governance, policies, & standards
impact analysis reporting for technical & development employees
real-
time metadata reporting vs. batch/ load metadata
regulatory accountability
data security, user access
data governance & stewardship
share & re-
use, integration architectures
versioning, change management, release management of data struct ures, programs, and systems ures, programs, and systems
scope of metadata proj ect, phased implementation
vendor / licensing issues, preferred providers vs. newer technologies
business user participation in metadata management
foreign language interfaces to metadata management (and data management) systems agement) systems
management support ($$$) -
j ustify need for resources & budget
diverse environment, lack of “ “ legacy source legacy source” ” or business definitions for data
integration of ETL rules with BI tools
auditing & compliance, maintenance, versioning of data (metadata) definitions ) definitions
Tuesday April 19, 2005 10:15:50 AM
29
Tuesday April 19, 2005 10:15:50 AM
30
Tuesday April 19, 2005 10:15:50 AM
31
Tuesday April 19, 2005 10:15:50 AM
32
Tuesday April 19, 2005 10:15:50 AM
33
– – typical solutions range from typical solutions range from “ “ Excel Excel spreadsheet(s spreadsheet(s) )” ” to to elaborate custom solutions elaborate custom solutions
Tuesday April 19, 2005 10:15:50 AM
34
the product source code should be “ “ proven yet current proven yet current ” ” meaning that it meaning that it ’ ’ s based on s based on “ “ 2005 technologies 2005 technologies” ”
web-
based user interface (HTML and/ or Java) that offers “ “ discovery discovery” ” , , “ “ intelligent search intelligent search” ” , and , and “ “ mapquest mapquest ” ” functionality functionality
an open, extensible, non-
proprietary metadata repository that is based on standard (widely used) RDBMS such as Oracle, MS is based on standard (widely used) RDBMS such as Oracle, MS-
QL/ S erver, DB2, and/ or S ybase S QL/ S erver, DB2, and/ or S ybase
a repository database schema (“ “ meta meta-
model” ” ) that is not overly ) that is not overly complicated complicated – – should be based more on should be based more on “ “ logical metadata obj ects logical metadata obj ects” ” rather rather than numerous, complex than numerous, complex “ “ physical metadata obj ects physical metadata obj ects” ”
Tuesday April 19, 2005 10:15:50 AM
35
within the
“ Corporate Met aData System” , George t ypes “ profit amount” into a metadata search engine (similar to using Google on the internet).
the meta-data
management system goes
the current applicat ion catalog, and/ or queries a pre-loaded reposit ory of metadata definit ions, looking for definit ions that have “ profit amount” within their name, contents, or relationships to other metadata obj ects. The search engine may also enlist the help of a “ thesaurus” file to look for data elements similarly named (synonyms) to “ profit amount” .
within split-second
t iming, George is presented with a list of 17 different metadata
potentially satisfy his
presented with a standard business definit ion for “ Profit Amount” and the abilit y to find out where is it located, used, and updated throughout the company’ s applicat ion cat alog environment.
similar to using a
card-catalog system in a library, George then browses the descriptions and definitions of each metadata obj ect t o find exactly what he is looking for. George can also physically launch specific development tools and database cat alogs through imbedded hyperlinks t o the t ool’ s file locat ions throughout the corporate net work.
“ George Developer”
has been asked to add a new data element to an important sales report. The field is a derived (calculated) field called “ Profit Amount” . As George starts the proj ect, he asks himself:
calculated ?
somewhere in a database
MetaData Management Application Repository
MetaData Sources: dat abase schemas BI reports dat a models ETL mappings business rules business definitions application portfolio etc …
MetaData S
George is successful
and highly accurate in completing his proj ect under budget and faster then expected. “ Enterprise metadata management is easy t o cost j ustify, because it helps you t o invent ory & catalog your corporate information assets. A good metadata management system is like a “ Google” for your business definitions, data processing systems & applicat ion components. They work like a “ card catalog system” for a library —helping you to inventory, catalog, research and easily access your informat ion assets and applicat ion components. A good metadata system is also like “ MapQuest” because it helps you to understand how definitions and application obj ects are related to each other, how to find them, etc … ”
Typical Reports:
definitions
Tuesday April 19, 2005 10:15:50 AM
36
Data Warehouse MetaData
Metadata definitions are needed to fully understand:
ales” ?
where do they come from ?
some vendors “ automate” this information
Tuesday April 19, 2005 10:15:50 AM
37 MetaData Management Application Repository
MetaData Sources: dat abase schemas BI reports dat a models ETL mappings business rules business definitions application portfolio etc …
Business MetaData in Action
Typical Reports:
within the
“ Corporate MetaData System” , S ally t ypes “ profit amount” into a metadata search engine (similar to using Google on the internet).
the met adata
management system goes
the current applicat ion catalog, and/ or queries a pre-loaded reposit ory of metadata definit ions, looking for definit ions that have “ profit amount” within their name, contents, or relationships to other metadata obj ects. The search engine may also enlist the help of a “ thesaurus” file to look for data elements similarly named (synonyms) to “ profit amount” .
within split-second
timing, Sally is presented with a list of 13 different metadata
potentially satisfy her
he is also presented with a standard business definit ion for “ Profit Amount” and the abilit y to find out where is it located, used, and updated throughout the company’ s applicat ion cat alog environment.
similar to using a
card-catalog system in a library, S ally t hen browses the descriptions and definitions of each metadata obj ect t o find exactly what she is looking for. S ally can also physically launch the specific reporting tools and office documents through imbedded hyperlinks t o the t ool’ s file locat ions throughout the corporate net work.
“ S
ally Analyst” has been asked by her management to find a sales report that calculates “ Profit Amount” . She thinks it is produced by a reporting tool called “ Business Obj ects” . Her key questions are:
located ?
access to the report ?
S
ally is successful and highly accurate in finding the right informat ion, faster then expected.
Tuesday April 19, 2005 10:15:50 AM
38
Tuesday April 19, 2005 10:15:50 AM
39
Most of the Vendors provide web-browser “ S earch & Discovery” User Interfaces
Tuesday April 19, 2005 10:15:50 AM
40
Tuesday April 19, 2005 10:15:50 AM
41
Tuesday April 19, 2005 10:15:50 AM
42
Ant icipat ed Benefit s from my workshops in January & April Ant icipat ed Benefit s from my workshops in January & April
fewer production issues
employee “ “ search & discovery search & discovery” ” and and “ “ self service self service” ” (find key data & reports on their own, fewer corporate (find key data & reports on their own, fewer corporate resources needed, faster response time). resources needed, faster response time). Less intervention needed from IT & support resources. Less intervention needed from IT & support resources. Reduced Reduced search time for Business Users. search time for Business Users.
impact analysis reporting for technical & development employees -
better research, “ “ homework homework” ” , & , & efficiency before working on development proj ects efficiency before working on development proj ects
consistent understanding & definitions of enterprise data
auditability & compliance & compliance
decommission or “ “ turn off turn off ” ” corporate resources (save money) due to centralization, better corporate resources (save money) due to centralization, better access to access to information, or lack of use information, or lack of use
recognize “ “ what we are not using what we are not using” ” , capture usage statistics on resources, reports, documents, dat , capture usage statistics on resources, reports, documents, dat a a structures, etc structures, etc … …
reduce duplication of data -
catalog and re-
use
support for compliance & regulation of key standards – – S OX, Basel, ISO, ACORD, etc S OX, Basel, ISO, ACORD, etc … …
captures knowledge of retiring employees & paid consultants. Facilitates knowledge transfer for new Facilitates knowledge transfer for new employees. employees.
increased data quality. Decreased data redundancy. Decreased data redundancy. Cleaner, better, information. Cleaner, better, information. More easily More easily accessible. accessible.
reduced security threats because of better data management. Better security compliance, classification, Better security compliance, classification, and access management. and access management.
better consistency of data & reporting across divisions or geographic regions. raphic regions.
greater “ “ confidence confidence” ” & reliability of corporate data & reliability of corporate data
Tuesday April 19, 2005 10:15:50 AM
43
S
S
PA Report No. S PA-
450-
876
– – i.e. Determining the effects and depth of changes / enhancements i.e. Determining the effects and depth of changes / enhancements / / new development. Understanding system problems. new development. Understanding system problems.
100 100 Developers / Business Users Developers / Business Users x $75,000 x $75,000
x 10% x 10% Time saved / gained Time saved / gained ========= =========
$750,000 $750,000 Total savings per year Total savings per year
Tuesday April 19, 2005 10:15:50 AM
44
Maj or Financial Company Maj or Financial Company -
New York, NY
Tuesday April 19, 2005 10:15:50 AM
45
MetaData loaded into Rochade Rochade
Tuesday April 19, 2005 10:15:50 AM
46
Tuesday April 19, 2005 10:15:50 AM
47
Tuesday April 19, 2005 10:15:50 AM
48
Tuesday April 19, 2005 10:15:50 AM
49
Tuesday April 19, 2005 10:15:50 AM
50
Tuesday April 19, 2005 10:15:50 AM
51
Conclusion: “Think outside the BOX !” “The best metadata management implementations are about VISION and EXECUTION.”
Tuesday April 19, 2005 10:15:50 AM
52
Tuesday April 19, 2005 10:15:50 AM
53
S
tandish Group International, Inc S
tandish Group International, Inc
Top 5 Reasons For Success
☺ High User Involvement ☺ Committed Executive Support ☺ Clear Statement of Requirements ☺ Proper Planning ☺ Realistic Expectations
Top 5 Reasons For Failure
Lack of User Input or Involvement Incomplete Requirements and
Specifications
Changing Requirements and
Specifications
Lack of Executive Support Technological Incompetence
Tuesday April 19, 2005 10:15:50 AM
54
price / value
unrealistic expectations
perceived complexity of software solution and/ or metadata proj ect/ scope ct/ scope
resource requirements
management support and corporate buy-
in for supporting the proj ect
user acceptance and usage of metadata solution
learning curve
lack of available content
workarounds & patches to metadata solution
internal processes / corporate politics and “ “ ownership
” of specific subj ect
areas
“ forced implementation forced implementation” ” or
“ big brother big brother” ” , , “ “ use it or die use it or die” ” approach approach
Tuesday April 19, 2005 10:15:50 AM
55
properly manage & set expectations -
“ under under-
promise & over-
deliver” ”
find the “ “ killer app killer app” ” to apply metadata to apply metadata mgmnt mgmnt to to
realistic planning, take “ “ baby steps baby steps” ” to success, don to success, don’ ’ t bite off too much t bite off too much too quickly too quickly
initial focus on “ “ low hanging fruit low hanging fruit ” ” and easy wins and easy wins
“ evangelist evangelist ” ” sponsor or corporate champion sponsor or corporate champion
implement good standards & procedures
sustainable Change Management process
leverage your metadata vendor, ask for help, training, support, & & customer reference contacts customer reference contacts
attend conferences and user groups
typical proj ect team = 2-
4 FTE first year, 1-
2 FTE thereafter
Tuesday April 19, 2005 10:15:50 AM
56
Formulate
– –
develop implementation plan & proj ect milestones develop implementation plan & proj ect milestones
Incorporate
– –
design/ build Information Models design/ build Information Models
Populate
– –
run scanners, busses, custom interfaces run scanners, busses, custom interfaces
Integrate
– –
refine/ augment linkages & descriptions refine/ augment linkages & descriptions
Disseminate
– –
“ “ package package” ” the repository for many users the repository for many users
– –
hold internal hold internal “ “ demo days demo days” ” to gain acceptance & use to gain acceptance & use
Reiterate
– –
repeat success across additional repeat success across additional “ “ technologies technologies” ” or
subj ect areas subj ect areas
Tuesday April 19, 2005 10:15:50 AM
57
Tuesday April 19, 2005 10:15:50 AM
58