1 Kinds of Networks Feed-forward Single layer Multi-layer - PDF document

CSE 592 Applications of Artificial Intelligence Neural Networks & Data Mining Henry Kautz Winter 2003 1

Kinds of Networks • Feed-forward • Single layer • Multi-layer • Recurrent Kinds of Networks Kinds of Networks • Feed-forward • Feed-forward • Single layer • Single layer • Multi-layer • Multi-layer • Recurrent • Recurrent 2

Basic Idea: Use error between target and actual output to adjust weights Multiply by η and you get the In other words: training take a step the rule! steepest downhill direction 3

Demos Training Rule Deriviative of the • Single sigmoid unit (a “soft” perceptron) sigmoid gives this ∆ w = ηδ x part i i where the erro r term δ = o ( 1 − o ) ( t − o ) • Multi-Layered network – Compute ∆ values for output units, using observed outputs – For each layer from output back: • Propagate the ∆ values back to previous layer • Update incoming weights 4

Weighted Derivative of output error 5

6

Be careful not to stop too soon! Break! 7

Data Mining ! What is the difference between machine learning and data mining? Data Mining Data Mining Data Mining ! What is the difference between machine ! What is the difference between machine learning and data mining? learning and data mining? ! Scale – DM is ML in the large ! Scale – DM is ML in the large ! Focus – DM is more interested in finding ! Focus – DM is more interested in finding “interesting” patterns than in learning to “interesting” patterns than in learning to classify data classify data ! Marketing! Mining Association Rules in Large Databases Data Mining: ! Introduction to association rule mining Association Rules ! Mining single-dimensional Boolean association rules from transactional databases ! Mining multilevel association rules from transactional databases ! Mining multidimensional association rules from transactional databases and data warehouse ! Constraint-based association mining ! Summary 8

What Is Association Rule Association Rules: Basic Concepts Mining? ! Given: (1) database of transactions, (2) each transaction is ! Association rule mining: a list of items (purchased by a customer in a visit) ! Finding frequent patterns, associations, correlations, or ! Find: all rules that correlate the presence of one set of causal structures among sets of items or objects in items with that of another set of items transaction databases, relational databases, and other ! E.g., 98% of people who purchase tires and auto information repositories. accessories also get automotive services done ! Applications: ! Applications ! Basket data analysis, cross-marketing, catalog design, loss- leader analysis, clustering, classification, etc. ! ? ⇒ Maintenance Agreement (What the store should do to boost Maintenance Agreement sales) ! Examples: ! Home Electronics ⇒ ? (What other products should ! Rule form: “Body → Η ead [support, confidence]”. the store stocks up?) ! buys(x, “diapers”) → buys(x, “beers”) [0.5%, 60%] ! Attached mailing in direct marketing ! major(x, “CS”) ^ takes(x, “DB”) → grade(x, “A”) [1%, 75%] Association Rules: Definitions Association Rules: Definitions II ! Set of items : I = {i 1 , i 2 , …, i m } ! The probability of a set A: ! Set of transactions : D = {d 1 , d 2 , …, d n } ∑  1 if ⊆ X Y ( , ) C A d i Where: ( , ) =  C X Y Each d i ⊆ I P ( A ) = i  0 else | | D ! An association rule : A ⇒ B where A ⊂ I, B ⊂ I, A ∩ B = ∅ ! k-itemset : tuple of items, or sets of items: Example: {A,B} is a 2-itemset • Means that to some extent A • A implies B. The probability of {A,B} is the probability of the set • • Need to measure how strong the A ∪ B, that is the fraction of transactions that contain B implication is. I both A and B. Not the same as P(A ∩ B). Rule Measures: Support and Association Rules: Definitions III Confidence Customer ! Find all the rules X ⇒ Y given buys both thresholds for minimum confidence ! Support of a rule A ⇒ B is the probability of the and minimum support. itemset {A,B}. This gives an idea of how often ! support, s , probability that a the rule is relevant. transaction contains {X, Y} Y: ! confidence, c, conditional ! support(A ⇒ B ) = P({A,B}) Customer X: Customer probability that a transaction buys diaper buys beer ! Confidence of a rule A ⇒ B is the conditional having X also contains Y With minimum support 50%, probability of B given A. This gives a measure Transaction ID Items Bought and minimum confidence of how accurate the rule is. 2000 A,B,C 50%, we have 1000 A,C ! confidence(A ⇒ B) = P(B|A) ! A ⇒ C (50%, 66.6%) 4000 A,D = support({A,B}) / support(A) ! C ⇒ A (50%, 100%) 5000 B,E,F 9

Mining Association Rules in Association Rule Mining: A Road Map Large Databases ! Boolean vs. quantitative associations (Based on the types of values ! Association rule mining handled) ! Mining single-dimensional Boolean association rules ! buys(x, “SQLServer”) ^ buys(x, “DMBook”) → buys(x, “DBMiner”) [0.2%, 60%] from transactional databases ! age(x, “30..39”) ^ income(x, “42..48K”) → buys(x, “PC”) [1%, 75%] ! Mining multilevel association rules from transactional ! Single dimension vs. multiple dimensional associations (see ex. Above) databases ! Single level vs. multiple-level analysis ! Mining multidimensional association rules from ! What brands of beers are associated with what brands of diapers? transactional databases and data warehouse ! Various extensions and analysis ! Correlation, causality analysis ! From association mining to correlation analysis ! Association does not necessarily imply correlation or causality ! Constraint-based association mining ! Maxpatterns and closed itemsets ! Summary ! Constraints enforced ! E.g., small sales (sum < 100) trigger big buys (sum > 1,000)? Mining Frequent Itemsets: the Mining Association Rules—An Example Key Step Transaction ID Items Bought ! Find the frequent itemsets : the sets of items that have Min. support 50% 2000 A,B,C at least a given minimum support Min. confidence 50% 1000 A,C ! A subset of a frequent itemset must also be a Frequent Itemset Support 4000 A,D {A} 75% frequent itemset 5000 B,E,F {B} 50% ! i.e., if { A, B } is a frequent itemset, both { A } and { B } {C} 50% should be a frequent itemset For rule A ⇒ C : {A,C} 50% ! Iteratively find frequent itemsets with cardinality support = support({ A, C }) = 50% from 1 to k (k- itemset ) confidence = support({ A, C })/support({ A }) = 66.6% ! Use the frequent itemsets to generate association The Apriori principle: rules. Any subset of a frequent itemset must be frequent The Apriori Algorithm The Apriori Algorithm — Example Database D ! Join Step: C k is generated by joining L k-1 with itself itemset sup. itemset sup. L 1 TID Items C 1 {1} 2 ! Prune Step: Any (k-1)-itemset that is not frequent cannot be {1} 2 100 1 3 4 {2} 3 a subset of a frequent k-itemset {2} 3 Scan D 200 2 3 5 {3} 3 ! Pseudo-code: {3} 3 300 1 2 3 5 {4} 1 C k : Candidate itemset of size k {5} 3 400 2 5 {5} 3 L k : frequent itemset of size k C 2 itemset C 2 itemset sup L 1 = {frequent items} itemset sup {1 2} L 2 Scan D {1 2} 1 for ( k = 1; L k != ∅ ; k ++) do begin {1 3} {1 3} 2 {1 3} 2 C k+1 = candidates generated from L k {1 5} for each transaction t in database do {1 5} 1 {2 3} 2 increment the count of all candidates in C k+1 {2 3} {2 3} 2 {2 5} 3 that are contained in t {2 5} {2 5} 3 {3 5} 2 L k+1 = candidates in C k+1 with min_support {3 5} {3 5} 2 end return ∪ k L k ; itemset L 3 itemset sup C 3 Scan D {2 3 5} {2 3 5} 2 10