Lecture 4: Rule-based classification and regression Felix Held, - - PowerPoint PPT Presentation
Lecture 4: Rule-based classification and regression Felix Held, Mathematical Sciences MSA220/MVE440 Statistical Learning for Big Data 1st April 2019 Amendment: Bias-Variance Tradeoff ] Variance Bias 2 Error Model complexity Overfit Underfit
2
βΆ Irreducible error cannot be changed βΆ Bias and variance of Λ
βΆ For a consistent estimator Λ
βΆ In many cases:
βΆ Caution: Theoretical guarantees are often dependent on
βΆ Popular because explicit formulas (or approximations)
βΆ Uses the most data for training possible βΆ More variable than π-fold CV for π < π since only one data
βΆ In praxis: Try out different values for π. Be cautious if
βΆ kNN creates local neighbourhoods in feature space and
βΆ Logistic regression divides feature space implicitly by
βΆ Discriminant analysis creates an explicit model of the
βΆ Complexity of partitioning:
βΆ Classification and Regression Trees create a sequence of
π (for continuous features) or a partition of the
π}
π}
βΆ Three common measures to determine impurity in a
βΆ All criteria are zero when only one class is present and
βΆ For regression trees the decrease in mean squared error
βΆ Minimum size of leaf nodes (e.g. 5 samples per leaf node) βΆ Minimum decrease in impurity (e.g. cutoff at 1%) βΆ Maximum tree depth, i.e. number of splits (e.g. maximum
βΆ Maximum number of leaf nodes
βΆ Pro: Outcome is easily interpretable βΆ Pro: Can easily handle missing data βΆ Neutral: Only suitable for axis-parallel decision
βΆ Con: Features with more potential splits have a higher
βΆ Con: Prone to overfitting/unstable (only the best feature
βΆ Tuning of stopping criteria: These can easily lead to early
βΆ Pruning: Build a max tree first. Then reduce its size by
βΆ Ensemble methods: Examples are bagging, boosting,
βΆ A common strategy is cost-complexity pruning. βΆ For a given π½ > 0 and a tree π its cost-complexity is
βΆ It can be shown that successive subtrees ππ of the max
βΆ The tree with the lowest cost-complexity is chosen
βΆ Through theoretical results (e.g. linear models) if
βΆ Linearisation for more complex models (e.g. nonlinear or
βΆ Nonparametric approaches using the data (e.g.
βΆ πΆ should be large (in the 1000β10000s) βΆ The distribution of
βΆ The bootstrap makes exactly one strong assumption:
1Check out this blog post!
βΆ Orange histogram shows original sample βΆ Blue line is the true density βΆ Black outlined histogram shows a bootstrapped sample βΆ Vertical lines are the mean of π¦ (dashed) and the 99% quantile
βΆ Different sampling strategies. Some examples:
βΆ π-out-of-π bootstrap: Draw π < π samples without
βΆ Draw from a smooth density estimate of the data βΆ Draw from a parametric distribution fitted to the original
βΆ Normal approximation doesnβt always apply and
βΆ Bootstrap-t: Instead of normal quantiles, estimate
βΆ Many other alternatives exist β¦
βΆ Number of samples needs to be quite large βΆ Extreme values (minimum, maximum very small or large
βΆ Many basic CI estimation algorithms assume that the
βΆ Bagging using averages approximates
βΆ For the conditional expected error in squared error loss
βΆ Some notes:
βΆ Remember the graphs of kNN from last lecture: Noisy
βΆ Bagging shows no effect on linear models
βΆ Bootstrap samples are correlated and increase total
βΆ Decreasing correlation between bootstrap samples would
βΆ During bootstrapping for large enough π, each sample has
βΆ For bagging the remaining samples are out-of-bag. βΆ These out-of-bag samples for tree ππ can be used as a test
βΆ Permute variable π in the out-of-bag samples and
1
βΆ The increase in error
1
βΆ Direct partitioning of feature space is a complex task βΆ Simplifications in form of binary splits resulting in tree
βΆ High interpretability of CART, but also high variability βΆ Random Forests tackles variance reduction though