a motivation for polynomial regression
play

A motivation for polynomial regression We have obtained input-output - PowerPoint PPT Presentation

Nov 02, 2023 •282 likes •450 views

A motivation for polynomial regression We have obtained input-output pairs { ( x t , y t ) } t over the last 200 time steps and aim to model their relationship 3.0 3.0 3.0 2.5 2.5 2.5

  2. A motivation for polynomial regression We have obtained input-output pairs { ( x t , y t ) } t over the last 200 time steps and aim to model their relationship 3.0 3.0 3.0 ● ● ● ● ● ● 2.5 ● ● 2.5 2.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2.0 ● ● ● 2.0 2.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.5 ● 1.5 1.5 ● y ● ● y ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.0 ● 1.0 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 ● 0.5 0.5 ● ● ● ● ● ● ● ● ● 0.0 0.0 0.0 0 1 2 3 4 5 0 0 1 1 2 2 3 3 4 4 5 5 x x Using linear regression does not look like such a good idea... 2/14

  3. Linear regression A simple linear relation is assumed between x and y , i.e., y t = β 0 + β 1 x t + ε t , t = t n − n , . . . , t n where β 0 and β 1 are the model parameters (called intercept and slope ) ε t is a noise term, which you may see as our forecast error we want to minimize The linear regression model can be reformulated in a more compact form as y t = β ⊤ x t + ε t , t = t n − n , . . . , t n with � � � 1 � β 0 β = , x t = β 1 x t 3/14

  4. Least Squares (LS) estimation Now we need to find the best value of β that describes this cloud of point Under a number of assumptions, which we overlook here, the (best) model parameters ˆ β can be readily obtained with Least-Squares (LS) estimation The Least-Squares (LS) estimate ˆ β of the linear regression model parameters is given by � 2 t ε i 2 = arg min β � ˆ y t − β ⊤ x t = ( X ⊤ X ) − 1 X ⊤ y β = arg min β � � t with     1 x t n − n y t n − n � ˆ 1 x t n − n +1 y t n − n +1 � β 0     ˆ β = , X =  , y = . . .     ˆ . . . β 1     . . .    1 x t n y t n 4/14

  5. Extending to polynomial regression We could also assume more generally a polynomial relation between x and y , i.e., y t = β 0 + � P p =1 β p x p t + ε t , t = t n − n , . . . , t n where β p , p = 0 , . . . , P are the model parameters ε t is a noise term, which you may see as our forecast error we want to minimize This polynomial regression can be reformulated in a more compact form as y t = β ⊤ x t + ε t , i = t n − n , . . . , t n with     β 0 1 β 1 x t     β =  , x t =     · · · · · ·    x P β P t 5/14

  6. Least Squares (LS) estimation As the model is linear we can still use LS estimation! The Least-Squares (LS) estimate ˆ β of the linear regression model parameters is given by � 2 t ε t 2 = arg min β � ˆ y t − β ⊤ x t = ( X ⊤ X ) − 1 X ⊤ y � � β = arg min β t with x 2  x P    ˆ 1 x t n − n . . . y t n − n   β 0 t n − n t n − n x 2 x P 1 x t n − n +1 . . . y t n − n +1 ˆ β 1  t n − n +1 t n − n +1    ˆ   β =  , X =  , y = . . . . .       . . . . . · · ·     . . . . .     ˆ β P x 2 x P 1 x t n . . . y t n t n t n 6/14

  7. Going back to our example We apply polynominal regression with P = 2 (quadratic) and P = 3 (cubic) 3.0 3.0 3.0 3.0 ● ● ● ● ● ● ● ● ● ● 2.5 2.5 ● 2.5 2.5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2.0 2.0 ● ● 2.0 2.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.5 1.5 ● 1.5 1.5 ● ● ● ● ● y ● y ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.0 1.0 ● 1.0 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.5 0.5 ● 0.5 0.5 ● ● ● ● ● ● ● ● ● 0.0 0.0 0.0 0.0 0 0 1 1 2 2 3 3 4 4 5 5 0 0 1 1 2 2 3 3 4 4 5 5 x x They both look quite nicer than the simple linear fit We are lucky here that the relationship truly is quadratic... if fitting higher-order polynominals, ˆ β i = 0 , p > 2 In general, higher-order may yield spurious results(!) 7/14

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A

Lecture 6: Multiple Linear Regression, Polynomial Regression and Model Selection CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Announcements Section : Friday 1:30-2:45pm : @ MD 123 (only this Friday) A-section: Today:

921 views • 65 slides
with Polynomial Filters Josiah Manson and Scott Schaefer Texas A&M University Motivation

with Polynomial Filters Josiah Manson and Scott Schaefer Texas A&M University Motivation

Analytic Rasterization of Curves with Polynomial Filters Josiah Manson and Scott Schaefer Texas A&M University Motivation Motivation Motivation Motivation Motivation Motivation Motivation Constant colors Color gradients Motivation

976 views • 70 slides
Contents 1 Introduction 1 2 Linear Transformations 2 3 Polynomial Regression 3 4

Contents 1 Introduction 1 2 Linear Transformations 2 3 Polynomial Regression 3 4

Diagnostics and Transformations Part 2 Contents 1 Introduction 1 2 Linear Transformations 2 3 Polynomial Regression 3 4 Orthogonal Polynomials 6 5 Empirically Motivated Nonlinear Transformations 7 5.1 Mosteller-Tukey Bulging

382 views • 15 slides
Review:$$ Model$Selec5on$ Training$vs.$Test$errors$ Polynomial$regression$

Review:$$ Model$Selec5on$ Training$vs.$Test$errors$ Polynomial$regression$

Review:$$ Model$Selec5on$ Training$vs.$Test$errors$ Polynomial$regression$ Model$complexity:$Degree$of$polynomial$ Is$larger$always$be^er?$ Test$ Error$ Train$ Model$complexity$ Model$Selec5on$Criterion$

404 views • 14 slides
Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/ Linear Regression with Polynomial Features , Cross Validation, and Hyperparameter Selection Many slides attributable to: Prof. Mike Hughes Erik Sudderth

298 views • 28 slides
Polynomial Selection Thorsten Kleinjung Ecole Polytechnique F ed erale de Lausanne

Polynomial Selection Thorsten Kleinjung Ecole Polytechnique F ed erale de Lausanne

Polynomial Selection Thorsten Kleinjung Ecole Polytechnique F ed erale de Lausanne Contents Brief summary of polynomial selection (no root sieve) Motivation (lattice sieving, monic algebraic polynomial) General case (reduction to

411 views • 30 slides
h ( x ) } Although we can generally find the best set of weights efficiently, the exact form of

h ( x ) } Although we can generally find the best set of weights efficiently, the exact form of

Practical Use of Linear Regression 25 25 25 20 20 20 Sales Sales Sales 15 15 15 10 10 10 5 5 5 Class #03: Linear and Polynomial Regression Models 0 50 100 200 300 0 10 20 30 40 50 0 20 40 60 80 100 TV Radio

207 views • 3 slides
Beta Regression: Summary Shaken, Stirred, Mixed, and Partitioned Achim Zeileis, Francisco

Beta Regression: Summary Shaken, Stirred, Mixed, and Partitioned Achim Zeileis, Francisco

Overview Motivation Shaken or stirred: Single or double index beta regression for mean and/or precision in betareg Mixed: Latent class beta regression via flexmix Partitioned: Beta regression trees via party Beta Regression: Summary Shaken,

455 views • 6 slides
On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine

On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine

Index On-line Support Vector Motivation and antecedents Formulation of SVM regression Machine Regression Characterization of vectors in SVM regression Procedure for Adding one vector Procedure for Removing one vector

520 views • 15 slides
Binary Decision Diagrams INF5140 1 Motivation Fig 1. Polynomial representation Boolean

Binary Decision Diagrams INF5140 1 Motivation Fig 1. Polynomial representation Boolean

Binary Decision Diagrams INF5140 1 Motivation Fig 1. Polynomial representation Boolean functions Extremely useful Some areas of application: Cryptography Propositional logic Verification/model checking Fig

322 views • 16 slides
Introduction to Machine Learning Polynomial Regression Models Learning goals Understand how to

Introduction to Machine Learning Polynomial Regression Models Learning goals Understand how to

Introduction to Machine Learning Polynomial Regression Models Learning goals Understand how to add flexibility to the linear model by using 2.0 f(x) for d = 1 (linear) f(x) for d = 5 1.5 polynomials f(x) for d = 25 1.0 0.5 y Understand

496 views • 8 slides
STAT 215 Polynomials, Multicollinearity Colin Reimer Dawson Oberlin College 4 November 2016

STAT 215 Polynomials, Multicollinearity Colin Reimer Dawson Oberlin College 4 November 2016

Outline Polynomial Regression Interactions Multicollinearity STAT 215 Polynomials, Multicollinearity Colin Reimer Dawson Oberlin College 4 November 2016 Outline Polynomial Regression Interactions Multicollinearity Outline Polynomial

517 views • 28 slides
CS 445 Introduction to Machine Learning Logistic Regression Instructor: Dr. Kevin Molloy Review

CS 445 Introduction to Machine Learning Logistic Regression Instructor: Dr. Kevin Molloy Review

CS 445 Introduction to Machine Learning Logistic Regression Instructor: Dr. Kevin Molloy Review Linear regression Finding the weights to assign to a polynomial so that the resulting line minimizes the "loss". ( ! ,

481 views • 11 slides
Diagnostics and Transformations Part 2 Bivariate Linear Regression James H. Steiger

Diagnostics and Transformations Part 2 Bivariate Linear Regression James H. Steiger

Introduction Linear Transformations Polynomial Regression Orthogonal Polynomials Empirically Motivated Nonlinear Transformations Diagnostics and Transformations Part 2 Bivariate Linear Regression James H. Steiger Department of Psychology

961 views • 46 slides
Polynomial Time corresponds to Solutions of Polynomial Ordinary Differential Equations of

Polynomial Time corresponds to Solutions of Polynomial Ordinary Differential Equations of

Polynomial Time corresponds to Solutions of Polynomial Ordinary Differential Equations of Polynomial Length Olivier Bournez, Daniel Graa and Amaury Pouly July 13, 2015 Main result and consequences Theorem (Informal) PTIME = PIVP of

633 views • 47 slides
Multiple Regression Analysis Independent Variables Mechanics and Interpretation of OLS

Multiple Regression Analysis Independent Variables Mechanics and Interpretation of OLS

Motivation for Multiple Regression The Model with k Multiple Regression Analysis Independent Variables Mechanics and Interpretation of OLS Interpreting the OLS Caio Vigo Regression Line The Expected Value of the OLS The University of

2.75k views • 89 slides
Optimal state estimation for numerical weather prediction using reduced order models A.S. Lawless

Optimal state estimation for numerical weather prediction using reduced order models A.S. Lawless

4D-Var Model reduction in DA Unstable systems Numerical results Conclusions Optimal state estimation for numerical weather prediction using reduced order models A.S. Lawless 1 , C. Boess 1 , N.K. Nichols 1 and A. Bunse-Gerstner 2 1 School of

737 views • 51 slides
Theory of Generalized Linear Models If Y has a Poisson distribution with parameter then P (

Theory of Generalized Linear Models If Y has a Poisson distribution with parameter then P (

Theory of Generalized Linear Models If Y has a Poisson distribution with parameter then P ( Y = y ) = y e y ! for y a non-negative integer. We can use the method of maximum likelihood to estimate if we have a sample Y 1 , . .

1.06k views • 18 slides
Detection and Estimation Theory Lecture 12 Mojtaba Soltanalian- UIC msol@uic.edu

Detection and Estimation Theory Lecture 12 Mojtaba Soltanalian- UIC msol@uic.edu

Detection and Estimation Theory Lecture 12 Mojtaba Soltanalian- UIC msol@uic.edu http://msol.people.uic.edu Based on ECE 531 Slides- 2011 (Prof. Natasha Devroye) Summary Need (at least some) information about the pdf. Least Square

437 views • 19 slides
On the Statistical Rate of Nonlinear Recovery in Generative Models with Heavy-tailed Data Xiaohan

On the Statistical Rate of Nonlinear Recovery in Generative Models with Heavy-tailed Data Xiaohan

On the Statistical Rate of Nonlinear Recovery in Generative Models with Heavy-tailed Data Xiaohan Wei , Zhuoran Yang, and Zhaoran Wang University of Southern California, Princeton University and Northwestern University June 12th, 2019 Generative

491 views • 17 slides
Applied Machine Learning Applied Machine Learning Regularization Siamak Ravanbakhsh Siamak

Applied Machine Learning Applied Machine Learning Regularization Siamak Ravanbakhsh Siamak

Applied Machine Learning Applied Machine Learning Regularization Siamak Ravanbakhsh Siamak Ravanbakhsh COMP 551 COMP 551 (winter 2020) (winter 2020) 1 Learning objectives Learning objectives Basic idea of overfitting and underfitting

1.48k views • 135 slides
Analysing Gene Expression Data Using Gaussian Processes Lorenz Wernisch School of

Analysing Gene Expression Data Using Gaussian Processes Lorenz Wernisch School of

Analysing Gene Expression Data Using Gaussian Processes Lorenz Wernisch School of Crystallography Birkbeck College London p.1/35 Overview Gene regulatory networks, microarrays Time-series analysis by linear regression Bayesian

598 views • 35 slides
Introduction to Mobile Robotics Basics of LSQ Estimation, Geometric Feature Extraction Wolfram

Introduction to Mobile Robotics Basics of LSQ Estimation, Geometric Feature Extraction Wolfram

Introduction to Mobile Robotics Basics of LSQ Estimation, Geometric Feature Extraction Wolfram Burgard, Cyrill Stachniss, Maren Bennewitz, Giorgio Grisetti, Kai Arras Slides by Kai Arras Last update: June 2010 1 Feature Extraction:

439 views • 20 slides
Module 4 Markov Processes CS 886 Sequential Decision Making and Reinforcement Learning

Module 4 Markov Processes CS 886 Sequential Decision Making and Reinforcement Learning

Module 4 Markov Processes CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Sequential Decision Making In general: exponentially large decision tree s1 a b . 9 . 1 . 2 . 8 s2 s3 s12 s13 a b a b

364 views • 12 slides

More recommend