Missing Data Imputation using Optimal Transport Boris Muzellec Julie - PowerPoint PPT Presentation

Dec 12, 2023 •366 likes •569 views

Missing Data Imputation using Optimal Transport Boris Muzellec Julie Josse Claire Boyer Marco Cuturi <latexit

Missing Data Imputation using Optimal Transport Boris Muzellec Julie Josse Claire Boyer Marco Cuturi
<latexit sha1_base64="xfKy83h87erRWERr/xzJ2E0Lv+E=">AB7HicbVDLTgJBEOzF+IL9ehlIjHxRHaNiR6JXjxiIo8ENmR26IUJM7ObmVkiIfyE8YZe/R3P/o0D7kHBOlV3Vae7K0oFN9b3v7zCxubW9k5xt7S3f3B4VD4+aZok0wbLBGJbkfUoOAKG5Zbge1UI5WRwFY0ul/orTFqwxP1ZCcphpIOFI85o9a12l0u3RY0vXLFr/pLkHUS5KQCOeq98me3n7BMorJMUGM6gZ/acEq15UzgrNTNDKaUjegAO/0xT42iEk04fV6ePCMXcaKJHSJZ1r/tUyqNmcjIeS1Q7OqLZr/aZ3MxrfhlKs0s6iYszgtzgSxCVl8TvpcI7Ni4ghlmrtDCRtSTZl1+ZRcAsHqv+ukeVUN/GrweF2p3eVZFOEMzuESAriBGjxAHRrAQMALzOHNU96rN/fef6wFL585hT/wPr4BZIGPEQ=</latexit> <latexit sha1_base64="xfKy83h87erRWERr/xzJ2E0Lv+E=">AB7HicbVDLTgJBEOzF+IL9ehlIjHxRHaNiR6JXjxiIo8ENmR26IUJM7ObmVkiIfyE8YZe/R3P/o0D7kHBOlV3Vae7K0oFN9b3v7zCxubW9k5xt7S3f3B4VD4+aZok0wbLBGJbkfUoOAKG5Zbge1UI5WRwFY0ul/orTFqwxP1ZCcphpIOFI85o9a12l0u3RY0vXLFr/pLkHUS5KQCOeq98me3n7BMorJMUGM6gZ/acEq15UzgrNTNDKaUjegAO/0xT42iEk04fV6ePCMXcaKJHSJZ1r/tUyqNmcjIeS1Q7OqLZr/aZ3MxrfhlKs0s6iYszgtzgSxCVl8TvpcI7Ni4ghlmrtDCRtSTZl1+ZRcAsHqv+ukeVUN/GrweF2p3eVZFOEMzuESAriBGjxAHRrAQMALzOHNU96rN/fef6wFL585hT/wPr4BZIGPEQ=</latexit> <latexit sha1_base64="xfKy83h87erRWERr/xzJ2E0Lv+E=">AB7HicbVDLTgJBEOzF+IL9ehlIjHxRHaNiR6JXjxiIo8ENmR26IUJM7ObmVkiIfyE8YZe/R3P/o0D7kHBOlV3Vae7K0oFN9b3v7zCxubW9k5xt7S3f3B4VD4+aZok0wbLBGJbkfUoOAKG5Zbge1UI5WRwFY0ul/orTFqwxP1ZCcphpIOFI85o9a12l0u3RY0vXLFr/pLkHUS5KQCOeq98me3n7BMorJMUGM6gZ/acEq15UzgrNTNDKaUjegAO/0xT42iEk04fV6ePCMXcaKJHSJZ1r/tUyqNmcjIeS1Q7OqLZr/aZ3MxrfhlKs0s6iYszgtzgSxCVl8TvpcI7Ni4ghlmrtDCRtSTZl1+ZRcAsHqv+ukeVUN/GrweF2p3eVZFOEMzuESAriBGjxAHRrAQMALzOHNU96rN/fef6wFL585hT/wPr4BZIGPEQ=</latexit> The missing data issue • Big data is plagued with missing values • What to do? Option 1: Remove entries with missing values information loss, not sustainable ⇒ = Example with 25% missing rate: 2d 3d 6d 10d With 1% missing rate: 5d: 95% rows kept 300d: 5% rows kept Option 2: Impute with reasonable guesses
Outline 1. Missing data and Optimal Transport 2. Non-parametric imputation with OT 3. Fitting parametric imputation models with OT
How to impute? - Mean imputation - Regression (conditional expectation) Deforms joint and marginal distributions Preserves distributions • Using a conditional model: - With logistic, multinomial, Poisson regressions: R’s mice (Van Buuren, 2011) • Assuming a joint model: - EM + Gaussian distribution: Amelia (Honacker et al., 2011) - Low-rank models: Softimpute (Mazumder et al., 2010) - VAE and GAN: MIWAE (Mattei & Frellsen, 2019), GAIN (Yoon et al., 2018) - … This work: Preserves distributions Parametric assumption not necessary

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data

Missing Data and Imputation NINA ORWITZ OCTOBER 30 TH , 2017 Outline Types of missing data Simple methods for dealing with missing data Single and multiple imputation R example Missing data is a complex problem We must consider:

377 views • 22 slides

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1.

Multiple Imputation for Missing Data in KLoSA Juwon Song Korea University and UCLA Contents 1. Missing Data and Missing Data Mechanisms 2. Imputation 3. Missing Data and Multiple Imputation in Baseline KLoSA Data Missing Data and Multiple

634 views • 35 slides

Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES,

WHAT ARE MISSING DATA ? HOW TO TREAT MISSING DATA ? LONGITUDINAL DATA, CAUSALITY, & ETHICS Missing data and data imputation with the Swiss Household Panel Andr Berchtold LIVES, LINES, Universit de Lausanne FORS SHP workshop June

1.2k views • 108 slides

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 ,

Attention-based Learning for Missing Data Imputation in HoloClean Richard Wu 1 , A oqian Zhang 1 , Ihab F. Ilyas 1 Theodoros Rekatsinas 2 1 2 Problem Missing data is a persistent problem in many fields Sciences Data mining

322 views • 28 slides

Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing

DataCamp Dealing With Missing Data in R DEALING WITH MISSING DATA IN R Performing and tracking imputation Nicholas Tierney Statistician DataCamp Dealing With Missing Data in R Lesson overview Using imputations to understand data structure

647 views • 53 slides

Handling missing data in Stata: Imputation and likelihood-based approaches Rose Medeiros

Introduction Multiple Imputation Full information maximum likelihood Conclusion Handling missing data in Stata: Imputation and likelihood-based approaches Rose Medeiros StataCorp LP 2016 Swiss Stata Users Group meeting Medeiros Handling

841 views • 33 slides

MixtComp software: Model-based clustering/imputation with mixed data, missing data and uncertain

Classifications(s): overview Mixture model solution Estimation Clustering with MixtComp Imputation with MixtComp Conclusion MixtComp software: Model-based clustering/imputation with mixed data, missing data and uncertain data

966 views • 58 slides

N 1 N 1 IV x i y i s i z i s i z i (2) 3. Imputation i 1

A Course in Applied Econometrics 1 . When Can Missing Data be Ignored ? Lecture 18 : Missing Data Linear model with IVs: y i x i u i , (1) Jeff Wooldridge IRP Lectures, UW Madison, August 2008 where x i is 1 K , instruments z

236 views • 10 slides

Recognition of Reverberant Speech by Missing Data Imputation and NMF Feature Enhancement Heikki

Recognition of Reverberant Speech by Missing Data Imputation and NMF Feature Enhancement Heikki Kallasjoki , Jort F . Gemmeke, Kalle J. Palomki, Amy V. Beeston, Guy J. Brown Department of Signal Processing and Acoustics Aalto University,

687 views • 36 slides

Reference based multiple imputation; for sensitivity analysis of clinical trials with missing

Reference based multiple imputation; for sensitivity analysis of clinical trials with missing data Suzie Cro MRC Clinical Trials Unit at UCL The London School of Hygiene and Tropical Medicine Outline Reference based multiple imputation;

586 views • 39 slides

Incremental Algorithms for Missing Data Imputation based on Recursive Partitioning Claudio

Incremental Algorithms for Missing Data Imputation based on Recursive Partitioning Claudio Conversano Department of Economics University of Cassino, via M. Mazzaroppi, I-03043 Cassino (FR) c.conversano@unicas.it, http//cds.unina.it/~conversa

810 views • 38 slides

Accurate Regression Parameters and Summary Statistics Estimation in Data with Censored Missing

Accurate Regression Parameters and Summary Statistics Estimation in Data with Censored Missing Values Yuliya Karpievitch Imputation And Censored Values Yuliya Karpievitch Missing Values and Imputation Some algorithms expect a

531 views • 16 slides

Continuous Imputation of Missing Values in Streams of Pattern-Determining Time Series Kevin

Continuous Imputation of Missing Values in Streams of Pattern-Determining Time Series Kevin Wellenzohn 1 ohlen 1 Michael H. B os 2 Johann Gamper 2 Hannes Mitterer 2 Anton Dign 1 Department of Computer Science University of Zurich 2 Faculty

356 views • 31 slides

Martingale Optimal Transport in Higher Hadrien De March Dimension Optimal transport

MOTDim d Martingale Optimal Transport in Higher Hadrien De March Dimension Optimal transport Formulation of the problems Optimal Hadrien De March transport in practice Martingale CMAP, Ecole Polytechnique optimal transport

687 views • 24 slides

Imputation of missing covariates: when standard methods may fail Nicole S. Erler 1 , 2 , Dimitris

Imputation of missing covariates: when standard methods may fail Nicole S. Erler 1 , 2 , Dimitris Rizopoulos 1 , Oscar H. Franco 2 , Emmanuel M.E.H. Lesaffre 1 , 3 1 Department of Biostatistics, Erasmus MC, Rotterdam, the Netherlands 2 Department

678 views • 38 slides

Optimal Transport for structured data with application on graphs Titouan Vayer Joint work with

Optimal Transport for structured data with application on graphs Titouan Vayer Joint work with Laetitia Chapel, Remi Flamary, Romain Tavenard and Nicolas Courty A novel distance between labeled graphs based on optimal transport Contributions:

357 views • 15 slides

Internet Governance in April 2016 26 April 2016 Main events in April 3-6 Apr: Global Privacy

Internet Governance in April 2016 26 April 2016 Main events in April 3-6 Apr: Global Privacy Summit 2016 (Washington DC) 3-8 Apr: IETF 95 (Buenos Aires) 4-6 Apr: IGF Open Consultations and MAG meeting (Geneva) 5-6 Apr: Launch of Council of

426 views • 20 slides

Nzt ut Uf int Zf Mst State evolution ME IT CZ's.is Z U U t em ECZE.is't's.is ECMEMssl

Halo 6 ELI lb maximize generalized p spin W Ico Wo GOE n SK Hnlo motel Zt ERN TEL 0,8 28 HIP I 8 I Wftcz't Ef dt fs.gl s zo Nlo S 4v Enl Eic Lene p line ENKE ENZI ZI I Z Z LEI Nco QI Zf Z I ELICZ s ZITI Q is.t.is Want fecz.at m

499 views • 5 slides

Internet of Things: doorway to a new reality Maarten Botterman, GNKS Consult GARNTER Hype Cycle

Internet of Things: doorway to a new reality Maarten Botterman, GNKS Consult GARNTER Hype Cycle 2014 2 www.smart-action.eu More and more connected 3 www.smart-action.eu A smarter world for all? 4 Benefits and challenges 5

717 views • 28 slides

2 3

2 3 4 5 6 7 Katunayake Export Processing Zone Presidential Secretariat Office, Galle Face 8

319 views • 11 slides

ON THE AGENDA What immediate responses have governments and mining industries taken to respond to

ON THE AGENDA What immediate responses have governments and mining industries taken to respond to COVID-19 challenges faced by workers? What does the future outlook for employment and skills development look like? How can we balance the need to

582 views • 16 slides

The impacts of external funding for district governments on tax collection and public goods

The impacts of external funding for district governments on tax collection and public goods provision in Ghana Edward Asiedu (University of Ghana) Dan Pavelesku (World Bank) Ryoko Sato (World Bank) Tomomi Tanaka (World Bank) Research

104 views • 9 slides

Open Board Forum Survey Results How are you connected to ISOC? 0 100 200 300 400 614

20 June 2018 Open Board Forum Survey Results How are you connected to ISOC? 0 100 200 300 400 614 Chapter member 346 responses Individual member 290 (~1.5% of registered members) SIG member 53 Notable other: - Youth@IGF

146 views • 11 slides

Support for the Hanoi Omega-Automata Format http://adl.github.io/hoaf/support.html ltl2dstar 0.5.3

Spot 2.0 A C++ library for model checking and -automata manipulation Alexandre D uret -L utz Alexandre L ewkowicz Amaury F auchille tienne R enault Laurent X u Thibaud M ichaud Monday, October 17th 1 / 19 A Generalized View of -Automata

500 views • 35 slides