Whos in the Picture? Tamara L. Berg Cse595 Words & Pictures - PowerPoint PPT Presentation

Who’s in the Picture? Tamara L. Berg Cse595 Words & Pictures

Face Recognition Datasets PIE database Yale Face Database M2VTS

What the world really looks like (scenes from an airport)

Challenging Even Up Close Face Scanner Failed -- Wired News 5/16/02

A More Realistic Dataset France's Amelie Mauresmo has pulled out of the Australian Open (news - Actress Winona Ryder (news) reacts to web sites) because of a knee injury. The remarks by prosecutor Ann Rundle during the French number one, ranked sixth in the sentencing hearing in her felony shoplifting case world, has been struggling with a Friday, Dec. 6, 2002 at the Beverly Hills, Calif., painful knee cap for two months and courthouse. At right is Ryder's attorney Mark said on December 18, 2002 she would Geragos . Ryder was sentenced to three years of not be fit in time to resume playing in probation and was ordered to perform 480 hours January. Mauresmo is shown during the of community service. (AP Kremlin Cup, Oct. 4. (Grigory Dukor/ Photo/Steve Grayson, POOL) Reuters)

Dataset (cont) Doctor Nikola shows a fork that was removed from an Israeli woman who swallowed it while trying to catch a bug President George W. Bush waves as he leaves that flew in to her mouth, in Poriah Hospital the White House for a day trip to North northern Israel July 10, 2003. Doctors Carolina, July 25, 2002. A White House performed emergency surgery and removed spokesman said that Bush would be compelled the fork. (Reuters) to veto Senate legislation creating a new department of homeland security unless changes are made. (Kevin Lamarque/Reuters)

Dataset (cont.) Actor Arnold Schwarzenegger Producer and director Bruce Paltrow has died discusses his plans for his campaign with at the age of 58 in Rome, Italy, the U.S. reporters after he announced that he will Consulate said on October 3, 2002. Paltrow be a candidate for governor of California had suffered from throat cancer for several in the recall election, August 6, 2003. years, but the cause of his death was not Schwarzenegger ended weeks of immediately known. He is seen with his speculation about his candidacy for the daughter actress Gwyneth Paltrow after the October 7 referendum. (Fred Prouser/ Academy Awards in Los Angles in March 21, Reuters 1999 file photo. (Fred Prouser/Reuters)

Names and Faces President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/ Reuters

Intuition George Bush

Names and Faces: Pipeline Proper Name Extraction Face Detection Face Rectification & Representation Assign names to faces by clustering

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

Proper Name Extraction President George W. Bush makes a statement in the Rose Garden while Detected Names : Secretary of Defense Donald Rumsfeld President George W. Bush, looks on, July 23, 2003. Rumsfeld said the United States would release graphic Defense Donald Rumsfeld, photographs of the dead sons of Saddam Hussein. Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

Face Detection Face Detector (Mikolajczyk Ph.D President George W. Bush makes a Thesis 2002) statement in the Rose Garden while Secretary of Defense Donald Rumsfeld 45,000 large faces looks on, July 23, 2003. Rumsfeld said mainly frontal the United States would release graphic photographs of the dead sons of Scaled to 86x86 pixels Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

Rectify Faces SVM feature detectors - trained to recognize nose, corners of eyes and mouth Determine affine transformation between detected and canonical positions Use gradient descent for fine alignment. Project images into canonical pose

Face Representation Faces initially represented as vectors of pixels KPCA -- dimension reduction onto nonlinear basis. 1. Compute kernel function, K(i,j) for each pair of images 2. Calculate eigenvectors of kernel matrix and project Nyström Approximation Avoid computing all of K mxm Approximate C as: NxN Only compute N*m entries in K Only compute eigenvectors of mxm matrices

Face Representation (cont.) Linear Discriminants Analysis Discriminants take advantage of class information Don’t know classes Build discriminants using one name/one face items, project all face vectors into this space

Image Model Learn what each individual looks like: Face Model: P(Person i | Face Image)

Depiction Model P(Pictured | Context) multiple independent cues Yes/No President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons What can context indicate about of Saddam Hussein to prove they were killed by American depiction? troops. Photo by Larry Downing/Reuters

Depiction Model P(Pictured | Context) multiple independent cues Yes/No President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld Cues: POS tags before and after name, looks on, July 23, 2003. location in caption, distance to closest Rumsfeld said the United States would release graphic . , ( ) (L) (C) (R) left right center photographs of the dead sons of Saddam Hussein to prove they were killed by American Cue Combination: troops. Photo by Larry Downing/Reuters Maximum Entropy Model

Goal: 1.) Learn what each individual looks like -- learn image model for each person. 2.) Learn what contextual cues indicate depiction – learn depiction model across names. Achieve both of these by considering a big assignment problem.

Assignment Problem

Assignment Problem Leonardo DiCaprio Anna Kournikova Colin Powell Denzel Washington Donald Rumsfeld George Bush Jennifer Connelly Britney Spears Lucy Liu Andy Roddick Use clustering to find best assignments Each distinct name in the dataset is a cluster

Assignment Problem Leonardo DiCaprio Anna Kournikova Colin Powell Denzel Washington Donald Rumsfeld George Bush Jennifer Connelly Britney Spears Lucy Liu Andy Roddick Assign each face to one of the detected names

Assignment Problem Leonardo DiCaprio Anna Kournikova Colin Powell Denzel Washington Donald Rumsfeld George Bush Jennifer Connelly Britney Spears Lucy Liu Andy Roddick Update Appearance Model for each name and Language Model for all names

Assignment Problem Leonardo DiCaprio Anna Kournikova Colin Powell Denzel Washington Donald Rumsfeld George Bush Jennifer Connelly Britney Spears Lucy Liu Andy Roddick Repeat until convergence

Results Without – Vice President Dick Without – CEO Summit Without – James Bond Cheney With – Martha Stewart With – Pierce Brosnan With – President George W. Bush Model Accuracy of labeling Vision model, No Lang model 67% Vision model + Lang model 78%

Results - Clusters

Results - Depiction IN - pictured, Classifier % correct OUT - not pictured Baseline (all pictured) 67% Learned Lang Model 86%

Results British director Sam Mendes and World number one Lleyton Hewitt of his partner actress Kate Winslet Australia hits a return to Nicolas arrive at the London premiere of Massu of Chile at the Japan Open 'The Road to Perdition', September tennis championships in Tokyo 18, 2002. The films stars Tom October 3, 2002. REUTERS/Eriko Hanks as a Chicago hit man who Sugita has a separate family life and co- stars Paul Newman and Jude Law . REUTERS/Dan Chung

Results US President George W. Bush (L) makes German supermodel Claudia Schiffer gave remarks while Secretary of State Colin birth to a baby boy by Caesarian section Powell (R) listens before signing the US January 30, 2003, her spokeswoman said. The Leadership Against HIV /AIDS , baby is the first child for both Schiffer, 32, Tuberculosis and Malaria Act of 2003 at the and her husband, British film producer Department of State in Washington, DC. Matthew Vaughn , who was at her side for The five-year plan is designed to help the birth. Schiffer is seen on the German prevent and treat AIDS, especially in more television show 'Bet It...?!' ('Wetten Dass...?!') than a dozen African and Caribbean nations in Braunschweig, on January 26, 2002. (AFP/Luke Frazza) (Alexandra Winkler/Reuters)

Face Dictionary http://tamaraberg.com/faces/faceDict/NIPSdict/index.html