Whos in the Picture? Tamara L. Berg Cse595 Words & Pictures - - PowerPoint PPT Presentation

Who’s in the Picture?

Tamara L. Berg Cse595 Words & Pictures

Face Recognition Datasets

M2VTS Yale Face Database PIE database

What the world really looks like (scenes from an airport)

Challenging Even Up Close

Face Scanner Failed -- Wired News 5/16/02

A More Realistic Dataset

Actress Winona Ryder (news) reacts to remarks by prosecutor Ann Rundle during the sentencing hearing in her felony shoplifting case Friday, Dec. 6, 2002 at the Beverly Hills, Calif.,

• courthouse. At right is Ryder's attorney Mark
• Geragos. Ryder was sentenced to three years of

probation and was ordered to perform 480 hours

• f community service. (AP

Photo/Steve Grayson, POOL) France's Amelie Mauresmo has pulled

• ut of the Australian Open (news -

web sites) because of a knee injury. The French number one, ranked sixth in the world, has been struggling with a painful knee cap for two months and said on December 18, 2002 she would not be fit in time to resume playing in

• January. Mauresmo is shown during the

Kremlin Cup, Oct. 4. (Grigory Dukor/ Reuters)

Dataset (cont)

Doctor Nikola shows a fork that was removed from an Israeli woman who swallowed it while trying to catch a bug that flew in to her mouth, in Poriah Hospital northern Israel July 10, 2003. Doctors performed emergency surgery and removed the fork. (Reuters) President George W. Bush waves as he leaves the White House for a day trip to North Carolina, July 25, 2002. A White House spokesman said that Bush would be compelled to veto Senate legislation creating a new department of homeland security unless changes are made. (Kevin Lamarque/Reuters)

Dataset (cont.)

Producer and director Bruce Paltrow has died at the age of 58 in Rome, Italy, the U.S. Consulate said on October 3, 2002. Paltrow had suffered from throat cancer for several years, but the cause of his death was not immediately known. He is seen with his daughter actress Gwyneth Paltrow after the Academy Awards in Los Angles in March 21, 1999 file photo. (Fred Prouser/Reuters) Actor Arnold Schwarzenegger discusses his plans for his campaign with reporters after he announced that he will be a candidate for governor of California in the recall election, August 6, 2003. Schwarzenegger ended weeks of speculation about his candidacy for the October 7 referendum. (Fred Prouser/ Reuters

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/ Reuters

Names and Faces

Intuition

George Bush

Names and Faces: Pipeline

Proper Name Extraction Face Detection Face Rectification & Representation Assign names to faces by clustering

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

Proper Name Extraction

Detected Names: President George W. Bush, Defense Donald Rumsfeld, Saddam Hussein.

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

Face Detection

Face Detector (Mikolajczyk Ph.D Thesis 2002) 45,000 large faces

mainly frontal Scaled to 86x86 pixels

Rectify Faces

SVM feature detectors - trained to recognize nose,

corners of eyes and mouth

Determine affine transformation between detected

and canonical positions

Use gradient descent for fine alignment.

Project images into canonical pose

Face Representation

Faces initially represented as vectors of pixels KPCA -- dimension reduction onto nonlinear basis.

• 1. Compute kernel function, K(i,j) for each pair of images
• 2. Calculate eigenvectors of kernel matrix and project

Nyström Approximation

Avoid computing all of K Only compute N*m entries in K Only compute eigenvectors of mxm matrices

Approximate C as: NxN mxm

Face Representation (cont.)

Linear Discriminants Analysis

Discriminants take advantage of class information Don’t know classes Build discriminants using one name/one face items,

project all face vectors into this space

Image Model

Face Model: P(Person i | Face Image) Learn what each individual looks like:

Depiction Model

P(Pictured | Context)

Yes/No multiple independent cues

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons

• f Saddam Hussein to prove

they were killed by American

• troops. Photo by Larry

Downing/Reuters

What can context indicate about depiction?

Depiction Model

Cues: POS tags before and after name, location in caption, distance to closest . , ( ) (L) (C) (R) left right center Cue Combination: Maximum Entropy Model

P(Pictured | Context)

Yes/No multiple independent cues

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons

• f Saddam Hussein to prove

they were killed by American

• troops. Photo by Larry

Downing/Reuters

Goal:

1.) Learn what each individual looks like -- learn image model for each person. 2.) Learn what contextual cues indicate depiction – learn depiction model across names. Achieve both of these by considering a big assignment problem.

Assignment Problem

Assignment Problem

George Bush Colin Powell Donald Rumsfeld Anna Kournikova Andy Roddick Denzel Washington Britney Spears Jennifer Connelly

Use clustering to find best assignments Each distinct name in the dataset is a cluster

Leonardo DiCaprio Lucy Liu

Assignment Problem

George Bush Colin Powell Donald Rumsfeld Lucy Liu Anna Kournikova Andy Roddick Denzel Washington Britney Spears Jennifer Connelly

Assign each face to one of the detected names

Leonardo DiCaprio

Assignment Problem

George Bush Colin Powell Donald Rumsfeld Lucy Liu Anna Kournikova Andy Roddick Denzel Washington Britney Spears Jennifer Connelly

Update Appearance Model for each name and Language Model for all names

Leonardo DiCaprio

Assignment Problem

George Bush Colin Powell Donald Rumsfeld Anna Kournikova Andy Roddick Denzel Washington Britney Spears Jennifer Connelly

Repeat until convergence

Leonardo DiCaprio Lucy Liu

Results

Without – CEO Summit With – Martha Stewart Without – James Bond With – Pierce Brosnan Without – Vice President Dick Cheney With – President George W. Bush Model Accuracy of labeling Vision model, No Lang model 67% Vision model + Lang model 78%

Results - Clusters

Results - Depiction

Classifier % correct Baseline (all pictured) 67% Learned Lang Model 86%

IN - pictured, OUT - not pictured

Results

British director Sam Mendes and his partner actress Kate Winslet arrive at the London premiere of 'The Road to Perdition', September 18, 2002. The films stars Tom Hanks as a Chicago hit man who has a separate family life and co- stars Paul Newman and Jude Law. REUTERS/Dan Chung World number one Lleyton Hewitt of Australia hits a return to Nicolas Massu of Chile at the Japan Open tennis championships in Tokyo October 3, 2002. REUTERS/Eriko Sugita

US President George W. Bush (L) makes remarks while Secretary of State Colin Powell (R) listens before signing the US Leadership Against HIV /AIDS , Tuberculosis and Malaria Act of 2003 at the Department of State in Washington, DC. The five-year plan is designed to help prevent and treat AIDS, especially in more than a dozen African and Caribbean nations (AFP/Luke Frazza) German supermodel Claudia Schiffer gave birth to a baby boy by Caesarian section January 30, 2003, her spokeswoman said. The baby is the first child for both Schiffer, 32, and her husband, British film producer Matthew Vaughn, who was at her side for the birth. Schiffer is seen on the German television show 'Bet It...?!' ('Wetten Dass...?!') in Braunschweig, on January 26, 2002. (Alexandra Winkler/Reuters)

Results

Face Dictionary

http://tamaraberg.com/faces/faceDict/NIPSdict/index.html