Personalized Image Aesthetics Assessment Leida Li School of - - PowerPoint PPT Presentation

▶
personalized image aesthetics
SMART_READER_LITE
LIVE PREVIEW

Personalized Image Aesthetics Assessment Leida Li School of - - PowerPoint PPT Presentation

Jan 24, 2023 325 likes •726 views

Personalized Image Aesthetics Assessment Leida Li School of Artificial Intelligence Xidian University 2020.05.27 Collaborators: Hancheng Zhu Jinjian Wu Sicheng Zhao Guiguang Ding Guangming Shi Weisi Lin CUMT XDU UC Berkeley Tsinghua

slide-1
SLIDE 1

Personalized Image Aesthetics Assessment

School of Artificial Intelligence Xidian University 2020.05.27

Leida Li

Collaborators:

Hancheng Zhu CUMT Jinjian Wu XDU Sicheng Zhao UC Berkeley Guiguang Ding Tsinghua Guangming Shi XDU Weisi Lin NTU

slide-2
SLIDE 2

Con

  • nte

tent nts

2 2020/5/28

  • Introduction
  • Related Work
  • Personality-assisted Multi-task Learning for Personalized

Image Aesthetics Assessment

  • Personalized Image Aesthetics Assessment via Meta-

learning

  • Conclusion
slide-3
SLIDE 3

Con

  • nte

tent nts

3 2020/5/28

  • Introduction
  • Related Work
  • Personality-assisted Multi-task Learning for Personalized

Image Aesthetics Assessment

  • Personalized Image Aesthetics Assessment via Meta-

learning

  • Conclusion
slide-4
SLIDE 4

2020/5/28 4

Introducti roduction

  • n to Aes

esthetic hetic Qua uality ity Chin ina has a prover verb

The love e of be beauty y is common

  • n to all peopl

ple

A n natural ural questi stion:

  • n: how to

to jud udge aes esthetics? etics?

slide-5
SLIDE 5

2020/5/28 5

Introducti roduction

  • n to Aes

esthetic hetic Qua uality ity

  • Photog
  • togra

raphy phy rules les

Rule of Thirds Symmetry Depth of Field Color Harmony

  • K. Michal, et al., Leveraging expert feature knowledge for predicting image aesthetics. IEEE Trans. Image Process., 2018.
slide-6
SLIDE 6

2020/5/28 6

https://baijiahao.baidu.com/s?id=1608952443864950129&wfr=spider&for=pc

  • Photog
  • togra

raphy phy rules les

Introducti roduction

  • n to Aes

esthetic hetic Qua uality ity

slide-7
SLIDE 7

2020/5/28 7

Introducti roduction

  • n to Aes

esthetic hetic Qua uality ity

  • Ap

Appl plicatio cation n scen enari rios

  • s
  • Advertisement

Alibaba’s Luban system

  • Launched on 11/11/2016
  • Designed 170 million posters
  • Improved hit rate by 100%
  • Equipped with IAA engine

http://www.mgzxzs.com/tmtbzxjc/2288.html

slide-8
SLIDE 8

2020/5/28 8

  • Ap

Appl plicatio cation n scen enari rios

  • s
  • Cover image selection

Introducti roduction

  • n to Aes

esthetic hetic Qua uality ity

slide-9
SLIDE 9

2020/5/28 9

  • Ap

Appl plicatio cation n scen enari rios

  • s
  • Photo auto-cropping

Introducti roduction

  • n to Aes

esthetic hetic Qua uality ity

slide-10
SLIDE 10

Con

  • nte

tent nts

10 2020/5/28

  • Introduction
  • Related Work
  • Personality-assisted Multi-task Learning for Personalized

Image Aesthetics Assessment

  • Personalized Image Aesthetics Assessment via Meta-

learning

  • Conclusion
slide-11
SLIDE 11
  • Aesthetics regression

2020/5/28 11

Generi ric c Aesthetic tic Quality ity

  • H. Zeng, et al., A unified probabilistic formulation of image aesthetic assessment. IEEE Trans. Image Process., 2020
  • Y. Deng, et al., Image aesthetic assessment an experimental survey. IEEE Signal Process. Mag., 2017.
  • Generic image aesthetics assessment (GIAA)
  • Aesthetics classification

Score = 4.0 Score = 3.8 Score = 2.4

  • Aesthetics distribution
slide-12
SLIDE 12

2020/5/28 12

Related Work

  • R. Datta, et al., Studying aesthetics in photographic images using a computational approach. ECCV 2006.
  • X. Tang, et al., Content-based photo quality assessment. IEEE Trans. Multimedia, 2013.
  • N. Murray, et al., AVA: A large-scale database for aesthetic visual analysis. CVPR 2012.
  • Conventional approaches with handcrafted features
  • Simple image features
  • Colorfulness
  • Contrast
  • Brightness
  • Image composition features
  • Low depth of field
  • Salient object
  • Rule of thirds
  • General-purpose features
  • SIFT descriptors
  • Bag of visual words (BOV)
  • Fisher vector (FV)

High colorfulness Low colorfulness Good composition Bad composition

Extracted handcrafted features for image aesthetics assessment

slide-13
SLIDE 13

2020/5/28 13

Rel elated ed Work

  • S. Kong, et al., Photo aesthetics ranking network with attributes and content adaptation. ECCV 2016.
  • Y. Kao, et al., Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Process., 2017.
  • Deep-learning approaches
  • Ranking deep network
  • Multi-task deep network

Aesthetics attribute Network architecture Network architecture

Significant progress has been achieved in GIAA.

slide-14
SLIDE 14

2020/5/28 14

Per ersona nalized ized Aes esthetic hetic Qua uality ity

One man's 's meat is is anoth ther er man's 's poison

  • n

China na has as another

  • ther proverb
  • verb
slide-15
SLIDE 15

2020/5/28 15

Per ersona nalized ized Aes esthetic hetic Qua uality ity

  • Personalized image aesthetics assessment (PIAA)
  • People have different tastes on image aesthetics, depending on their

subjective preferences.

Score = 4.0 Score = 3.8 Score = 2.4

Ratings = (3,3,4,5,5) Ratings = (2,3,4,5,5) Ratings = (1,2,2,3,4)

slide-16
SLIDE 16

2020/5/28 16

Rel elated ed Work

  • J. Ren, et al., Personalized image aesthetics. ICCV 2017.
  • Adapting from generic aesthetics

Personalized image aesthetics model with a residual-based model adaptation scheme.

slide-17
SLIDE 17

2020/5/28 17

Rel elated ed Work

  • P. Lv, et al., USAR: an interactive user-specific aesthetic ranking framework for images. ACM MM 2018.
  • User interaction

User-friendly aesthetic ranking framework via deep neural network and a small amount of interaction.

slide-18
SLIDE 18

2020/5/28 18

Rel elated ed Work

Challenges：

1. Existing works leverage objective visual features (e.g., contents and attributes) for modeling users’ subjective aesthetic preferences. This may be insufficient, because the subjective factors (e.g., personality traits) in rating image aesthetics are not fully investigated. 2. The generic model learned from average aesthetics cannot accurately capture the shared aesthetic prior knowledge when people gauge image aesthetics, since it simply uses the average score as the training target, which counteracts the differences of individual aesthetic perception.

slide-19
SLIDE 19

Con

  • nte

tent nts

19 2020/5/28

  • Introduction
  • Related Work
  • Personality-assisted Multi-task Learning for Personalized

Image Aesthetics Assessment

  • Personalized Image Aesthetics Assessment via Meta-

learning

  • Conclusion
slide-20
SLIDE 20

PA PA_IAA IAA

2020/5/28 20

Images liked by users with high extraversion Images liked by users with low extraversion

  • H. Zhu, L. Li, et al., Evaluating attributed personality traits from scene perception probability, Pattern Recogn Lett, 2018.
  • S. C. Guntuku, et al., Who likes what, and why? insights into personality modeling based on image “likes“, IEEE Trans Affect Comput, 2018.

Personality-assisted Multi-task Learning for Generic and Personalized Image Aesthetics Assessment (L. Li, H. Zhu, et al., IEEE TIP, 2020)

  • As an important subjective trait, personality trait is believed as a key factor in

modeling humans’ subjective preferences.

  • What is the relationship between aesthetics assessment and personality prediction

from images?

slide-21
SLIDE 21

2020/5/28 21

PA PA_IAA IAA

  • Big-Five personality traits
  • Openness: tendency to be open, curious, etc.
  • Conscientiousness: tendency to be responsible and reliable.
  • Extraversion: tendency to interact and spend time with others.
  • Agreeableness: tendency to be kind, generous, etc.
  • Neuroticism: tendency to be anxious, sensitive, etc.
  • B. Rammstedt, et al., Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German, J. Res. Pers. 2007.
  • S. Ruder, CoRR, 2017. Online: http://arxiv.org/abs/1706.05098
  • Multi-task learning

An effective way in capturing useful information contained in multiple related tasks, which can be used to improve the generalization performance of all tasks. Convolution

. . . . . .

Fully Connected

Aesthetics scores Personality traits

Task 1 Task 2

Related tasks

slide-22
SLIDE 22

2020/5/28 22

PA PA_IAA IAA

  • A multi-task learning network with shared weights is proposed to predict the aesthetics distribution
  • f an image and Big-Five (BF) personality traits of people who like the image
  • To capture the common representation of image aesthetics and people’s personality traits, a

Siamese network is trained using aesthetics data and personality data jointly.

  • Inter-task fusion is introduced to generate individual’s personalized aesthetic scores.
slide-23
SLIDE 23

2020/5/28 23

PA PA_IAA IAA

⚫

Generic aesthetics training samples: {𝐽𝑏

𝑗 ; 𝒕𝑏 𝑗 }𝑗=1 𝑂𝑏 , where 𝒕𝑏 𝑗 = {𝑡𝑏𝑜 𝑗 }𝑜=1 𝑂

; Aesthetic deep features: 𝒆𝑏

  • Estimated aesthetic distribution:

Ƹ 𝑡𝑏𝑜

𝑗

=

𝑓𝑿𝒃𝒐

𝑈 𝒆𝑏

σ𝑘=1

𝑂

𝑓

𝑿𝒃𝒌 𝑈 𝒆𝑏

  • Generic aesthetics loss function:

𝑀𝑏 =

1 𝑂𝑏 1 𝑂 σ𝑗=1

𝑂𝑏 σ𝑜=1 𝑂

Ƹ 𝑡𝑏𝑜

𝑗

−𝑡𝑏𝑜

𝑗 2 2

⚫ Personality training samples: {{𝐽𝑞

𝑣𝑛}𝑛=1 𝑁

, {𝑇𝑞

𝑣𝑗}𝑗=1 5

}𝑣=1

𝑉

, 𝑉 is the number of users, 𝑁 is the number of images liked by a user. Personality deep features: 𝒆𝑞

  • Predicted personality distribution:

෡ 𝑻𝑞

𝑣𝑛 = 𝑓𝑿𝑞

𝑈𝒆𝑞−𝑓−𝑿𝑞 𝑈𝒆𝑞

𝑓𝑿𝑞

𝑈𝒆𝑞+𝑓−𝑿𝑞 𝑈𝒆𝑞

  • Personality loss function:

𝑀𝑞 = 1

5 1 𝑉 1 𝑁 σ𝑣=1

𝑉

σ𝑛=1

𝑁

σ𝑗=1

5

መ 𝑇𝑞

𝑣𝑛𝑗−𝑇𝑞 𝑣𝑗 2 2

⚫ Personalized aesthetics training samples: {𝐽𝑐

𝑗; 𝑡𝑐 𝑗 }𝑗=1 𝑂𝑐 ;

  • Estimated personalized aesthetic score:

Ƹ 𝑡𝑐

𝑗 = Ƹ

𝑡𝑏

𝑗 + 𝑿𝑐ො

𝒕𝑞

𝑗

  • Personalized aesthetic loss function:

𝑀𝑐 =

1 𝑂𝑐 σ𝑗=1

𝑂𝑐

Ƹ 𝑡𝑐

𝑗 −𝑡𝑐 𝑗 2 2

slide-24
SLIDE 24

2020/5/28 24

Ex Experiments eriments

Databases:

  • Aesthetics databases:

GIAA: AVA(250,000 images, 230,000 for training, 20,000 for testing) PIAA: FLICKR-AES (40,000 images; 210 users, 173 for training, 37 for testing)

  • Personality database:

PsychoFlickr(60,000 liked images of 300 users (200 images per user)) Hyper-parameters: weight decay of 1e-5, momentum of 0.9, batch size of 50, initial learning rate of 1e-4, drops to a factor of 0.9 every epoch, and total epoch of 50.

  • N. Murray, L. Marchesotti, and P. F., “AVA: a large-scale database for aesthetic visual analysis,” ICCV 2012.
  • S. Kong, X. Shen, Z. Lin, R. Mech, “Photo aesthetics ranking network with attributes and content adaptation,” ECCV 2016.
  • M. Cristani et al., “Unveiling the multimedia unconscious: implicit cognitive processes and multimedia content analysis,” ACM MM 2013.

Criteria: Classification: Overall Accuracy (ACC) ↑ Regression: Spearman Rank Order Correlation Coefficients (SROCC) ↑ Distribution: Earth Mover’s Distance (EMD) ↓

slide-25
SLIDE 25

2020/5/28 25

Aesthetics distribution prediction

Ex Experiments eriments

Performance comparison on AVA database (Aesthetics classification)

slide-26
SLIDE 26

2020/5/28 26

PIAA Database: FLICKR-AES

  • 40,000 images, each image is labeled with aesthetic score by five different users.
  • Training: 35,263 images rated by 173 users
  • Testing: 4737 images rated by the other 37 users

FLICKR-AES : J. Ren, X. Shen, Z. Lin, R. Mech, and D. J. Foran, Personalized image aesthetics, ICCV 2017.

Ex Experiments eriments

Performance comparison on FLICKR-AES database PA_IAA outperforms MT_IAA by a large margin, which indicates that the personality prediction task of the proposed model has made a significant contribution to PIAA.

slide-27
SLIDE 27

2020/5/28 27

Ex Experiments eriments

  • Personality prediction performance on FLICKR-AES database
  • Performance comparison of test individuals

Multi-task learning module also contributes helpful information for personality prediction.

slide-28
SLIDE 28

Con

  • nte

tent nts

28 2020/5/28

  • Introduction
  • Related Work
  • Personality-assisted Multi-task Learning for Personalized

Image Aesthetics Assessment

  • Personalized Image Aesthetics Assessment via Meta-

learning

  • Conclusion
slide-29
SLIDE 29

BLG BLG-PI PIAA AA

2020/5/28 29

Personalized Image Aesthetics Assessment via Meta-learning with Bi-level Gradient Optimization (H. Zhu, L. Li, et al., IEEE TCYB, 2020)

  • Existing PIAA models: fine-tuning generic image aesthetics assessment (GIAA) model

as prior knowledge, which is not sufficient.

  • Meta-learning is adopted to learn the shared prior knowledge among different

people.

  • Fast adaptation to unknown user for PIAA using very few sample images.

PIAA using “average aesthetics” based GIAA model as prior knowledge

slide-30
SLIDE 30

2020/5/28 30

BLG BLG-PI PIAA AA

  • Commonality and Individuality

High Aesthetics Low Aesthetics

High agreement in binary classification Share aesthetic knowledge Commonality-based Individuality

slide-31
SLIDE 31

2020/5/28 31

Meta-Learning (learn to learn)

  • Knowledge-driven machine learning framework, which is able to extract meta

knowledge from many related tasks.

  • Deep meta learner can fast adapt to a new task with limited training data.

Vanschoren et al. Meta-Learning: A Survey. arXiv, 2018.

BLG BLG-PI PIAA AA

slide-32
SLIDE 32

2020/5/28 32

BLG BLG-PI PIAA AA

Key idea:

  • Meta-learning is adopted to learn the shared prior knowledge among different people

in judge aesthetics.

  • Fast adaptation to unknown user for PIAA using very few sample images.
slide-33
SLIDE 33

2020/5/28 33

BLG BLG-PI PIAA AA

  • Meta-training set: {𝒠𝑢𝑠𝑡

𝜐𝑗 , 𝒠𝑢𝑠𝑟 𝜐𝑗 }𝑗=1 𝑂 ;

  • Deep neural network: 𝑔

𝜄;

  • Loss function: 𝑀τ =

1 𝑁 σ𝑗=1 𝑁

ො 𝑧𝑗−𝑧𝑗 2

2;

  • Gradient of network parameters: 𝑕𝜄 = 𝛼𝜄𝑀𝜐𝑔

𝜄;

  • First moment of 𝑕𝜄(𝑡): 𝑛𝜄(𝑡) = 𝜈1𝑛𝜄(𝑡−1) + (1 − 𝜈1)𝑕𝜄(𝑡);
  • Second moment of 𝑕𝜄(𝑡): 𝑤𝜄(𝑡) = 𝜈1𝑤𝜄(𝑡−1) + 1 − 𝜈1 𝑕2

𝜄 𝑡 ;

  • 𝐵𝑒𝑏𝑛 𝑀τ, 𝜄 :

𝜄 ← 𝜄 − 𝛽 σ𝑡=1

𝑇 𝑛𝜄 𝑡 𝑤𝜄 𝑡 +𝜗 ;

slide-34
SLIDE 34

2020/5/28 34

Performances on FLICKR-AES database.

Ex Experiments eriments

BLG-PIAA further achieves 3.7% and 8.6% performance improvement when 10 and 100 images are used for training, respectively

*BA-PIAA: baseline PIAA method based on “average aesthetics”

slide-35
SLIDE 35

Visu sual al re resu sults lts

2020/5/28 35

BLG-PIAA predicts user’s aesthetic score more accurately than BA-PIAA.

Visual Analysis

slide-36
SLIDE 36

Visu sual al re resu sults lts

2020/5/28 36

  • The gradients map of user’s PIAA model are more concentrated in salient regions than that of prior model.
  • Users with different aesthetic ratings on an image have different areas of interest.

https://github.com/sar-gupta/convisualize_nb

Visual Analysis

slide-37
SLIDE 37

Con

  • nte

tent nts

37 2020/5/28

  • Introduction
  • Related Work
  • Personality-assisted Multi-task Learning for Personalized

Image Aesthetics Assessment

  • Personalized Image Aesthetics Assessment via Meta-

learning

  • Conclusion
slide-38
SLIDE 38

Con

  • nclu

clusion sion

2020/5/28 38

  • Personality, a key factor in subjective traits, has been utilized for personalized

image aesthetics assessment under a multi-task learning framework. Personality data and aesthetics data were jointly used to learn the common features for predicting both aesthetics distribution and personality. Inter-task fusion was introduced to learn the influence of personality traits in individuals’ aesthetic preferences on images.

  • Meta-learning has been utilized to learn the shared prior knowledge in aesthetics
  • assessment. By treating each individual’s aesthetic assessment as a separate task,

prior knowledge was learned, based on which PIAA was achieved by fast adaptation using only small samples.

  • User portrait facilitates deeper understanding of user’s aesthetic preference,

personal interest, which is expected to benefits PIAA. This can be done using social data.

slide-39
SLIDE 39

2020/5/28 39

Th Than ank k yo you! u!