Affect- and Personality-based Recommender Systems Part I: - - PowerPoint PPT Presentation

Affect- and Personality-based Recommender Systems

Part I: Motivation, Models

Marko Tkalčič, Free University of Bozen-Bolzano

ACM Summer School on Recommender Systems 2017 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 1/64

Netflix

Netflix . . . one of the greatest players in recommender systems!

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 2/64

Netflix

Netflix . . . one of the greatest players in recommender systems! What is Netflix recommending us?

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 2/64

Netflix

Netflix . . . one of the greatest players in recommender systems! What is Netflix recommending us? Movies/films . . . really?

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 2/64

Netflix

Netflix . . . one of the greatest players in recommender systems! What is Netflix recommending us? Movies/films . . . really? “I want to watch a funny movie tonight”

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 2/64

Netflix

Netflix . . . one of the greatest players in recommender systems! What is Netflix recommending us? Movies/films . . . really? “I want to watch a funny movie tonight” Funny is all you want?

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 2/64

But there’s more!!

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 3/64

But there’s more!!

Question: Can (rating/genre/year/director) summarize that rollercoaster?

Thanks to Shlomo Berkovski for the inspiring example from the EMPIRE 2015 keynote. Image source: http://yhvh.name/?w=2646

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 3/64

Table of Contents

Introduction Why are theory-driven models important? Models of Emotion and Personality

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 4/64

Who am I?

Marko Tkalčič

• 2016 - now : assistant professor at Free University of

Bozen-Bolzano

• 2013 - 2015: postdoc at Johannes Kepler University,

Linz

• 2011 - 2012: postdoc at University of Ljubljana
• 2008 - 2010: PhD student at University of Ljubljana

My research explores ways in which psychologically-motivated user characteristics, such as emotions and personality, can be used to improve recommender systems (personalized systems in general). It employs methods such as user studies and machine learning.

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 5/64

Book, 2016

• Tkalčič, M., Carolis, B. De, Gemmis, M. de, Odić, A., &

Košir, A. (Eds.). (2016). Emotions and Personality in Personalized Services. Springer International Publishing. https://doi.org/10.1007/978-3-319-31413-6

• Authors from
• Stanford, Cambridge, Imperial College, UCL . . .
• topics:
• psychological models
• acquisition of emotions/personality
• personalization techniques
• http://www.springer.com/gp/book/9783319314112

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 6/64

Goal of this Talk/Learning Outcomes

• complements other talks of the Summer School
• off the beaten track . . . meant to open new ideas
• part of the audience should say
• this is BS
• this is inspiring

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 7/64

Goal of this Talk/Learning Outcomes

• complements other talks of the Summer School
• off the beaten track . . . meant to open new ideas
• part of the audience should say
• this is BS
• this is inspiring

We will learn

• Part I (Tuesday, 16:30 - 18:30)
• why models borrowed from psychology and social sciences are important
• models (emotions, personality)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 7/64

Goal of this Talk/Learning Outcomes

• complements other talks of the Summer School
• off the beaten track . . . meant to open new ideas
• part of the audience should say
• this is BS
• this is inspiring

We will learn

• Part I (Tuesday, 16:30 - 18:30)
• why models borrowed from psychology and social sciences are important
• models (emotions, personality)
• Part II (Thursday, 8:15 - 10:15)
• tools for acquiring E&P
• usage of E&P in recommender systems

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 7/64

Table of Contents

Introduction Why are theory-driven models important? Models of Emotion and Personality

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 8/64

What are we predicting in RS?

. . . for each user, we want to choose such item that maximizes the user’s utility/rating. (Adomavicius, Tuzhilin, 2005)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 9/64

What are we predicting in RS?

. . . for each user, we want to choose such item that maximizes the user’s utility/rating. (Adomavicius, Tuzhilin, 2005) Recommender Systems (RSs) are software tools and techniques that provide suggestions for items that are most likely of interest to a particular user (Ricci et al., 2015)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 9/64

What are we predicting in RS?

. . . for each user, we want to choose such item that maximizes the user’s utility/rating. (Adomavicius, Tuzhilin, 2005) Recommender Systems (RSs) are software tools and techniques that provide suggestions for items that are most likely of interest to a particular user (Ricci et al., 2015)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 9/64

What are we predicting in RS?

. . . for each user, we want to choose such item that maximizes the user’s utility/rating. (Adomavicius, Tuzhilin, 2005) Recommender Systems (RSs) are software tools and techniques that provide suggestions for items that are most likely of interest to a particular user (Ricci et al., 2015)

• what influences (which features)?
• how (which algorithm)?

References

Adomavicius, G., and Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6), 734–749. Ricci, F., Rokach, L., and Shapira, B. (2015). Recommender Systems: Introduction and Challenges. In Recommender Systems Handbook (Vol. 54, pp. 1–34). Boston, MA: Springer US. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 9/64

Decision making

Liking, purchasing, rating, clicking etc. . . actions triggered by decisions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 10/64

Decision making

Liking, purchasing, rating, clicking etc. . . actions triggered by decisions One way of looking at recommender systems is the one taken by (Jameson et al., 2015): we view recommender systems as tools for helping people to make better choices —not large, complex choices, such as where to build a new airport, but the small- to medium-sized choices that people make every day: what products to buy, what documents to read, which people to contact. References

Jameson, A., Willemsen, M. C., Felfernig, A., de Gemmis, M., Lops, P., Semeraro, G., and Chen, L. (2015). Human Decision Making and Recommender Systems. In Recommender Systems Handbook (Vol. 54, pp. 611–648). Boston, MA: Springer US. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 10/64

Decision making models

There are many decision-making models

• ASPECT/ARCADE (Jameson et al.)
• Somatic Markers (Damasio)
• Two Systems (Kahneman and Tversky)
• Multi-system model (Lerner et al.)

References

Jameson, A., Willemsen, M. C., Felfernig, A., de Gemmis, M., Lops, P., Semeraro, G., and Chen, L. (2015). Human Decision Making and Recommender Systems. In Recommender Systems Handbook (Vol. 54, pp. 611–648). Boston, MA: Springer US. Damasio, A. (1994) Descartes’ Error: Emotion, Reason, and the Human Brain Kahneman, D. (2003). A perspective on judgment and choice: mapping bounded rationality. The American Psychologist, 58(9), 697–720. Lerner, J. S., Li, Y., Valdesolo, P., and Kassam, K. S. (2015). Emotion and Decision Making. Annual Review of Psychology, 66(1), 799–823. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 11/64

ASPECT/ARCADE models

• ASPECT model of choice-making: based on human-choice patterns
• ARCADE model: strategies to support decision making - i.e. by RS

References

Jameson, A., Willemsen, M. C., Felfernig, A., de Gemmis, M., Lops, P., Semeraro, G., and Chen, L. (2015). Human Decision Making and Recommender Systems. In Recommender Systems Handbook (Vol. 54, pp. 611–648). Boston, MA: Springer US. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 12/64

Decision making and emotions - Damasio

• physiological/evolutionary aspect
• emotional processes guide (or bias) behavior, particularly decision-making
• changes in both body and brain states in response to different stimuli
• these physiological signals (or somatic markers) and their evoked emotion are

consciously or unconsciously associated with their past outcomes and bias decision-making References

Damasio, A. (1994) Descartes’ Error: Emotion, Reason, and the Human Brain Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 13/64

Kahneman Tversky two systems

• Decision:
• System 1: fast, intuitive, emotion-driven
• system 2: slow, rational

References

Kahneman, D. (2003). A perspective on judgment and choice: mapping bounded rationality. The American Psychologist, 58(9), 697–720. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 14/64

Kahneman Tversky two systems

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 15/64

Decision making model - Lerner

emotions and personality influence our decisions! References

Lerner, J. S., Li, Y., Valdesolo, P., and Kassam, K. S. (2015). Emotion and Decision Making. Annual Review of Psychology, 66(1), 799–823. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 16/64

Personality and preferences

Personality traits (extraverted/introverted, open/conservative etc.) are linked to music genre preferences (Rentfrow et al, 2003) References

Rentfrow, P. J., and Gosling, S. D. (2003). The do re mi’s of everyday life: The structure and personality correlates of music

• preferences. Journal of Personality and Social Psychology, 84(6), 1236–1256.

Tkalčič, M., Ferwerda, B., Hauger, D., and Schedl, M. (2015). Personality Correlates for Digital Concert Program Notes. In UMAP 2015, Lecture Notes On Computer Science 9146 (Vol. 9146, pp. 364–369). Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 17/64

Emotions are related to other things as well

Why we choose to consume some kind of content?

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 18/64

Emotions are related to other things as well

Why we choose to consume some kind of content? One of the main reasons why people consume music (Lonsdale, 2011) and films (Oliver, 2008) is emotion regulation. References

Lonsdale, A. J., and North, A. C. (2011). Why do we listen to music? A uses and gratifications analysis. British Journal of Psychology (London, England : 1953), 102(1), 108–34. https://doi.org/10.1348/000712610X506831 Oliver, M. B. (2008). Tender affective states as predictors of entertainment preference. Journal of Communication, 58(1), 40–61. https://doi.org/10.1111/j.1460-2466.2007.00373.x Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 18/64

However, it can get complicated

Why do we like drama, sad films?

• . . . under some circumstances, individuals may choose to view entertainment for

reasons that may not be best described as driven by hedonic motivations but rather as driven by eudaimonic motivations: greater insight, self reflection, or contemplations of poignancy or meaningfulness (e.g., what makes life valuable).

• The Hangover
• hedonic quality (comedy)
• no eudaimonic quality
• La vita e’ bella
• hedonic quality (comedy)
• eudaimonic quality

References

Oliver, M. B. (2008). Tender affective states as predictors of entertainment preference. Journal of Communication, 58(1), 40–61. https://doi.org/10.1111/j.1460-2466.2007.00373.x Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 19/64

What we have today?

Features

• genre
• actors (performers, authors, directors

. . . )

• rating
• timestamp
• location
• price
• year

Algorithms

• Content-based
• Collaborative Filtering
• Knowledge-based
• Hybrid

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 20/64

Are these the right features?

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 21/64

Are these the right features?

Light Darkness genre emotions action personality timestamp actors rating location

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 21/64

Are these the right features?

Thesis: besides using existing data/features and the respective data-driven models, theory-driven features and models should be investigated to improve recommender systems. Let’s look at a couple of sparse examples illustrating the above thesis.

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 22/64

Chomsky-Skinner Debate

Behaviorism During the first half of the twentieth century, John B. Watson devised methodological behaviorism, which rejected introspective methods and sought to understand behavior by only measuring observable behaviors and events (Wikipedia).

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 23/64

Chomsky-Skinner Debate

Behaviorism During the first half of the twentieth century, John B. Watson devised methodological behaviorism, which rejected introspective methods and sought to understand behavior by only measuring observable behaviors and events (Wikipedia). On language acquisition

• Skinner (behaviorist):
• operant conditioning : child goes through trial-and-error she tries and fails to use correct

language until she succeeds; with reinforcement and shaping provided by the parents gestures (smiles, attention and approval) which are pleasant to the child.

• there’s no need to understand the underlying hardware
• Chomsky (structuralist):
• operant conditioning could not account for a child’s ability to create or understand an

infinite variety of novel sentences

• language acquisition has an innate structure, it is a function of the human brain.
• there are structures of the brain that control the interpretation and production of speech
• it is important to understand the underlying hardware

References

Andresen, J. (1991). Skinner and Chomsky 30 years later. Or: The return of the repressed . The Behavior Analyst, 14(1), 49–60. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 23/64

Claudia Perlich, keynote @Recsys 2016

• Q: In the world of deep learning, XGBoost etc. why logistic regression?
• A: I studied neural networks in 1995 . . . downgraded to decision trees in 2004

and won 3 KDD cups with linear models. Personally I find it easier to build up a model with interesting feature construction. . . I can look under the hood and see what these things are doing. https://www.youtube.com/watch?v=1WmqqfXNFZ4&feature=youtu.be&t=3003

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 24/64

Gary Marcus (NYU): let’s put “top down” and “bottom up” knowledge on equal footing

To get computers to think like humans, we need a new A.I. paradigm, one that places top down and bottom up knowledge on equal footing. Bottom-up knowledge is the kind of raw information we get directly from our senses, like patterns of light falling on

• ur retina. Top-down knowledge comprises cognitive models of the world and how it

works. References

Gary Marcus, Artificial Intelligence Is Stuck. Here’s How to Move It Forward. New York Times, July 29, 2017 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 25/64

Come again?

Why, again, models-driven approaches complementing data-driven?

• models from other disciplines can be inspiring
• such models can account for information variance
• Marvin Minsky created the first neural net borrowing the concept from neurology

https://en.wikipedia.org/wiki/History_of_artificial_intelligence

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 26/64

But, how can we relate this to recommender systems?

• the gap between existing RS and psychological models is too wide
• psychological models of DM can be
• too generic
• too complex
• hard to implement

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 27/64

But, how can we relate this to recommender systems?

• the gap between existing RS and psychological models is too wide
• psychological models of DM can be
• too generic
• too complex
• hard to implement
• let’s make smaller steps
• measure new features and use our (recsys) algorithms
• emotions
• personality
• let’s shed new light

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 27/64

Table of Contents

Introduction Why are theory-driven models important? Models of Emotion and Personality

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 28/64

Personality, Mood and Emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 29/64

Personality, Mood and Emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 30/64

Personality, Mood and Emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 31/64

Personality, Mood and Emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 32/64

Personality, Mood and Emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 33/64

Personality, Mood and Emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 34/64

Personality

• What is personality?
• accounts for individual differences ( = explains the variance in users) in our enduring

emotional, interpersonal, experiential, attitudinal, and motivational styles

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 35/64

Personality

• What is personality?
• accounts for individual differences ( = explains the variance in users) in our enduring

emotional, interpersonal, experiential, attitudinal, and motivational styles

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 35/64

Models of Personality - 4 temperaments

• Four temperaments Hippocratess (cca 400 BC)
• sanguine (pleasure-seeking and sociable)
• choleric (ambitious and leader-like)
• melancholic (analytical and quiet)
• phlegmatic (relaxed and peaceful)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 36/64

Models of Personality - Big5

• The five factor model (FFM) – Big5:
• Extraversion
• Agreeableness
• Conscientousness
• Neuroticism
• Openness (to new experiences)

The inverse of Neuroticism is sometimes referred to as Emotional Stability, References

McCraMcCrae, R. R., and John, O. P. (1992). An Introduction to the Five-Factor Model and its Applications. Journal of Personality, 60(2), p175 – 215. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 37/64

Individual Factors

• Openness to Experience (O),
• high: imaginative, creative people (individualistic, non conforming and are very aware of

their feelings)

• low: down-to-earth, conventional people. (simple and straightforward thinking over

complex, ambiguous and subtle)

• sub-factors are imagination, artistic interest, emotionality, adventurousness, intellect and

liberalism.

• Conscientiousness
• high : prudent, organized
• low: impulsive, disorganized.
• sub-factors are self-efficacy, orderliness, dutifulness, achievement-striving, self-discipline

and cautiousness.

• Extraversion
• high: degree of engagement with the external world, react with enthusiasm and often

have positive emotions

• low: lack of engagement with the external world, quiet, low-key and disengaged in social

interactions.

• sub-factors of E are friendliness, gregariousness, assertiveness, activity level,

excitement-seeking and cheerfulness.

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 38/64

Individual Factors

• Agreeableness
• high : need for cooperation and social harmony
• low : opposite
• subfactors: trust, morality, altruism, cooperation, modesty and sympathy.
• Neuroticism
• high: emotionally reactive. They tend to respond emotionally to relatively neutral stimuli.

They are often in a bad mood, which strongly affects their thinking and decision making

• low: calm, emotionally stable and free from persistent bad mood.
• sub-factors are anxiety, anger, depression, self-consciousness, immoderation and

vulnerability.

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 39/64

Models of personality - others

• RIASEC occupational model:
• Realistic (Doers),
• Investigative (Thinkers),
• Artistic (Creators),
• Social (Helpers),
• Enterprising (Persuaders)
• Conventional (Organizers)
• Bartle gamers model:
• Killers: interfere with the functioning
• f the game world or the play

experience of other players

• Achievers: accumulate status tokens by

beating the rules-based challenges of the game world

• Explorers: discover the systems

governing the operation of the game world

• Socializers: form relationships with
• ther players by telling stories within

the game world

• Myers-Briggs Personality Type
• Extraversion (E) or Introversion (I).
• Sensing (S) or Intuition (N).
• Thinking (T) or Feeling (F).
• Judging (J) or Perceiving (P).
• Learning styles (Felder and Silverman

Learning Style Model):

• active/reflective,
• sensing/intuitive,
• visual/verbal,
• sequential/global
• Thomas-Kilmann Conflict Mode
• Assertiveness
• Cooperativeness

There are correlations between these models.

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 40/64

Thomas-Kilmann Conflict Mode

• was developed to measure the conflict resolution styles in groups

COMPETING COLLABORATING COMPROMISING AVOIDING ACCOMODATING COOPERATIVENESS ASSERTIVENESS HIGH HIGH LOW LOW

References

Thomas, K. L., and Kilman, R. H. (n.d.). Thomas-Kilman Conflict Mode Instrument. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 41/64

Domain-specific personality models - tourism

• questionnaire -> factor analysis
• tourist specific personality constructs
• Sun and Chill-out
• Knowledge and Travel
• Independence and History
• Culture and Indulgence
• Social and Sport
• Action and Fun
• Nature and Recreation
• picture-based measuring instruments
• pick and rank 10 images out of a pool

References

Neidhardt, J., Seyfang, L., Schuster, R., and Werthner, H. (2015). A picture-based approach to recommender systems. Information Technology and Tourism, 15(1), 49–69. https://doi.org/10.1007/s40558-014-0017-5 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 42/64

Measuring the FFM

• Extensive questionnaires (from 5 to several 100s questions)
• BFI: 44 questions
• TIPI : 10 questions
• NEO-IPIP: 300 questions
• For each user u a five tuple b = (b1, b2, b3, b4, b5)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 43/64

TIPI - Ten-Items Personality Inventory

TIPI: I see myself as (1-7 . . . agree/disagree):

• 1. Extraverted, enthusiastic.
• 2. Critical, quarrelsome.
• 3. Dependable, self-disciplined.
• 4. Anxious, easily upset.
• 5. Open to new experiences,

complex.

• 6. Reserved, quiet.
• 7. Sympathetic, warm.
• 8. Disorganized, careless.
• 9. Calm, emotionally stable.
• 10. Conventional, uncreative.
• b1 = q1 + (8 − q6) = Extraversion
• b2 = q2 + (8 − q7) = Agreeableness
• b3 = q3 + (8 − q8) = Conscientiousness
• b4 = q4 + (8 − q9) = Emotional Stability
• b5 = q5 + (8 − q10) = Openness to Experiences

References

Gosling, S. D., Rentfrow, P. J., and Swann, W. B. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37(6), 504–528. doi:10.1016/S0092-6566(03)00046-1 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 44/64

Let’s measure ourselves

https://goo.gl/forms/qN64cjEaMMlor2F42

Age ¡group Extraversion Agreeableness Conscientiousness Emotional ¡Stability Openness 15 ¡to ¡20 Mean 3.79 4.47 4.41 4.61 5.43 SD 1.55 1.22 1.39 1.47 1.17 n ¡= ¡ 54973 54973 54973 54973 54973 21 ¡to ¡30 Mean 3.73 4.5 4.57 4.64 5.49 SD 1.54 1.2 1.39 1.46 1.13 n ¡= ¡ 40737 40737 40737 40737 40737 31 ¡to ¡40 Mean 3.81 4.55 4.77 4.63 5.49 SD 1.54 1.21 1.35 1.42 1.12 n ¡= ¡ 14752 14752 14752 14752 14752 41 ¡to ¡50 Mean 3.85 4.7 4.96 4.72 5.41 SD 1.54 1.18 1.35 1.39 1.17 n ¡= ¡ 7668 7668 7668 7668 7668 51 ¡to ¡60 Mean 3.87 4.89 5.11 4.8 5.39 SD 1.54 1.18 1.31 1.38 1.2 n ¡= ¡ 3532 3532 3532 3532 3532 61 ¡and ¡older Mean 3.85 4.95 5.26 4.92 5.37 SD 1.49 1.17 1.3 1.34 1.26 n ¡= ¡ 905 905 905 905 905 Male

http://gosling.psy.utexas.edu/scales-weve-developed/ten-item-personality-measure- tipi/

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 45/64

Emotion vs. Mood vs. Sentiment

Let’s clear some terminology

• Affect : umbrella term for describing the topics of emotion, feelings, and moods
• Emotion:
• brief in duration
• consist of a coordinated set of responses (verbal, physiological, behavioral, and neural

mechanisms)

• triggered
• Mood:
• last longer
• less intense than emotions
• no trigger
• Sentiment:
• towards an object
• positive/negative

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 46/64

Models of Emotions

• Emotions are complex human experiences
• Evolutionary based
• Several definitions, we take with simple models, easy to incorporate in computers:
• Basic emotions
• Dimensional model
• Plutchik wheel
• Geneva Emotion Wheel

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 47/64

Basic Emotions

• Discrete classes model
• Different sets
• Darwin: Expression of emotions in man and animal
• Ekman definition (6 + neutral):
• Happiness
• Anger
• Fear
• Sadness
• Disgust
• Surprise

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 48/64

Dimensional model of Emotions

Three continuous dimensions

• Valence/Pleasure (positive-negative)

Each emotion is a point in the VAD space Self-Assessment Manikin (SAM)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 49/64

Dimensional model of Emotions

Three continuous dimensions

• Valence/Pleasure (positive-negative)
• Arousal (high-low )

Each emotion is a point in the VAD space Self-Assessment Manikin (SAM)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 49/64

Dimensional model of Emotions

Three continuous dimensions

• Valence/Pleasure (positive-negative)
• Arousal (high-low )
• Dominance (high-low )

Each emotion is a point in the VAD space Self-Assessment Manikin (SAM)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 49/64

Dimensional model of Emotions

Three continuous dimensions

• Valence/Pleasure (positive-negative)
• Arousal (high-low )
• Dominance (high-low )

Each emotion is a point in the VAD space Self-Assessment Manikin (SAM)

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 49/64

Dimensional model of Emotions

Three continuous dimensions

• Valence/Pleasure (positive-negative)
• Arousal (high-low )
• Dominance (high-low )

Each emotion is a point in the VAD space Self-Assessment Manikin (SAM) References

Bradley, M. M., and Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49–59. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 49/64

Circumplex model of Emotions

mapping between basic emotions and dimensional model References

Posner, J., Russell, J. a, and Peterson, B. S. (2005). The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and Psychopathology, 17(3), 715–734. https://doi.org/10.1017/S0954579405050340 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 50/64

Plutchik Wheel of emotions

• Robert Plutchik
• eight basic emotions: Joy, Trust, Fear,

Surprise, Sadness, Disgust, Anger and Anticipation.

• intensity (radius)
• twenty-four Primary, Secondary, and

Tertiary dyads (a feeling composed of two emotions)

• Primary dyad = one petal apart =

Love = Joy + Trust

• Secondary dyad = two petals apart =

Envy = Sadness + Anger

• Tertiary dyad = three petals apart =

Shame = Fear + Disgust

• Opposite emotions = four petals apart

= Anticipation != Surprise

References

Plutchik, R. (1982). A psychoevolutionary theory of emotions. Social Science Information, 21(4–5), 529–553. https://doi.org/10.1177/053901882021004003 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 51/64

Geneva Emotion Wheel

• model
• instrument

for measuring emotions References

Sacharin, V., Schlegel, K., and Scherer, K. R. (2012). Geneva Emotion Wheel Rating Study. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 52/64

Music-specific emotion model - GEMS

• domain-specific emotion model
• Geneva Emotional Music Scale (GEMS)
• 9-factorial model of music-induced emotions
• transcendence
• wonder
• joyful activation
• power
• tension
• sadness
• tenderness
• nostalgia
• peacefulness

References

Zentner, M., Grandjean, D., and Scherer, K. R. (2008). Emotions evoked by the sound of music: characterization, classification, and

• measurement. Emotion (Washington, D.C.), 8(4), 494–521. https://doi.org/10.1037/1528-3542.8.4.494

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 53/64

Modeling content with emotions

• user-related emotions
• how does (recommended) content relate to emotions?
• “I want to watch a funny movie tonight”

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 54/64

Modeling movies with emotions

• “A film is - or should be - more like music than like fiction. It should be a

progression of moods and feelings. The theme, what’s behind the emotion, the meaning, all that comes later.” – Stanley Kubrick

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 55/64

Modeling movies with emotions

• “A film is - or should be - more like music than like fiction. It should be a

progression of moods and feelings. The theme, what’s behind the emotion, the meaning, all that comes later.” – Stanley Kubrick

• “If my films make one more person miserable, I’ll feel I have done my job.” –

Woody Allen

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 55/64

Modeling movies with emotions

• “A film is - or should be - more like music than like fiction. It should be a

progression of moods and feelings. The theme, what’s behind the emotion, the meaning, all that comes later.” – Stanley Kubrick

• “If my films make one more person miserable, I’ll feel I have done my job.” –

Woody Allen

• “Through careful manipulation and good storytelling, you can get everybody to

clap at the same time, to laugh at the same time, and to be afraid at the same time.” – Steven Spielberg

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 55/64

Kurt Vonnegut story arc

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 56/64

Modeling movies with emotions - screenwriting workshop

Source: https://iedgameresearch.wordpress.com/2014/04/30/noel-mccauley-workshop-staging-emotional-environments/

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 57/64

Modeling movies with emotions

feature-film-length emotion maps http://yhvh.name/?w=-97&emap=1

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 58/64

Modeling music with emotion

(Juslin et al.) distinguish between three types of music-related emotions:

• expressed emotions
• the composer or performer wants to express
• perceived
• how a listener perceives (but not necessarily feels)
• induced
• truly felt by the listener

References

Juslin, P. N., and Laukka, P. (2004). Expression, Perception, and Induction of Musical Emotions: A Review and a Questionnaire Study

• f Everyday Listening. Journal of New Music Research, 33(3), 217–238. https://doi.org/10.1080/0929821042000317813

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 59/64

Music emotion detection from movements

• movements stimulated by music
• captured by mobile phone (raw

acceleration data)

• extract motion features
• linear regressoin predicting GEMS
• R2 between 0.14 and 0.50

References

Irrgang, M., and Egermann, H. (2016). From motion to emotion: Accelerometer data predict subjective experience of music. PLoS ONE, 11(7), 1–20. https://doi.org/10.1371/journal.pone.0154360 Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 60/64

Music emotion recognition (MER)

• perceived emotion = a single point in VA space
• annotations
• features:
• acoustic (loudness, timbre, pitch, rhythm,melody, harmony)
• lyrics
• models:
• regression techniques
• probabilistic (take into account the variance)

Dancing Queen (Abba) Civil war (GNR) Suzanne (Leonard Cohen) All I have to do is dream (Everly Brothers)

References

Wang, J.-C., Yang, Y., and Wang, H. (2016). Affective Music Information Retrieval. In M. Tkalčič, B. De Carolis, M. de Gemmis, A. Odić, and A. Košir (Eds.), Emotions and Personality in Personalized Services: Models, Evaluation and Applications (pp. 227–261). Springer. Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 61/64

Wrap-up

• Motivation: emotions and personality should be investigated

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 62/64

Wrap-up

• Motivation: emotions and personality should be investigated
• Models of personality and emotions

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 62/64

Wrap-up

• Motivation: emotions and personality should be investigated
• Models of personality and emotions
• Measurement with self-reporting

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 62/64

Next

• unobtrusive measurement
• usage in recommender systems

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 63/64

Questions?

Marko Tkalčič, RecSys2017SummerSchool-Part1-AffectRecSys 64/64