Intended for the 2015 FedCASIC Meeting by James R Caplan PhD James - - PowerPoint PPT Presentation

Intended for the 2015 FedCASIC Meeting by James R Caplan PhD James R. Caplan, PhD.

This presentation is my own and does not represent any Official Position of the Department of Defense

 Cognitive psychology formed around how words  Cognitive psychology formed around how words

and ideas are connected. Think Berkeley, Hume and John Stuart Mill from the 18th Century J y

 Reemerged in the 1950s based on the WWII focus

• n human performance and attention,

p developments in computer science, especially artificial intelligence, and interest in linguistics. h k Ch k d Cl ll d Think Chomsky and McClelland

 Basis of my graduate training. 1968 Masters

th i b d d i ti thesis based on word associations

2

 Early studies were human-powered  Early studies were human-powered  Subjects sorted statements into “buckets”

based on how they seemed to “go together” based on how they seemed to go together

 Group would discuss results

d h and reach consensus

3

 Category definitions changed with ongoing  Category definitions changed with ongoing

context – requiring resorting

 Definitions were hard to keep in mind as  Definitions were hard to keep in mind as

number of buckets increased past 9 or 10

 Consensus building arbitrary and results  Consensus-building arbitrary and results

unreliable across groups

4

 Employee attitude surveys typically included  Employee attitude surveys typically included

• pen-ended questions, comments, and

“Other/Specify” responses Other/Specify responses

 Contractors sanitized personal information,

places and expletives then categorized and places, and expletives then categorized and coded answers

5

 Expensive time-consuming  Expensive, time-consuming  Added months to final analysis, obviating the

advantages of computer administration advantages of computer administration

 Eventually, open-ended questions were

dropped from our employee surveys dropped from our employee surveys

6

 Important way to know if some questions  Important way to know if some questions

were confusing or ambiguous

 Lost alternatives we never considered  Lost alternatives we never considered  The ability for respondents to interact with

us: perhaps an important positive motivator us: perhaps, an important positive motivator

7

 SPSS comes up with “Text Analysis for Surveys ”  SPSS comes up with Text Analysis for Surveys,

• approx. 2008 with promise of automated coding and

categorizing R lit li d t di ti i d i t i

 Reality: relies on data dictionaries and intensive

human intervention- it’s a note taker

 Quote from Roller and Lavrakes (2015), “computer  Quote from Roller and Lavrakes (2015), computer

software programs can provide important assistance in the coding of manifest content, but these programs cannot handle the coding of complex latent programs cannot handle the coding of complex latent content for which the human brain is best suited."

 Problem: extensive preparation required, no context

sensitivity, no serious natural language processing

 Three versions later, no real improvement

8

 2009 IBM purchases SPSS  2009, IBM purchases SPSS  Many other solutions emerge, Cognos, IBM

Media Analytics Watson just by IBM Media Analytics, Watson, just by IBM

 Other solutions emerge but emphasis on

marketing research biological and medical marketing research, biological and medical research, brand and product preferences, analysis of Big Data and national security analysis of Big Data, and national security

 University of Maryland, Institute for Advanced

Computing develops Topic Analysis based on Computing develops Topic Analysis, based on natural language processing

9

 Application of topic analysis to survey data  Application of topic analysis to survey data  Solves the problem of valence/affect (known as

sentiment analysis by market researchers) f

 Sanitized dataset from 2008  Question: “If you have comments or concerns that

you were not able to express in answering this you were not able to express in answering this survey, please enter them in the space provided. Any comments you make on this questionnaire will be kept confidential and no follow up action will be kept confidential, and no follow-up action will be taken in response to any specifics reported.”

 Very preliminary results – IBM threw this into Watson

y p y with no instructions. I haven’t had a chance to interact with it yet

10

11

1

Comments about the Survey itself (too

1.

Comments about the Survey, itself (too long, redundant) (59.2% negative)

2

Comments about specific questions (50 2%

2.

Comments about specific questions (50.2% negative)

3

Comments about the organization (57 7%

3.

Comments about the organization (57.7% negative)

4

Comments about work/job satisfaction

4.

Comments about work/job satisfaction (55.1% negative) Etc

5.

Etc.

12

 Survey researchers need to explore these  Survey researchers need to explore these

tools

• Refine our sentiment analysis
• Refine our sentiment analysis
• See how clusters correlate with known

demographics g p

• Check out some “Other/Specify” responses

 Still requires the human brain but the heavy

lifting can be done for us with these techniques

13

james.r.caplan2.civ@mail.mil

14