Measurement: Concepts in Practice Department of Government London - - PowerPoint PPT Presentation

▶
measurement concepts in practice
SMART_READER_LITE
LIVE PREVIEW

Measurement: Concepts in Practice Department of Government London - - PowerPoint PPT Presentation

Sep 17, 2023 104 likes •980 views

Observation and Description Measurement Assessing Measurement Quality Measurement: Concepts in Practice Department of Government London School of Economics and Political Science Observation and Description Measurement Assessing Measurement

slide-1
SLIDE 1

Observation and Description Measurement Assessing Measurement Quality

Measurement: Concepts in Practice

Department of Government London School of Economics and Political Science

slide-2
SLIDE 2

Observation and Description Measurement Assessing Measurement Quality

1 Observation and Description 2 Measurement 3 Assessing Measurement Quality

slide-3
SLIDE 3

Observation and Description Measurement Assessing Measurement Quality

1 Observation and Description 2 Measurement 3 Assessing Measurement Quality

slide-4
SLIDE 4

Observation and Description Measurement Assessing Measurement Quality

Goals of Descriptive Research

1 To answer research questions 2 To generate research questions

slide-5
SLIDE 5

Observation and Description Measurement Assessing Measurement Quality

Goals of Descriptive Research

1 To answer research questions 2 To generate research questions 3 To do both of these, iteratively

slide-6
SLIDE 6

Observation and Description Measurement Assessing Measurement Quality

Ex.: Galileo’s Drawings of Sunspots

Source: Public Domain, NASA

slide-7
SLIDE 7

Observation and Description Measurement Assessing Measurement Quality

Ex.: Broad Street Cholera 1854 outbreak of cholera in London

Around Broad Street (Soho) 616 eventual deaths

slide-8
SLIDE 8

Observation and Description Measurement Assessing Measurement Quality

Ex.: Broad Street Cholera 1854 outbreak of cholera in London

Around Broad Street (Soho) 616 eventual deaths

Causal RQ: What causes transmission of cholera?

slide-9
SLIDE 9

Observation and Description Measurement Assessing Measurement Quality

Ex.: Broad Street Cholera 1854 outbreak of cholera in London

Around Broad Street (Soho) 616 eventual deaths

Causal RQ: What causes transmission of cholera? Descriptive RQ: Who exactly is contracting cholera?

slide-10
SLIDE 10

Observation and Description Measurement Assessing Measurement Quality

slide-11
SLIDE 11

Observation and Description Measurement Assessing Measurement Quality

slide-12
SLIDE 12

Observation and Description Measurement Assessing Measurement Quality

slide-13
SLIDE 13

Observation and Description Measurement Assessing Measurement Quality

slide-14
SLIDE 14

Observation and Description Measurement Assessing Measurement Quality

slide-15
SLIDE 15

Observation and Description Measurement Assessing Measurement Quality

slide-16
SLIDE 16

Observation and Description Measurement Assessing Measurement Quality

slide-17
SLIDE 17

Observation and Description Measurement Assessing Measurement Quality

slide-18
SLIDE 18

Observation and Description Measurement Assessing Measurement Quality

slide-19
SLIDE 19

Observation and Description Measurement Assessing Measurement Quality

slide-20
SLIDE 20

Observation and Description Measurement Assessing Measurement Quality

Ex.: Gapminder Data

slide-21
SLIDE 21

Observation and Description Measurement Assessing Measurement Quality

Quantitative Description

slide-22
SLIDE 22

Observation and Description Measurement Assessing Measurement Quality

Quantitative Description

A rectangular, case-by-variable dataset

“dataset observations” (DSOs)

slide-23
SLIDE 23

Observation and Description Measurement Assessing Measurement Quality

Quantitative Description

A rectangular, case-by-variable dataset

“dataset observations” (DSOs)

A clear unit of analysis

slide-24
SLIDE 24

Observation and Description Measurement Assessing Measurement Quality

Quantitative Description

A rectangular, case-by-variable dataset

“dataset observations” (DSOs)

A clear unit of analysis Multiple cases/units

slide-25
SLIDE 25

Observation and Description Measurement Assessing Measurement Quality

Quantitative Description

A rectangular, case-by-variable dataset

“dataset observations” (DSOs)

A clear unit of analysis Multiple cases/units Quantitative and qualitative measures

slide-26
SLIDE 26

Observation and Description Measurement Assessing Measurement Quality

Quantitative Description

A rectangular, case-by-variable dataset

“dataset observations” (DSOs)

A clear unit of analysis Multiple cases/units Quantitative and qualitative measures Calculation of summary statistics

slide-27
SLIDE 27

Observation and Description Measurement Assessing Measurement Quality

Dataset Observation (DSO)

“a score for a case on a variable”

State Year Var1 Var2 Afghanistan 2016 1 TRUE Afghanistan 2015 1 TRUE Algeria 2016 1 FALSE Algeria 2015 TRUE . . .

slide-28
SLIDE 28

Observation and Description Measurement Assessing Measurement Quality

Dataset Observation (DSO)

“a score for a case on a variable”

State Year Var1 Var2 Afghanistan 2016 1 TRUE Afghanistan 2015 1 TRUE Algeria 2016 1 FALSE Algeria 2015 TRUE . . .

slide-29
SLIDE 29

Observation and Description Measurement Assessing Measurement Quality

Dataset Observation (DSO)

“a score for a case on a variable”

State Year Var1 Var2 Afghanistan 2016 1 TRUE Afghanistan 2015 1 TRUE Algeria 2016 1 FALSE Algeria 2015 TRUE . . .

slide-30
SLIDE 30

Observation and Description Measurement Assessing Measurement Quality

Dataset Observation (DSO)

“a score for a case on a variable”

State Year Var1 Var2 Afghanistan 2016 1 TRUE Afghanistan 2015 1 TRUE Algeria 2016 1 FALSE Algeria 2015 TRUE . . .

slide-31
SLIDE 31

Observation and Description Measurement Assessing Measurement Quality

Description Beyond DSOs

DSOs are not the only kind of data Non-DSOs do not fit in a rectangular dataset

slide-32
SLIDE 32

Observation and Description Measurement Assessing Measurement Quality

Description Beyond DSOs

DSOs are not the only kind of data Non-DSOs do not fit in a rectangular dataset Sometimes hear about “qualitative” and “quantitative” research

This divide is illusory because all research is qualitative and some involves quantitative data description

Return to this in a few weeks

slide-33
SLIDE 33

Observation and Description Measurement Assessing Measurement Quality

slide-34
SLIDE 34

Observation and Description Measurement Assessing Measurement Quality

1 Observation and Description 2 Measurement 3 Assessing Measurement Quality

slide-35
SLIDE 35

Observation and Description Measurement Assessing Measurement Quality

An Example: Opinion

Opinion is a summary evaluation of a particular object Only one necessary feature: evaluation/favorability How do we measure this?

slide-36
SLIDE 36

Observation and Description Measurement Assessing Measurement Quality

Operationalization

1 Measure features

Level of measurement How to score each case on each feature Be concrete

2 Aggregate feature measurements

Sum? Average? AND logical? Level of measurement of final scale Range of possible values Justify against criticisms/alternatives

slide-37
SLIDE 37

Observation and Description Measurement Assessing Measurement Quality

Operationalization I

To study concepts, we need to be able to observe those concepts and encode them as variables

slide-38
SLIDE 38

Observation and Description Measurement Assessing Measurement Quality

Operationalization I

To study concepts, we need to be able to observe those concepts and encode them as variables The definition of variable:

A dimension that describes an

  • bservation
slide-39
SLIDE 39

Observation and Description Measurement Assessing Measurement Quality

Operationalization I

To study concepts, we need to be able to observe those concepts and encode them as variables The definition of variable:

A dimension that describes an

  • bservation

Or, the operationalization of a concept

slide-40
SLIDE 40

Observation and Description Measurement Assessing Measurement Quality

Some definitions!

Variable: A dimension that describes an

  • bservation

Operationalization: the process of deciding on measures for concepts Coding: Assigning a score for a variable to an observation

slide-41
SLIDE 41

Observation and Description Measurement Assessing Measurement Quality

Some definitions!

Variable: A dimension that describes an

  • bservation

Operationalization: the process of deciding on measures for concepts Coding: Assigning a score for a variable to an observation

Manual or hand coding Automated coding

slide-42
SLIDE 42

Observation and Description Measurement Assessing Measurement Quality

Operationalization II Concept

Attribute Attribute Attribute Concept Definition

slide-43
SLIDE 43

Observation and Description Measurement Assessing Measurement Quality

Operationalization II Concept

Attribute Attribute Attribute Concept Definition

Measure(s) Measure(s) Measure(s)

Operation- alization

slide-44
SLIDE 44

Observation and Description Measurement Assessing Measurement Quality

Operationalization III Definition

slide-45
SLIDE 45

Observation and Description Measurement Assessing Measurement Quality

Operationalization III Definition → Feature

slide-46
SLIDE 46

Observation and Description Measurement Assessing Measurement Quality

Operationalization III Definition → Feature → Indicator(s)

slide-47
SLIDE 47

Observation and Description Measurement Assessing Measurement Quality

Operationalization III Definition → Feature → Indicator(s)

Indicators might be scaled or potential alternatives

slide-48
SLIDE 48

Observation and Description Measurement Assessing Measurement Quality

Operationalization III Definition → Feature → Indicator(s)

Indicators might be scaled or potential alternatives

slide-49
SLIDE 49

Observation and Description Measurement Assessing Measurement Quality

Example: Democracy Democracy

How do we operationalize this concept?

slide-50
SLIDE 50

Observation and Description Measurement Assessing Measurement Quality

Example: Democracy Democracy → Free and fair elections

How do we operationalize this concept?

slide-51
SLIDE 51

Observation and Description Measurement Assessing Measurement Quality

Example: Democracy Democracy → Free and fair elections → ?

How do we operationalize this concept?

slide-52
SLIDE 52

Observation and Description Measurement Assessing Measurement Quality

Questions?

slide-53
SLIDE 53

Observation and Description Measurement Assessing Measurement Quality

Once we have an operationalization, coding turns observations of attributes into DSOs Case Measure1 Measure2 Measure3 UK ? ? ? France ? ? ? Germany ? ? ? Spain ? ? ? . . .

slide-54
SLIDE 54

Observation and Description Measurement Assessing Measurement Quality

Types of Measures

1 Categorical

Binary

2 Ordinal 3 Interval

Qualitative Quantitative

Note: Ratio scale measures are interval measures with a non-arbitrary zero value

slide-55
SLIDE 55

Observation and Description Measurement Assessing Measurement Quality

Activity

Concept: Democracy Attribute: Free and fair elections Measure:

1 Categorical 2 Ordinal 3 Numeric

slide-56
SLIDE 56

Observation and Description Measurement Assessing Measurement Quality

Why do we care?

Once we have measured variables for

  • bservations, we can conduct analysis!
slide-57
SLIDE 57

Observation and Description Measurement Assessing Measurement Quality

Why do we care?

Once we have measured variables for

  • bservations, we can conduct analysis!

And once we have analysis, we can draw inferences and make evidence-based claims.

slide-58
SLIDE 58

Observation and Description Measurement Assessing Measurement Quality

Questions?

slide-59
SLIDE 59

Observation and Description Measurement Assessing Measurement Quality

1 Observation and Description 2 Measurement 3 Assessing Measurement Quality

slide-60
SLIDE 60

Observation and Description Measurement Assessing Measurement Quality

Assessing Measurement Quality

1 Conceptual clarity 2 Construct validity

Convergent validity Divergent validity

3 Accuracy and precision

slide-61
SLIDE 61

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures I

Conceptual clarity is about knowing what we want to measure Sloppy concepts make for bad measures

Ambiguity Vagueness

slide-62
SLIDE 62

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures I

Conceptual clarity is about knowing what we want to measure Sloppy concepts make for bad measures

Ambiguity Vagueness

Revise concept definition as needed

slide-63
SLIDE 63

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures II

Construct validity is the degree to which a variable measures a concept

slide-64
SLIDE 64

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures II

Construct validity is the degree to which a variable measures a concept Construct validity is high if a variable is a measure of the concept we care about

slide-65
SLIDE 65

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures II

Construct validity is the degree to which a variable measures a concept Construct validity is high if a variable is a measure of the concept we care about Construct validity is low if a variable is actually a measure of something else

slide-66
SLIDE 66

Observation and Description Measurement Assessing Measurement Quality

Example: Polity IV1

Institutionalized Democracy: Democracy is conceived as three essential, interdependent elements. One is the presence of institutions and procedures through which citizens can express effective preferences about alternative policies and leaders. Second is the existence of institutionalized constraints on the exercise of power by the executive. Third is the guarantee of civil liberties to all citizens in their daily lives and in acts of political participation. Other aspects of plural democracy, such as the rule of law, systems of checks and balances, freedom of the press, and so on are means to, or specific manifestations

  • f, these general principles. We do not include coded data on

civil liberties.

1http://www3.nd.edu/ mcoppedg/crd/PolityIVUsersManualv2002.pdf

slide-67
SLIDE 67

Observation and Description Measurement Assessing Measurement Quality

Example: Polity IV1

Institutionalized Democracy: Democracy is conceived as three essential, interdependent elements. One is the presence of institutions and procedures through which citizens can express effective preferences about alternative policies and leaders. Second is the existence of institutionalized constraints on the exercise of power by the executive. Third is the guarantee of civil liberties to all citizens in their daily lives and in acts of political participation. Other aspects of plural democracy, such as the rule of law, systems of checks and balances, freedom of the press, and so on are means to, or specific manifestations

  • f, these general principles. We do not include coded data on

civil liberties.

1http://www3.nd.edu/ mcoppedg/crd/PolityIVUsersManualv2002.pdf

slide-68
SLIDE 68

Observation and Description Measurement Assessing Measurement Quality

Example: Polity IV1

Institutionalized Democracy: Democracy is conceived as three essential, interdependent elements. One is the presence of institutions and procedures through which citizens can express effective preferences about alternative policies and leaders. Second is the existence of institutionalized constraints on the exercise of power by the executive. Third is the guarantee of civil liberties to all citizens in their daily lives and in acts of political participation. Other aspects of plural democracy, such as the rule of law, systems of checks and balances, freedom of the press, and so on are means to, or specific manifestations of, these general principles. We do not include coded data on civil liberties.

1http://www3.nd.edu/ mcoppedg/crd/PolityIVUsersManualv2002.pdf

slide-69
SLIDE 69

Observation and Description Measurement Assessing Measurement Quality

Example: Polity IV1

Institutionalized Democracy: Democracy is conceived as three essential, interdependent elements. One is the presence of institutions and procedures through which citizens can express effective preferences about alternative policies and leaders. Second is the existence of institutionalized constraints on the exercise of power by the executive. Third is the guarantee of civil liberties to all citizens in their daily lives and in acts of political participation. Other aspects

  • f plural democracy, such as the rule of law, systems of checks

and balances, freedom of the press, and so on are means to, or specific manifestations of, these general principles. We do not include coded data on civil liberties.

1http://www3.nd.edu/ mcoppedg/crd/PolityIVUsersManualv2002.pdf

slide-70
SLIDE 70

Observation and Description Measurement Assessing Measurement Quality

Example: Polity IV1

Institutionalized Democracy: Democracy is conceived as three essential, interdependent elements. One is the presence of institutions and procedures through which citizens can express effective preferences about alternative policies and leaders. Second is the existence of institutionalized constraints on the exercise of power by the executive. Third is the guarantee of civil liberties to all citizens in their daily lives and in acts of political participation. Other aspects of plural democracy, such as the rule of law, systems of checks and balances, freedom of the press, and so

  • n are means to, or specific manifestations of, these general
  • principles. We do not include coded data on civil liberties.

1http://www3.nd.edu/ mcoppedg/crd/PolityIVUsersManualv2002.pdf

slide-71
SLIDE 71

Observation and Description Measurement Assessing Measurement Quality

slide-72
SLIDE 72

Observation and Description Measurement Assessing Measurement Quality

Assessing Construct Validity

Multiple measures! Look for:

Convergence (Convergent validity) Discrimination (Discriminant validity)

slide-73
SLIDE 73

Observation and Description Measurement Assessing Measurement Quality

Assessing Construct Validity

Multiple measures! Look for:

Convergence (Convergent validity) Discrimination (Discriminant validity)

For example, the multi-trait, multi-method matrix

slide-74
SLIDE 74

Observation and Description Measurement Assessing Measurement Quality

Using Multiple Indicators

Choose the “best” one Apply an AND operator

Must have all indicators to be coded 1 Treat indicators as “ordinal” in Gerring’s sense

Scale the indicators (e.g., sum or mean)

slide-75
SLIDE 75

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures III

slide-76
SLIDE 76

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures III Accuracy

slide-77
SLIDE 77

Observation and Description Measurement Assessing Measurement Quality

Accurate

Synonyms: true, correct, unbiased, valid

Image Source: Wikimedia, Public Domain

slide-78
SLIDE 78

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures III Accuracy

slide-79
SLIDE 79

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures III Accuracy Precision

slide-80
SLIDE 80

Observation and Description Measurement Assessing Measurement Quality

Precise

Synonyms: certain, exact, specific, low variance

Image Source: Wikimedia, Nevit Dilmen

slide-81
SLIDE 81

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures III Accuracy Precision

slide-82
SLIDE 82

Observation and Description Measurement Assessing Measurement Quality

Assessing Measures III Accuracy Precision Reliability

slide-83
SLIDE 83

Observation and Description Measurement Assessing Measurement Quality

Reliable

Synonyms: dependable, replicable, repeatable, consistent

Typically used in the context of: Multiple measures used in a scale Multiple scores at different times Multiple individuals coding using one method

slide-84
SLIDE 84

Observation and Description Measurement Assessing Measurement Quality

Questions?

slide-85
SLIDE 85

Observation and Description Measurement Assessing Measurement Quality

Key Points

1 We want to make claims about concepts 2 But we only observe and can only

analyse observed, measured variables

3 So our task as scientists is to:

Link the concepts we care about to

  • bservable phenomena

Draw out theoretical implications from measures

slide-86
SLIDE 86