Putting Text in Context: How to Estimate Better Left-Right Positions - - PowerPoint PPT Presentation

▶
putting text in context how to estimate better left right
SMART_READER_LITE
LIVE PREVIEW

Putting Text in Context: How to Estimate Better Left-Right Positions - - PowerPoint PPT Presentation

Feb 10, 2023 186 likes •452 views

Putting Text in Context: How to Estimate Better Left-Right Positions by Scaling Party Manifesto Data Kenneth Benoit aubler Thomas D LSE and Trinity College Dublin University of Mannheim and MZES Kenneth Benoit and Thomas D

slide-1
SLIDE 1

Putting Text in Context: How to Estimate Better Left-Right Positions by Scaling Party Manifesto Data

Kenneth Benoit§ Thomas D¨ aubler∗

§LSE and Trinity College Dublin ∗University of Mannheim and MZES Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-2
SLIDE 2

Questions

◮ How should we understand the “left-right dimension”? ◮ How should we construct a left-right index using content

analysis results from political documents (here: manifestos)?

◮ What does this say about the past, present, and future of

measuring left-right?

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-3
SLIDE 3

Locating Parties on a Left-Right Dimension

◮ Two approaches

◮ a priori: specify a fixed scale in advance Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-4
SLIDE 4

Locating Parties on a Left-Right Dimension

◮ Two approaches

◮ a priori: specify a fixed scale in advance ◮ a posteriori: estimate a scale using inductive methods from

data (higher-dimensional measurements)

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-5
SLIDE 5

Locating Parties on a Left-Right Dimension

◮ Two approaches

◮ a priori: specify a fixed scale in advance ◮ a posteriori: estimate a scale using inductive methods from

data (higher-dimensional measurements)

◮ One common application: estimating left-right from category

counts based on human-coding of policy documents

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-6
SLIDE 6

Locating Parties on a Left-Right Dimension

◮ Two approaches

◮ a priori: specify a fixed scale in advance ◮ a posteriori: estimate a scale using inductive methods from

data (higher-dimensional measurements)

◮ One common application: estimating left-right from category

counts based on human-coding of policy documents

◮ The CMP/MARPOR’s main deliverable is its fixed scale: RILE

“Its a priori, deductive nature is important in allowing its application in all places at all times without the qualifications about content or context which apply to inductive scales. It is a substantively invariant measure whose numeric values always carry the same meaning.” (Budge and Meyer 2013: 88)

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-7
SLIDE 7

An IRT Model for Counts of Unordered Categorical Outcomes

◮ IRT modelling (Rasch 1960, Bock 1972)

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-8
SLIDE 8

An IRT Model for Counts of Unordered Categorical Outcomes

◮ IRT modelling (Rasch 1960, Bock 1972) ◮ Let yij represent the counts of coded text units from

i = 1, ..., N documents, falling into category j = 1, ..., J. yij ∼ Neg.Binomial(µij, φj) log(µij) = αi + ζj + λjθi

◮ Interpretation:

αi the variable length of the document (total length) ζj the baseline frequency of a category (issue coverage) λj responsiveness of the category to the latent variable θi the latent variable (e.g. left-right position) 1/φj

  • verdispersion parameter

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-9
SLIDE 9

Parameters and Outcomes

−3 −2 −1 1 2 3 50 100 150

Expected category counts (α = 1 and all ζ = 1)

Latent position θ Expected number of items λ = −1 λ = −0.5 λ = 0 λ = 0.5 λ = 1 −3 −2 −1 1 2 3 0.0 0.2 0.4 0.6 0.8 1.0

Item response category functions (any α, all ζ = 1)

Latent position θ Expected probability/proportion λ = −1 λ = −0.5 λ = 0 λ = 0.5 λ = 1

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-10
SLIDE 10

Identification

◮ There are five fundamental indeterminacies ◮ The following constraints constitute one way of identifying the

model: ζj = 0 ¯ λj = 0 θi ∼ N(0, 1) λj > λj′

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-11
SLIDE 11

Inference

◮ Bayesian approach (using HMC in Stan) with priors as follows:

αi ∼ N(µα, σα) ζj ∼ N(µζ, σζ) λj ∼ N(0, σλ) θi ∼ N(0, 1) µα ∼ N(0, 5) µζ ∼ N(0, 5) σα ∼ Half-Cauchy(0, 5) σζ ∼ Half-Cauchy(0, 5) σλ ∼ Half-Cauchy(0, 5) 1/φ ∼ Uniform(0, 200)

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-12
SLIDE 12

The CMP/MARPOR Data

◮ Human coding of 3000+ election manifestos from 50+ countries ◮ Standard coding scheme has 56 categories, some directed, others

not

◮ Frequently used “canned” RILE index

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-13
SLIDE 13

And Now for Some Results...

◮ Sample of 2288 manifestos from post-1972 democracies ◮ The most and least frequent categories (ζ values):

Top 10 Categories Category ζ Welfare State Expansion + 2.20

  • Environm. Protection +

1.72 Technology and Infrastr. + 1.70 Education Expansion + 1.66 Political Authority + 1.63 Social Justice + 1.53

  • Gov. and Admin. Effic. +

1.50 Non-Econ. Demogr. Gr. + 1.37 Democracy + 1.35 Farmers + 1.25 Bottom 10 Categories Category ζ Marxist Analysis +

  • 3.37

Education Limitation +

  • 3.22

Labour Groups -

  • 2.71

National Way of Life -

  • 2.18

Centralisation +

  • 1.86

Foreign Special Rel. -

  • 1.46

Protectionism -

  • 1.45

Constitutionalism -

  • 1.29

Corporatism +

  • 1.23

Keynesian Dem. Man. +

  • 1.15

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-14
SLIDE 14

Left-Right Positions in Italy: CMP Rile

M5S Five Star Movement RC Civil Revolution SEL Left Ecology Freedom Autonomy Progress Federalism PD Democratic Party 3L Labour and Freedom List SC Civic Choice PdL People of Freedom SVP South Tyrolean People’s Party FDI−CDN Brothers of Italy CD Democratic Centre UdC Union of the Center UdC Union of the Center LN Northern League PD Democratic Party IdV List Di Pietro − Italy of Values PdL People of Freedom

  • 2008

2013 −40 −20 20 40

CMP rile Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-15
SLIDE 15

Left-Right Positions in Italy: Poisson IRT

PD Democratic Party SEL Left Ecology Freedom M5S Five Star Movement RC Civil Revolution Autonomy Progress Federalism UdC Union of the Center SVP South Tyrolean People’s Party 3L Labour and Freedom List CD Democratic Centre SC Civic Choice FDI−CDN Brothers of Italy PdL People of Freedom LN Northern League IdV List Di Pietro − Italy of Values PD Democratic Party UdC Union of the Center PdL People of Freedom

  • 2008

2013 −2 −1 1 2

Scaling Estimates of Theta Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-16
SLIDE 16

Left-Right Positions in Italy: NegBin IRT

M5S Five Star Movement SEL Left Ecology Freedom RC Civil Revolution PD Democratic Party Autonomy Progress Federalism 3L Labour and Freedom List SVP South Tyrolean People’s Party SC Civic Choice PdL People of Freedom CD Democratic Centre UdC Union of the Center FDI−CDN Brothers of Italy LN Northern League IdV List Di Pietro − Italy of Values UdC Union of the Center PD Democratic Party PdL People of Freedom

  • 2008

2013 −2 −1 1 2

Scaling Estimates of Theta Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-17
SLIDE 17

Validation with Expert Survey Placements

2 4 6 8 10 −3 −2 −1 1 2 Expert Survey Estimates of Left−Right Scaling Estimates of Theta

  • r = 0.82 W Europe

r = 0.55 E Europe r = 0.89 Pacific & N America r = 0.75 All

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-18
SLIDE 18

Country-by-Country Validation (vs. CMP Rile)

  • 0.2

0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

  • Corr. of CMP rile w experts
  • Corr. of IRT w experts

Romania not shown (r for rile -.55, r for IRT theta -.44) Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-19
SLIDE 19

ˆ λ (post-1972 democracies)

Military − Nationalisation + Anti−Imperial./−Colonial. Peace + Controlled Economy + Labour Groups + Democracy + Economic Planning + Internationalism + Market Regulation + Welfare State Expansion + Protectionism + Education Expansion + Marxist Analysis + National Way of Life − Anti−Growth Economy + Traditional Morality − Social Justice + Multiculturalism + European Integration −

  • Environm. Protection +
  • Underpriv. Minority Gr. +

Keynesian Dem. Man. + Foreign Special Rel. − Non−Econ. Demogr. Gr. + Decentralisation + Political Corruption − Culture + Internationalism − Corporatism + Foreign Special Rel. + Constitutionalism − Economic Goals Middle Class, Profess. Gr. + Technology and Infrastr. + European Integration + Centralisation + Farmers + Productivity +

  • Gov. and Admin. Effic. +

Multiculturalism − Education Limitation + Labour Groups − Freedom and Hum. Rights + Constitutionalism + Social Harmony + Political Authority + Law and Order + Incentives + Protectionism − Economic Orthodoxy + National Way of Life + Military + Traditional Morality + Free Enterprise + Welfare State Limitation +

  • Right

Neutral Left −3 −2 −1 1 2 3 λ ^

slide-20
SLIDE 20

ˆ λ (post-1972 democracies)

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3

Western Europe λ ^

j

E Europe λ ^

j

101 102 103 104 105 106 107 108 109 110 201 202 203 204 301 302 303 304 305 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 501502 503 504 505 506 507 601 602 603 604 605 606 607 608 701 702 703 704 705 706

Rile LEFT Rile RIGHT R = 0.37

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3

Western Europe λ ^

j

Pacific λ ^

j

101 102 103 104 105 106 107 108 109 110 201 202 203 204 301 302 303 304 305 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 501502 503 504 505 506 507 601 602 603 604 605 606 607 608 701 702 703 704 705 706

Rile LEFT Rile RIGHT R = 0.64

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3

Western Europe λ ^

j

N America λ ^

j

101 102 103 104 105 106 107 108 109 110 201 202 203 204 301 302 303 304 305 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 501502 503 504 505 506 507 601 602 603 604 605 606 607 608 701 702 703 704 705 706

Rile LEFT Rile RIGHT R = 0.49

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3

Western Europe λ ^

j

Far East λ ^

j

101 102 103 104 105 106 107 108 109 110 201 202 203 204 301 302 303 304 305 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 501502 503 504 505 506 507 601 602 603 604 605 606 607 608 701 702 703 704 705 706

Rile LEFT Rile RIGHT R = −0.097

slide-21
SLIDE 21

Take Me to Another Dimension

◮ In the two-dimensional case, we model the mean of the counts

as: log(µij) = αi + ζj + λ1jθ1i + λ2jθ2i

◮ The model requires 13 constraints for identification

◮ Mean of positions zero in each dimension ◮ “Dimension founders” ◮ 1 single, 1 single, 1 double, 2 further zero constraints on λdj ◮ Reference category for ζj Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-22
SLIDE 22

ˆ λd for 2D-model (post-1972 democracies)

Internationalism − Anti−Imperial./−Colonial. European Integration − Military − Political Authority + Foreign Special Rel. − Social Justice + Peace + Centralisation + Internationalism +

  • Environm. Protection +

Constitutionalism + Constitutionalism − Foreign Special Rel. +

  • Gov. and Admin. Effic. +

Culture + European Integration + Decentralisation + Farmers + Military + Marxist Analysis + Nationalisation + Anti−Growth Economy + Controlled Economy + Labour Groups + Keynesian Dem. Man. + Market Regulation + Welfare State Expansion + Protectionism + Corporatism + Middle Class, Profess. Gr. + Education Expansion + Economic Planning + Economic Goals Technology and Infrastr. + Productivity + Incentives + Protectionism − Economic Orthodoxy + Welfare State Limitation + Free Enterprise + Education Limitation + Labour Groups −

  • Pure Economic

Economic and Social −3 −2 −1 1 2 3

β ^

(a) Economic Dimension

National Way of Life − Multiculturalism + Traditional Morality −

  • Underpriv. Minority Gr. +

Non−Econ. Demogr. Gr. + Democracy + Freedom and Hum. Rights + Social Harmony + Law and Order + Traditional Morality + Multiculturalism − National Way of Life + Decentralisation +

  • Environm. Protection +

Military − Culture + Farmers + European Integration + Peace + Internationalism + Constitutionalism − Social Justice +

  • Gov. and Admin. Effic. +

Foreign Special Rel. + Constitutionalism + Centralisation + Military + European Integration − Anti−Imperial./−Colonial. Foreign Special Rel. − Political Authority + Internationalism −

  • Economic and Social

Pure Social −3 −2 −1 1 2 3

β ^

(b) Social Dimension

slide-23
SLIDE 23

Validation with Expert Survey Placements - 2D

  • 2

4 6 8 10 −4 −2 2

Economic

Expert Survey Estimates of Taxes vs Spending Scaling Estimates of Theta (Economic Dimension) r = 0.73

  • 2

4 6 8 10 −4 −2 2

Social

Expert Survey Estimates of Social Dimension Scaling Estimates of Theta (Social Dimension) r = 0.53

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-24
SLIDE 24

Using Alternative Items - Belgian CAP data

SP AGALEV ECOLO PS PSC CD&V VU VLD PRL VB AGALEV ECOLO SP PS PSC CVP PRL PVV VU VB ECOLO AGALEV SP PS PSC CVP VU VB VLD PRL AGALEV SPA CD&V SPIRIT VLD NVA VB

  • 2003

1995 1991 1999 −3 −2 −1 1 2 Scaling Estimates of Theta

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-25
SLIDE 25

Validation of Belgian CAP Data Results

Expert Surveys

−1.5 −0.5 0.5 1.0

0.83 0.84

−20 −15 −10 −5 5 2 3 4 5 6 7

0.79

−1.5 −0.5 0.5 1.0

  • Agendas Project IRT

0.46 0.43

  • CMP IRT

−1.0 −0.5 0.0 0.5

0.84

2 3 4 5 6 7 −20 −15 −10 −5 5

  • −1.0

−0.5 0.0 0.5

  • RILE

Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data

slide-26
SLIDE 26

Conclusion

◮ IRT model as inductive approach for inferring a left-right

dimension from category counts

◮ Left-right as a super-issue rather than defined by content ◮ Superior to fixed content-based index (RILE) ◮ Can incorporate a priori information explicitly ◮ Flexible in terms of extensions:

◮ Differential item functioning ◮ Model the parameters with covariates ◮ . . . Kenneth Benoit and Thomas D¨ aubler Scaling Manifesto Data