Evaluating Soul City Evaluating Soul City A multi level A multi - - PowerPoint PPT Presentation

SLIDE 1

Evaluating Soul City Evaluating Soul City A multi level Communication for Social Change initiative Shereen Usdin A multi level Communication for Social Change initiative Shereen Usdin

SLIDE 2

Soul City Soul City

A Sout h Af r i c an NGO wor k i ng i n par t ne r s hi p wi t h l

• c

al NGOs i n 8 SADC c

• unt

r i e s t

• de

v e l

• p

i ndi g e nous CF SC i nt e r v e nt i

• ns

i n Sout he r n Af r i c a A Sout h Af r i c an NGO wor k i ng i n par t ne r s hi p wi t h l

• c

al NGOs i n 8 SADC c

• unt

r i e s t

• de

v e l

• p

i ndi g e nous CF SC i nt e r v e nt i

• ns

i n Sout he r n Af r i c a

SLIDE 3

We are complex beings… We are complex beings…

“ We do not g r

• w

abs

• l

ut e l y , c hr

• nol
• g

i c al l y . We g r

• w

s

• me

t i me s i n 1 di me ns i

• n

and not i n anot he r ; une v e nl y . We g r

• w

par t i al l y . We ar e r e l at i v e . We ar e mat ur e i n

• ne

r e al m, c hi l di s h i n anot he r . T he pas t , pr e s e nt and f ut ur e mi ng l e and pul l us bac k war d, f

• r

war d,

• r

f i x us i n t he pr e s e nt . We ar e made up

• f

l ay e r s , c e l l s , c

• ns

t e l l at i

• ns

. ” Anai sNi n*

*F r

• m

D. Re e l e r , ‘ A t he

• r

y

• f

Soc i al Chang e ” , CDRA

“ We do not g r

• w

abs

• l

ut e l y , c hr

• nol
• g

i c al l y . We g r

• w

s

• me

t i me s i n 1 di me ns i

• n

and not i n anot he r ; une v e nl y . We g r

• w

par t i al l y . We ar e r e l at i v e . We ar e mat ur e i n

• ne

r e al m, c hi l di s h i n anot he r . T he pas t , pr e s e nt and f ut ur e mi ng l e and pul l us bac k war d, f

• r

war d,

• r

f i x us i n t he pr e s e nt . We ar e made up

• f

l ay e r s , c e l l s , c

• ns

t e l l at i

• ns

. ” Anai sNi n*

*F r

• m

D. Re e l e r , ‘ A t he

• r

y

• f

Soc i al Chang e ” , CDRA

SLIDE 4

Individual

• Knowledge
• Attitudes/Beliefs/

/Risk perception

• Skills
• Self-efficacy
• Support for policy action
• Subjective norms
• Link to services

Community

• Dialogue/debate
• Social norms
• Diffusion of innovations
• Community efficacy
• Community action
• Social networks
• Social cohesion

Socio-economic + political national environment

• Healthy public policy
• Re-orientate services

S

• c

i a l M

• b

i l i s a t i

• n

M a s s M e d i a A d v

• c

a c y

The Soul City Model of Social Change The Soul City Model of Social Change

SLIDE 5

Experimental designs Experimental designs

• Gr
• up

1 g e t pi l l and c

• nt

r

• l

Gr p2, a pl ac e bo

• I

nv e s t i g at e c aus e

• and-

e f f e c t r e l at i

• ns

hi ps

• At

t r i but e e f f e c t t

• t

he i nt e r v e nt i

• n
• Gr
• up

1 g e t pi l l and c

• nt

r

• l

Gr p2, a pl ac e bo

• I

nv e s t i g at e c aus e

• and-

e f f e c t r e l at i

• ns

hi ps

• At

t r i but e e f f e c t t

• t

he i nt e r v e nt i

• n

SLIDE 6

So Soc i al al Chang hang e i s s c

• c
• mpl

e x e x and and s

• s
• CF

CF SC i nt nt e r e r v e v e nt i

• ns
• ns

ne e d e e d mo mor e e c

• c
• mpl

e x e x , nuanc nuanc e d e v e v al uat uat i

• n
• n

de de s i g n g ns So Soc i al al Chang hang e i s s c

• c
• mpl

e x e x and and s

• s
• CF

CF SC i nt nt e r e r v e v e nt i

• ns
• ns

ne e d e e d mo mor e e c

• c
• mpl

e x e x , nuanc nuanc e d e v e v al uat uat i

• n
• n

de de s i g n g ns

SLIDE 7

As exposure to Soul City increases … Can we say that Soul City causes increased condom use? People exposed to Soul City are also likely to be exposed to other HIV interventions. Condom us e i nc r e as e d

Challenges: noisy confounders Challenges: noisy confounders

????? ?????

SLIDE 8

• Caus

al i t y i s di f f i c ul t t

• pr
• v

e be c aus e al mos t impossible to get a pristine control group

• Chang

e i s not uni di r e c t i

• nal

and pe

• pl

e may we l l r e l aps e

• Mai

nt ai ni ng pos i t i v e be hav i

• ur

al s

• c
• unt

s

• Chang

e

• c

c ur s i n a c

• mpl

e x e nv i r

• nme

nt ( f

• r

e x ampl e a pr e s i de nt

• f

a c

• unt

r y s ay i ng t hat HI V doe s not c aus e AI DS –i t i s har d t

• me

as ur e t he ne g at i v e i mpac t

• f

t hi s )

• Caus

al i t y i s di f f i c ul t t

• pr
• v

e be c aus e al mos t impossible to get a pristine control group

• Chang

e i s not uni di r e c t i

• nal

and pe

• pl

e may we l l r e l aps e

• Mai

nt ai ni ng pos i t i v e be hav i

• ur

al s

• c
• unt

s

• Chang

e

• c

c ur s i n a c

• mpl

e x e nv i r

• nme

nt ( f

• r

e x ampl e a pr e s i de nt

• f

a c

• unt

r y s ay i ng t hat HI V doe s not c aus e AI DS –i t i s har d t

• me

as ur e t he ne g at i v e i mpac t

• f

t hi s )

More Challenges More Challenges

SLIDE 9

• Chang

e may not happe n

• v

e r ni g ht and e v al uat i

• ns

may be t

• e

ar l y t

• s

e e r e s ul t s

• Of

t e n ne e d t

• me

as ur e i nt e r me di at e i ndi c at

• r

s ( di al

• g

ue /i nt e nt i

• n…)

whi c h may pr e di c t s uc c e s s

• Unl

e s s y

• u

t e s t , i t i s har d t

• l

i nk di r e c t l y t

• c

hang e i n HI V i nc i de nc e /pr e v al e nc e

• Chang

e may not happe n

• v

e r ni g ht and e v al uat i

• ns

may be t

• e

ar l y t

• s

e e r e s ul t s

• Of

t e n ne e d t

• me

as ur e i nt e r me di at e i ndi c at

• r

s ( di al

• g

ue /i nt e nt i

• n…)

whi c h may pr e di c t s uc c e s s

• Unl

e s s y

• u

t e s t , i t i s har d t

• l

i nk di r e c t l y t

• c

hang e i n HI V i nc i de nc e /pr e v al e nc e

Things to consider… Things to consider…

SLIDE 10

Individual

• 1. National Quant

Survey

Community Society

• 2. Sentinel Site Studies
• 5. Media Monitoring

and Analysis

• 4. Partnership

Study

• 6. Institutional

/ Organisational Impact Study

• 7. Cost Effectiveness Study

3.National Qualitative Impact Assessment

Study Design Study Design

SLIDE 11

Evidence of impact attributed to the intervention Evidence of impact attributed to the intervention

• Qualitative evidence:

Qualitative evidence:

– Direct mention / direct attribution – Consistency – Data Triangulation

• Quantitative evidence:

Quantitative evidence:

– Simple descriptive analysis – More complex inferential analysis (regression analysis allowing us to make stronger associations – Dose response

• Qualitative evidence:

Qualitative evidence:

– Direct mention / direct attribution – Consistency – Data Triangulation

• Quantitative evidence:

Quantitative evidence:

– Simple descriptive analysis – More complex inferential analysis (regression analysis allowing us to make stronger associations – Dose response

SLIDE 12

Evidence of impact attributed to the intervention Evidence of impact attributed to the intervention

• Consistency of results

Consistency of results – Across evaluations of multiple SC interventions over time – Across studies within evaluations of one specific intervention

• Triangulation

Triangulation – Different Methods / designs – Different Researchers for each study – Different sources of data (ngo’s; gov; civil leaders etc)

• Consistency of results

Consistency of results – Across evaluations of multiple SC interventions over time – Across studies within evaluations of one specific intervention

• Triangulation

Triangulation – Different Methods / designs – Different Researchers for each study – Different sources of data (ngo’s; gov; civil leaders etc)

SLIDE 13

Speedy implementation

• f the Domestic

Violence Act

"There were pressures [to implement the DVA], there were pressures from occasions where people held marches and stuff like that.” [SA Police Services] “a lot of articles [on domestic violence and the DVA] started to appear … they create expectations in the community which makes the government deliver….” [Government’s Victim Empowerment Unit, Domestic Violence Task Team]

“Soul City influenced us to organize the march, emotions were high … “ [Rural Female] Participation in public protest against VAW “I think my work with Soul City and also some of the NGOs, the Network for Violence Against Women, opens your eyes.” [Government - Domestic Violence Task Team] Comparison over the evaluation period: press coverage and Soul City / NNVAW Domestic Violence press releases

Media Advocacy Community mobilization

Direct lobbying

DV related coverage attributable to the SC / NNVAW partnership Initiated by the partnership, and associated with exposure to the edutainment vehicle

10 20 30 40 June July Aug Sept Oct Nov Dec % of total coverage

VAW: Non-DV press coverage VAW: DV press coverage DV Press Releases

1 3 3 5 2 4 6 %

No SC 1 source 2 sources 3 sources

* 49% of print media monitored dealt with / referred to DV * Approximately 1 in 5 articles directly referred to SC and/or the NNVAW

SLIDE 14

Some textured stuff is measurable E.g. dialogue * subjective norms * social cohesion * networking Some textured stuff is measurable E.g. dialogue * subjective norms * social cohesion * networking

F Fi ig gu ur re e 2 2: : P Pe er rc ce ep pt ti io

• n

ns s

• f

f s so

• c

ci ia al l n no

• r

rm ms s b by y e ex xp po

• s

su ur re e t to

• S

So

• u

ul l C Ci it ty y m me ed di ia a

Most people in your community disagree that a man is right in expecting a woman to have sex with him without using a condom

40 60 80

No SC exposure 1 SC media source 2 SC media sources 3 SC media sources

"Disagree" - Evaluation Survey %

SLIDE 15

“ N “ Not e v e v e r e r y t y t hi hi ng ng t hat hat c

• unt
• unt

s s c an an be be c

• unt
• unt

e d e d. An And not not e v e v e r e r y t y t hi hi ng ng t hat hat c an an be be c

• unt
• unt

e d e d, c

• unt
• unt

s . s . ”– ”– Al Al be be r t Ei Ei ns ns t e i e i n* n*

**F r

• m

D. Re e l e r , ‘ A t he

• r

y

• f

Soc i al Chang e ” , CDRA

“ N “ Not e v e v e r e r y t y t hi hi ng ng t hat hat c

• unt
• unt

s s c an an be be c

• unt
• unt

e d e d. An And not not e v e v e r e r y t y t hi hi ng ng t hat hat c an an be be c

• unt
• unt

e d e d, c

• unt
• unt

s . s . ”– ”– Al Al be be r t Ei Ei ns ns t e i e i n* n*

**F r

• m

D. Re e l e r , ‘ A t he

• r

y

• f

Soc i al Chang e ” , CDRA

But….. But…..

SLIDE 16

“Markers” “Markers”

SLIDE 17

SLIDE 18

• Some

t i me s y

• ur

t e ar s c

• me

down whe n y

• u

ar e wat c hi ng be c aus e t he par t t hat y

• u

s e e l

• ok

s l i k e r e al l y how y

• u

l i v e . . .

( Mal e , L i mpopo, Rur al , 25- 35)

• Some

t i me s y

• ur

t e ar s c

• me

down whe n y

• u

ar e wat c hi ng be c aus e t he par t t hat y

• u

s e e l

• ok

s l i k e r e al l y how y

• u

l i v e . . .

( Mal e , L i mpopo, Rur al , 25- 35)

SLIDE 19

SLIDE 20

New Stuff: propensity score analysis New Stuff: propensity score analysis

SLIDE 21

People assigned to groups based

• n propensibility to be exposed so

they are “comparable / similar” Exposure group “Control group”

?

Knowledge Attitudes Subjective norms Intentions Self- efficacy Behaviour

SLIDE 22

Into the future…. Into the future….

– More advanced methodologies to investigate change at community level through social networks/capital/cohesion – experimenting with “less complex” sentinel sites @ schools – Explore advances in quantitative analysis – More advanced methodologies to investigate change at community level through social networks/capital/cohesion – experimenting with “less complex” sentinel sites @ schools – Explore advances in quantitative analysis

SLIDE 23

Is Soul City exposure over 10 years associated with social change and more positive behaviours?

… especially young people exposed to Soul Buddyz (possible link with decrease in HIV rates in latest ANC survey in youngest age group)

Analysis of 3 year panel study with same people examining impact over time Longitudinal qualitative studies – with children Using complexity theory to analyse data Is Soul City exposure over 10 years associated with social change and more positive behaviours?

… especially young people exposed to Soul Buddyz (possible link with decrease in HIV rates in latest ANC survey in youngest age group)

Analysis of 3 year panel study with same people examining impact over time Longitudinal qualitative studies – with children Using complexity theory to analyse data