Methodological concerns emerging from matched sample design of - - PowerPoint PPT Presentation

methodological concerns emerging from matched sample
SMART_READER_LITE
LIVE PREVIEW

Methodological concerns emerging from matched sample design of - - PowerPoint PPT Presentation

Methodological concerns emerging from matched sample design of disabled people living in Hungary by Bori Simonovits, PhD Quality of Life and Costs of Living and Services of Disabled People in Various Residential Arrangements in Hungary


slide-1
SLIDE 1

Methodological concerns emerging from matched sample design of disabled people living in Hungary by Bori Simonovits, PhD

Quality of Life and Costs of Living and Services

  • f Disabled People in Various Residential

Arrangements in Hungary (VP/2013/013/0057)

slide-2
SLIDE 2

Structure of the presentation:

  • Background and aim of the research project on disabled

people living in Hungary

  • Why they are hard-to-survey?
  • Research design
  • Sampling: matched sampling design » criteria for

matching

  • Methodological challenges and limitations of the

research

  • Feedback on the results to the target group 
slide-3
SLIDE 3

Background of the project:

  • Objectives: (1) to examine and compare the quality of life of adults with

disabilities living in various residential arrangements; (2) to analyse the cost- effectiveness of the three residential arrangements in terms of social care provision.

  • Target groups: (1) adults with intellectual disabilities, (2) adults with autism

(ASD) (3) adults with profound intellectual and multiple disabilities (4) adults with psychosocial disabilities (schizophrenia)

  • Residential arrangements: (1) larger residential institutions, (2) smaller group

homes and (3) private households

  • Time frame: March 2015 – January 2016
  • Commissioned by the Equal Opportunities of Persons with Disabilities Non-profit Ltd.

(FSZK) and Hand in Hand Foundation (Kézenfogva Alapítvány)

slide-4
SLIDE 4

Disabled people as hard-to-survey target groups:

  • Based on the typology of Tourangeau (2014) on hard-to-survey groups:

– Hard-to-sample, due to the lack of proper sampling frame and low prevalence in the population (e.g. the prevalence of autism is around 0,6-1%, schizophrenia approx. 1%) – Hard-to-identify: esp. people with autism, due to the lack of diagnosis in case

  • f older generations (they have other primary diagnosis, mainly intellectual

disabilities) and false negative cases in case of psychosocial disabilities and motivated misreporting of schizophrenia – Hard-to-contact: esp. those who live at institutions (due to gatekeepers) and to certain extent family homes (could not be contacted or explicitly refused) – Hard-to-persuade: esp. people with psychosocial disabilities and people with autism (high refusal rate » response bias) – Hard-to interview: all target groups, but esp. people with profound intellectual and multiple disabilities » proxies were helped them to translate the questions;

slide-5
SLIDE 5

Research design: “linked questionnaires”:

  • A self-report “client questionnaire”:

– the respondent’s subjective well-being in eight quality of life domains and some basic socio-demographic characteristics and personal opinion; – Answered by the clients themselves or with the help of a „translator” carer upon request

  • A “carer questionnaire”:

– answered by a carer (paid or unpaid/family carer) who knows the client and their everyday life well; – provides information on the living conditions, infrastructure and some aspects of

  • bjective quality of life of the individual and it repeats the questions on quality of life

from the “client questionnaire”;

  • A “manager/head of household questionnaire”:

– answered by someone who is familiar with the financial situation – budget, income, spending etc. – of the setting where the participant lives (i.e. institution, group home,

  • r private household).

– On the costs and infrastructure of the social provision and care of the client

slide-6
SLIDE 6

Measuring Quality of life: based on (Schalock et

  • al. 2002)

(1) emotional well-being (2) material well-being (3) interpersonal relations (4) personal development (5) physical well-being (6) self-determination (7) social inclusion (8) rights

  • These QoL domains have been

validated both in terms of internal structure and stability by various international studies (see e.g. Verdugo et al. 2005) and in different cultural contexts (Schalock et al. 2005).

  • In Hungary we developed a scale
  • f 16 item scale (2 items per

domains)

  • Combined quality of life score (0-

16)

slide-7
SLIDE 7

Sampling and recruitment of the participants

Stratified sample, but not representative (taking into account location and client profile of the institutions and group homes); Various channels (day service providers, advocacy organisations) in case of recruiting private households; serious selection bias in all residential arrangements and all target groups; different case mix (by age, gender, degree of disability) in the different residential arrangements In order to eliminate bias matching was carried out, after the data collection

slide-8
SLIDE 8
  • To examine the quality of life outcomes and cost-effectiveness

across different residential arrangements, the criteria of matching varied by groups;

  • The basis of the matching:

– Intellectual disabilities and autism sub-samples: on the basis of adaptive behaviour (i.e. the collection of practical and social skills used in everyday life) that is considered a key determinant

  • f quality of life for people with intellectual disabilities (Ericsson-

Mansell 1996). – The sub-sample of people with psychosocial disabilities was matched on the basis of their age. (due to the lack of sufficient information on the time of the first diagnosis of their schizophrenia)

The aim of the research » matching criteria

slide-9
SLIDE 9

Composition of the achieved sample (N=359)

Intellectual disabilities Autism Profound intellectual and multiple disabilities Psychosocial disability Total prior to matching total 130 82 112 138 462 after matching Institution 33

  • 31

35 99 Group home 33 30 31 35 130 Private household 33 30 31 35 130 Total 99 60 93 105 359

slide-10
SLIDE 10

Methodological limitations and concerns

1. Internal validity concerns stemming from:

  • Lacking proper validation of the measurement tool;
  • The use of different groups as proxy respondents in

the different residential settings; 2. Selection bias throughout the sampling and recruitment process; 3. External validity concerns, resulting from the non representative (matched) sample;

slide-11
SLIDE 11
  • 1. Internal validity concerns # 1:
  • The quality of life question block was not validated properly

(due mainly to time constrains…) » Therefore scores on individual quality of life items and the combined quality of life scores are not comparable across target groups.

  • We focused on certain quality of life areas that are relatively

easy to measure by survey tools disregards others, that are equally important for quality of life (e.g. the quality

  • f interpersonal relationships was not measured).

» We were only able to measure certain aspects of QoL.

slide-12
SLIDE 12
  • 1. Internal validity concerns #2: different types of respondents
  • Clients: individual variations can occur in the ability to self-report,

particularly among those with more severe cognitive limitations;

  • Proxy respondents (if the clients were not able to answer the questions):

– different groups as proxy respondents in the different residential settings (i.e. paid carers and family carers) and we could not take into account any potential differences between their responses; – paid carers as proxies tend to rate quality of life higher than family carers, esp. in those cases when paid carers know the individual less well (Rand and Caiels 2015); » In general: Social desirability bias (i.e. respondents to answer questions in a manner that will be viewed favourably by others) might be a serious methodological concern in our study

slide-13
SLIDE 13
  • 2. Selection bias:

Although the study did not aim to use a representative sample, we must take into account that institutions and group homes that agreed to take part were systematically and significantly different from those who did not want to take part in research. Similarly, it is unclear to what extent families that were recruited via parents’ associations and advocacy organisations represent “typical” families. » these kinds of selection bias occurs in any types of surveys…

slide-14
SLIDE 14
  • 3. External validity concerns

Low number of observations

Case-by-case matching was not possible, groups were created based on: » adaptive behaviour categories (e. g independent in less than 4 areas/independent in 5-9 areas of everyday life); » age groups categories (e. g 30-47 years old / 48-65 years old) in case of the people with psychosocial disability; Our matching procedure was far from optimal…

External validity problems: International comparison?

  • 3 residential arrangements (HU) ↔ typically 2 arrangements are

compared (institutions and group homes);

  • 4 target groups (HU) ↔ international research examples typically focus
  • n 1 target group;
slide-15
SLIDE 15

Conclusion

  • Partly due to the research design (various residential

arrangements and too many target groups)

  • Partly due to the specific research goal» matched

sample design –Numerous methodological compromise have been made throughout the implementation of the research; –Raising serious validity (both internal and external) and reliability (mainly due to the interview situation itself and using proxies) concerns emerged...

slide-16
SLIDE 16

Finally…

  • We found important providing feedback

about research finding to the target group.

  • How to present results to the target

group..? » An easy-to-read summary was design to show the results of the empiric research...

slide-17
SLIDE 17

References:

  • Ericsson, Kent –Mansell, Jim (1996): Introduction: towards
  • deinstitutionalization. In Jim Mansell and Kent Ericsson (eds.):

Deinstitutionalization and Community Living. Intellectual disability services in Britain, Scandinavia and the USA. Springer, H. N. 1-16

  • Schalock, R. L., Verdugo, M. A., Jenaro, C., Wang, M., Wehmeyer, M.,

Jiancheng, X., & Lachapelle, Y. (2005). Cross-cultural study of quality of life

  • indicators. American journal of mental retardation: AJMR, 110(4), 298-311.
  • Schalock, R. L., Brown, I., Brown, R., Cummins, R. A., Felce, D., Matikka,

L., & Parmenter, T. (2002). Conceptualization, Measurement, and Application of Quality of Life for Persons With Intellectual Disabilities: Report

  • f an International Panel of Experts. Mental Retardation, 40(6), 457-470.
  • Schepers, W., Juchtmans, G., & Nicaise, I. (2015). Reaching out to hard-to-

survey groups among the poor. Survey protocols, statistical issues and research design. Milestone 20.5. Leuven: InGRID project.

  • TOURANGEAU R. EDWARDS B., JOHNSON T., WOLTER K., BATES N.
  • (eds., 2014), Hard-to-survey populations, Cambridge University Press,

Chapter 1

slide-18
SLIDE 18

THANK YOU FOR YOUR ATTENTION!

simonovits@tarki.hu

slide-19
SLIDE 19

APPENDICES

slide-20
SLIDE 20

Why involve people with disabilities directly?

  • Collection of quantitative data has traditionally been limited to proxy-

report.

  • Empowerment and recognition of the lived experiences of people with

IDD is important.

  • Quality of life is not only about objective circumstances, but subjective

aspects are equally important.

slide-21
SLIDE 21

Some issues and challenges in using self-report with people with IDD

– Difficulties with time and quantitative judgements; direct comparisons; socially reflexive questions; abstract concepts and generalised judgements; unfamiliar or sensitive content; multiple-choice formats. – Can lead to bias, such as acquiescence bias, last/first choice responding etc. The self-report questionnaires were extensively piloted with people with different disabilities and in different residential arrangements. As a result:

  • Number of QoL items reduced from 48 to 16 items (2 per domain).
  • Response scale changed from a 4-level frequency scale to a simple

Yes/No/So-so.

slide-22
SLIDE 22

HOW TO MEASURE QUALITY OF LIFE?

  • In the initial phases of the study the research team reviewed different

quality of life and life satisfaction questionnaires for people with disabilities developed and used in other countries.

  • As we could not identify a single instrument that met the requirements of
  • ur study and could have been readily adapted to the Hungarian context.
  • Therefore we relied on a number of existing instruments that all had in

common that are based on the internationally accepted conceptualisation

  • f quality of life (QoL)