education academic and administrative staff. Conducted consultancies - - PowerPoint PPT Presentation

Organised by:

SAT 29

JUNE 2019

• Born in UK
• Worked in higher education for over 35 years.
• Vice-rector at USJ and Director of the DLCR.
• Quality assurance, accreditation, educational development and training for higher

education academic and administrative staff.

• Conducted consultancies and technical reports for governments, in several countries.
• Reviewer for 25 international journals and for several leading international

publishers.

• Author of

17 academic books, over 100 academic journal papers, over 50 international conference papers, chapters in 15 books, and over 300 other educational publications.

• Prize winning music scholar and cathedral choral scholar.
• Organist, a member of the Royal College of Organists and a Fellow of the Royal

Society of Arts.

• Prof. KEITH MORRISON

Vice-Rector and Director of DLCRE University of Saint Joseph - Macau

SAT 29

JUNE 2019

RESEARCH EVIDENCE IN EDUCATION

Keith Morrison

SAT 29

JUNE 2019

SEARCHING FOR CERTAINTY

• ‘Knowledge is not so precise a concept as is commonly thought .

. . A great many things which have been thought to be certain turn out to be untrue, and there’s no shortcut to knowledge. . . . [W]e ought not to be dogmatic’ (Bertrand Russell)

SAT 29

JUNE 2019

EXTRACTS FROM THE MISSION, VISION AND AIMS

• Mission statement: “. . . to have an impact on education and society in Macau and beyond”.
• Vision statement: “. . .

a centre of excellence in research in education, through quality research training, dissemination, uptake and impact”.

• Aims:
• To have an impact on educational work and society in Macau and beyond, linking

research and educational practice, policy making, decision making and action.

• To disseminate the results of research through a range of channels and to be a resource

for educational research findings and their dissemination.

• To run training and development on research in education.
• To provide seminars, workshops, conferences etc. on research in education.

SAT 29

JUNE 2019

2009 2002

2018 2016

SAT 29

JUNE 2019

2017 2017

SAT 29

JUNE 2019

Evidence-based learning and ‘What Works’

2019

SAT 29

JUNE 2019

“In education, new policies and interventions are rarely based on good prior evidence of effectiveness and of their side effects. Many policy areas are evidence-resistant . . . .” (p. 3).

SAT 29

JUNE 2019

TWO WORLDS

RESEARCHERS Evidence

POLICY MAKERS

Agendas

SAT 29

JUNE 2019

2009 2012

SAT 29

JUNE 2019

SAT 29

JUNE 2019

• School students tend to already know 40-50 per

cent of what we teach them, but the 40-50 per cent differs across students.

• Roughly one third of what students learn will be

unique to them, i.e. not known by any other student.

• We have little or no idea of what is going on

inside a student’s mind.

• If students encounter a concept on at least 3

different occasions, there is an 80% chance that they will know it after 6 months.

• 80% of the feedback that students get is from

each other, and 80% of that is wrong.

REFORMULATE ‘WHAT WORKS’

• ‘What works, for whom, in the presence of which factors and conditions and

in the absence of which factors and conditions, singly or in combination, in what ways and with what side effects, following what causal chains, how successfully, based on what and whose evidence and criteria, with what level of trustworthiness, and in whose eyes?’

SAT 29

JUNE 2019

TEN CAUTIONS IN USING EDUCATIONAL RESEARCH

1. The need for robust research (internal validity). 2. External validity, transferability and generalizability. 3. Generalizability and transferability through research syntheses, meta-analysis and meta-meta-analysis. 4. Appropriate and secure methods of data analysis 5. Educational research and certainty. 6. ‘Evidence’ and neutrality. 7. Educational research, policy making and practice: ‘is’ and ‘ought’, ethics, morals and values. 8. Privileging certain types of educational research. 9. Teachers as decision makers in the ‘what works’ agenda.

• 10. Teachers’ professional responsibility in using research.

SAT 29

JUNE 2019

THE NEED FOR ROBUST RESEARCH (INTERNAL VALIDITY)

• The research design does not fit the research purposes or the research questions.
• Causality is confused with correlation.
• Insufficient controls are placed on the data and the data analysis.
• Counterfactuals, comparison groups and counterfactual analysis are absent.
• Claims for causality are spurious.
• The specific contingencies of specific situations are excluded, overlooked or neglected.
• Studies are too small scale for correct statistical power, statistical analysis or secure

generalization.

• Missing data and attritions bring about unreliability in the data analysis.
• Replication studies are few and unreliable (unrepeatable).
• Measurements are not as exact as they are taken to be.
• Conflicts of interest may suppress negative findings.

SAT 29

JUNE 2019

EXTERNAL VALIDITY, TRANSFERABILITY AND GENERALIZABILITY

• Just because research has shown that such-and-such might ‘work’ in a such-and-such a

research setting, this is no reason to believe that it will work in a different temporal, locational, contextual setting, or even the same setting, a second time.

• Findings don’t always travel well. ‘Causal roles do not travel well’ (p. 88). Contexts,

causal conditions and their relative strengths differ.

Cartwright & J. Hardie (2012) Evidence-Based Policy: A Practical Guide to Doing It Better. New York, NY: Oxford University Press.

?

SAT 29

JUNE 2019

MANY WORLDS

RESEARCHERS PARENTS STUDENTS EDUCATIONISTS POLICY MAKERS The Alliance for Useful Evidence (2018): ‘We identified examples

• f

different

• rganisations

reaching different conclusions about the same intervention;

• ne

thought it worked well, and the

• ther was less confident’.

SAT 29

JUNE 2019

GOVERNMENT HEALTH WARNING

• ‘What works’ should indicate the limits of its applicability, its

negative as well as positive effects, the limits on what can and cannot be safely concluded from the evidence, the conditions under which it might or might not ‘work’, and a covering statement to say that, actually, it might not ‘work’ in specific contexts.

SAT 29

JUNE 2019

GENERALIZABILITY AND TRANSFERABILITY THROUGH RESEARCH SYNTHESES, META-ANALYSIS AND META-META-ANALYSIS

• Meta-analysis
• Meta-meta-analyses
• Research syntheses
• Systematic reviews

SAT 29

JUNE 2019

EDUCATIONAL ENDOWMENT FOUNDATION

SAT 29

JUNE 2019

SAT 29

JUNE 2019

2009 2012

SAT 29

JUNE 2019

2014 2019

SAT 29

JUNE 2019

• A synthesis of over 800 meta-analyses of over 50,000 studies,

150,000 effect sizes, and circa 240 million students.

• Later increased to over 1500 meta-analyses.

Heavily criticised: Bergeron 2017: ‘pseudo-science Slavin (2018):

• Overlooks the importance of smaller effects achieved by

different groups;

• Uses effect size studies that are correlational, with insufficient

controls, lacking in control groups, or are pre-post designs);

• Uses studies without sufficient attention to the meaning or

quality of these studies, i.e. does not screen them sufficiently. SA T 29 JU NE 20 19

• 1. Meta-analysis is simply ‘statistical alchemy for the 21st century. Important

inconsistencies are ignored and buried in statistical slurry.’

Feinstein, A. (1995). Meta-analyses: statistical alchemy for the 21st century. Journal of Clinical Epidemiology, 48(1), pp. 71-79.

• 2. Hattie: ‘is merely shovelling meta-analyses containing massive bias into meta-meta-

analyses that reflect the same biases’.

Slavin, R. E. (2018) John Hattie is Wrong. Robert Slavin’s Blog, 21 June, 2018. https://robertslavinsblog.wordpress.com/2018/06/21/john-hattie-is-

wrong/

• 3. See, B.H. (2018): quality problems; sampling problems; scope; conflicts of interest;

biased reporting; insufficient warrants for the conclusions drawn and claims made; data quality; lack of trials; non-experimental designs; lack of clarity; data analysis.

See, B.H. (2018) 'Evaluating the evidence in evidence-based policy and practice : examples from systematic reviews of literature.' Research in Education, 102 (1). pp. 37-61.

SAT 29

JUNE 2019

APPROPRIATE AND SECURE METHODS OF DATA ANALYSIS

• Statistical analysis uses the wrong

statistics

• Null hypothesis significance testing is

discredited

• Effect size is used incorrectly
• Coding damages the integrity of

qualitative data

• Misinterpreting data and what they

(claim to) show

SAT 29

JUNE 2019

EDUCATIONAL RESEARCH AND CERTAINTY

‘The enormous problem faced in basing policy on research is that it is almost

impossible to make educational policy that is not based on research. Almost every educational practice that has ever been pursued has been supported with data by

• somebody. I don’t know a single failed policy, ranging from the naturalistic teaching
• f reading, to the open classroom, to the teaching of abstract set-theory in third-

grade math that hasn’t been research-based. Experts have advocated almost every conceivable practice short of inflicting permanent bodily harm.’

(https://www.mathematicallycorrect.com/edh2cal.htm. Address to California State Board of Education, April 10, 1997, by E.

• D. Hirsch, Jr.)

SAT 29

JUNE 2019

‘EVIDENCE’ AND NEUTRALITY

• ‘Evidence’ is not neutral.
• ‘Evidence’ is what we do with data.
• ‘Evidence’ is the substance of making a case; to prove beyond a

reasonable doubt, to support, to challenge, to confirm, to disconfirm to contribute to a claim.

• What is the burden of proof?

SAT 29

JUNE 2019

AN ALPHABET OF EVIDENCE REQUIREMENTS

A. Admissible; B. Verisimilitude; C. Coherence; D. Justifiability; E. Contextuality; F. Ethically, legally, fairly and properly obtained; G. Relevant and logically defensible for the case in point; H. Fair, i.e. not prejudiced or misleading; I. Factual (rather than circumstantial); J. Sufficient to be convincing; K. Worthwhile (‘evidential worth’); L. Of sufficient weight, salience and importance to warrant inclusion; M. Open to challenge and interrogation; N. At a high standard of proof; O. Clear and defensible presumptions and assumptions; P. Reliable, trustworthy, valid and believable; Q. Cogent and compelling; R. True (the truth, the whole truth and nothing but the truth, with no errors of commission or omission); S. Unprejudiced and disinterested: objective; T. Clear on type and status (e.g. hearsay, fact, opinion, first-hand,

• ral, written etc.);

U. Provided by suitably competent people/sources (involving the question of ‘who is the expert?’); V. Verifiable and able to be corroborated; W. Consistent; X. Demonstrating probity (e.g. goodness and absolute moral correctness); Y. Presented in a suitable, appropriate form; Z. Available for public disclosure. SAT 29

JUNE 2019

EDUCATIONAL RESEARCH, POLICY MAKING AND PRACTICE: ‘IS’ AND ‘OUGHT’; ETHICS; MORALS AND VALUES

• Educational policy making and educational practice need more than

research; they concern values, ethics and morals.

• ‘Is’ and ‘ought’ are separate questions.
• Moving from descriptive to normative statements is a category error.
• Conducting poor quality research is unethical.

Educational research has an ethical and moral

• bligation to ensure that it clears a high bar of rigour

and that the claims made from it are defensible.

SAT 29

JUNE 2019

PRIVILEGING CERTAIN TYPES OF EDUCATIONAL RESEARCH

Level 1 (highest level) Well conducted, suitably powered randomised control trial (RCT) Level 2 Well conducted, but small and under-powered RCT Level 3 Non-randomised observational studies Level 4 Non-randomised study with historical controls Level 5 (lowest level) Case series without controls

Bagshaw, S. M. & Bellomo, R. (2008) The need to reform our assessment of evidence from clinical trials: A commentary. Philosophy, Ethics, and Humanities in Medicine, 3 (23), pp. 1-11.

SAT 29

JUNE 2019

TEACHERS AS DECISION MAKERS IN ‘WHAT WORKS’

• ‘Teachers are not going to be given a recipe for ‘what works’ from research; by its

nature, educational research cannot provide certainty of outcome. What it can achieve is to provide reasonable warrant for decisions that must be taken by teachers, in full knowledge of the circumstances in which they work’ (Winch et al., 2015, p. 210).

• ‘Researchers should not presume to tell teachers how to deal with the situations

they face’ (Sheldon, 2016, p. 3).

Sheldon, J. (2016) “What Works” doesn’t work: The problem with the call for evidence based practices in the classroom. Paper Badass Teachers Association White Paper Collection (2). http:www.badassteacher.org. November 6, 2016. Winch, C. Oancea, A and Orchard, J. (2015) The contribution of educational research to teachers’ professional learning: Philosophical understandings. Oxford Review of Education, 4 (2), pp. 202-216.

SAT 29

JUNE 2019

TEACHERS’ PROFESSIONAL RESPONSIBILITY IN USING RESEARCH

• Teachers, as a professional and moral

duty, should be aware of the best research evidence available, together with regarding it with some caution.

Ontario College of Teachers

SAT 29

JUNE 2019

• 1. Catholic schooling
• 2. Inclusive Education
• 3. Early years education
• 4. ICT
• 5. Leadership and management
• 6. Curriculum and pedagogy
• 7. Education and society
• 8. Education and well-being

8 INITIAL RESEARCH STRANDS

SAT 29

JUNE 2019

• Research:
• The Catholic Schools in Macau: Roots, Identity and Mission;
• Inclusive Education: From Theory to Practice;
• Catholic Schooling, Identity and Social Justice in Macau.
• Conferences, talks and seminars:
• Looking to the Future with Confidence: Renewing out Commitment to Catholic Education;
• Effective and Constructive Ways to Handle the Media during a Crisis;
• 'What Works' best in Education;
• Excellence through Diverse Classrooms.
• Courses:
• Training Inspectors for Catholic Schools.
• Workshops:
• Support workshops for young researchers.

INITIAL PROJECTS

SAT 29

JUNE 2019

Thank you

www.dlcre.usj.edu.mo dlcre@usj.edu.mo

SAT 29

JUNE 2019