Welcome! Audio for this webinar is available through your computers - - PowerPoint PPT Presentation

▶
welcome
SMART_READER_LITE
LIVE PREVIEW

Welcome! Audio for this webinar is available through your computers - - PowerPoint PPT Presentation

Dec 06, 2023 777 likes •1.23k views

Welcome! Audio for this webinar is available through your computers web browser as part of the live stream. Make sure your speakers are unmuted and your volume is turned up to hear the audio. Closed-captioning is also available. At the

slide-1
SLIDE 1

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Welcome!

  • Audio for this webinar is available through your computer’s web browser as part of the

live stream. Make sure your speakers are unmuted and your volume is turned up to hear the audio.

  • Closed-captioning is also available. At the bottom of your audience console are

multiple application widgets that you can use. If you need closed caption click on the red “CC” widget at the bottom of your screen.

  • A full recording of the webinar will be posted in about 24 hours at the same link used

to participate live.

1

slide-2
SLIDE 2

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Delaware ● District of Columbia ● Maryland ● New Jersey ● Pennsylvania

Using data to refine your remote learning strategies: The Evidence to Insights (e2i) Coach

Mikia Manley Brian Gill Researcher, Director, REL Mid-Atlantic REL Mid-Atlantic

slide-3
SLIDE 3

I E S ~I~ Institute of

?ti~ Education Sciences

Our Presenters

  • / e21Coach /

tll

Mathematica solution for progress

Mikia Manley Researcher REL Mid-Atlantic Brian Gill Director REL Mid-Atlantic

3

slide-4
SLIDE 4

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Agenda

Introductions Key research questions during COVID-19 A brief overview of the Evidence to Insights (e2i) Coach What you need to get started Q&A

4

slide-5
SLIDE 5

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Key terms

  • Remote learning is a method of instruction in which teachers and students are separated by

time, location, or both and interact through technology-based and non-technology-based mechanisms.

  • Virtual learning is instruction during which students and teachers are separated by time,

location, or both and interact through Internet-connected computers or other electronic devices (National Forum for Education Statistics 2015).

  • Synchronous: A teacher and student or students interacting at the same time (via video, phone,

chat, or other medium).

  • Asynchronous: Students work on their own, and interaction with teacher is not simultaneous

(for example, via email, text message, or completing instructional packets).

5

slide-6
SLIDE 6

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Key research questions during COVID-19

6

slide-7
SLIDE 7

School closures across the United States, April 2020

Coronavirus & School c·

L

..

, v

Coronavirus & School-Building Clos.u

CLOSED INTO APRIL CLOSED INTO MAY CLOSED OR ACADEMIC YEAR CLOSED UNTI l FURT ER NOTICE

q~

~ RECOMME OED CLOSURE

1

I E

S ~I~

Institute of

  • -

"1tl~ Education Sciences

Source: EdWeek, Map: Coronavirus and school closures 2020

7

slide-8
SLIDE 8

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

8

slide-9
SLIDE 9

Figure 1. Mathematics forecast

240

School! Closure

I

Typical Last Day

.

.

230

8th Grade

7th Grade

1

  • k~r:------------ =
  • -------_-_-

____________

_

  • .

220

(1) ~

S 210

J.....

  • a:

6th Gra _

  • 200

4th Grade 180

Typical Growth/Summer LO,SS, COVI0 Slowdown COVI[) SUde Sept 1 Oct 1 Nov 1 Dec 1 Jan 1 Feb 1 Mar 1

Institute of Education Sciences

...........

  • I e ,. •

. .

  • --------------~--------------
.. • • ■ I ..
  • I
  • I
I I
  • ~
  • ii •
  • .. .

. .

.

. .

. .

..

· ........ .

....... . .

.

. .. ..... . . .

' . ..

Apr 1 May 1 Jun 1 Jul 1 Aug 1 Sept 1

nwea

RESEARCH

Figure 2. Read ng forecast 230 220

Qi) 210

'-

(J

en

~ -

a: 200

190

180

School Closure

I

Typical Last Day

  • ------------:
  • -----~~~ · ~-

I

a K=.=.t---:.l '-- .-. ~ r: r. ~. -. ~. -. ;-- - -. :- :-. :-. -. ':',

=

"-

8th Grade·

7th Grade,

6th Grade,

  • Sept 1

Oc't 1

  • Typ,

ical Growth/Summer Loss COVI0 Slowdown

COVID Slide

  • •••• I •••••••••••• f
  • T

I ..... :· ............•...............

Ah r
  • ----- - ----- • -----
m e- ~
  • ■
.. I
  • I
I I I I I ii I I
  • e •
I ■ I
  • ■
I ■ I I
  • . e I
  • I
I

I ............. .

.

.

......... .

.

~-------------------------

  • ,t •
I I I I ■ 1 I 1 1 I
  • e • "' I
I
  • ......
, ..................

.

~~~~~~~~-------:--------------

  • ••••
  • l
  • e • •
I .. ·

·····•~ ............ .

  • I
■ e • • 4 • t, •
  • .

Nov 1 Dec 1 Jan 1

Feb 1 Mar · 1 Apr 1 May 1 Jun 1

Jul 1

Aug 1

Sept 1

nweaRESEARCH

Students’ projected learning losses

Melt refers to students gaining no ground during the school closures. Slide refers to students losing ground during the school closures.

Source: Tarasawa and Kuhfeld 2020

9

slide-10
SLIDE 10

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Schools are facing a lot of challenges

  • Promoting students’ well-being
  • Navigating federal and state

guidance

  • Maintaining financial health
  • Providing continuity in academic

services

  • Ensuring equity
  • Protecting students’ data and

privacy

10

slide-11
SLIDE 11 ·a

I E S ~I~

Institute of

?ti~ Education Sciences

Schools are facing a lot of challenges

11

slide-12
SLIDE 12

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

The research landscape: Making the most of evidence-informed insights

Strategy What might we take away from this? Online learning Face-to-face or synchronous time with teachers is critical Feedback, tutoring, and Students require ongoing feedback, support, and relationships with their support teachers and their peers Project-based learning Employ human-centered design principles to solve problems and foster

  • ut-of-the-box thinking

Gaming or virtual simulations Employ activities that engage students and use techniques drawn from behavioral science to remove friction points Plan beyond content delivery Provide resources and materials to help supplement student learning

12

slide-13
SLIDE 13

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Some key research questions

  • Which remote learning strategies most effectively engage students?
  • Which remote learning strategies best promote academic achievement?
  • Which strategies are most effective for specific students, such as students with

disabilities or English language learners?

  • Which strategies support and engage parents?

13

slide-14
SLIDE 14
  • 1

E S ~I~ Institute of

"1t1~ Education Sciences

Data are critical for answering key learning questions

Attendance Grades on state assessments Engagement Feedback

Data we typically use

14

slide-15
SLIDE 15

1

E s ~I~

Institute of

~ ~

Education Sciences

·

471,~

I

r ,,

Data are critical for answering key learning questions

Data that might be available while buildings are closed

  • Account logins
  • Session duration
  • Synchronous

attendance

  • Synchronous

participation

  • Posts on

class discussion boards

  • Competency-based

assessments

  • In-app assessments
  • Assignment or project

completion

  • Module completion
  • Surveys of

students, parents, and teachers

15

slide-16
SLIDE 16

A continuum of evidence

Low confidence

Quasi-

in results Percentage of students logging on to a tool at least once per day

experimental (matched Anecdotal Pre/Post comparison) Descriptive Correlative

High confidence in results

Experimental (Randomized Controlled Trial)

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

16

slide-17
SLIDE 17

A continuum of evidence

Percentage of students attending synchronous meetings Low confidence in results

Anecdotal Descriptive Pre/Post Correlative Quasi- experimental (matched comparison)

High confidence in results

Experimental (Randomized Controlled Trial)

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

17

slide-18
SLIDE 18

A continuum of evidence

Low confidence

Quasi-

in results Association between session duration and assessment scores

experimental (matched Anecdotal Pre/Post comparison) Descriptive Correlative

High confidence in results

Experimental (Randomized Controlled Trial)

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

18

slide-19
SLIDE 19

A continuum of evidence

Low confidence

Quasi-

in results

experimental (matched Anecdotal Pre/Post comparison) Descriptive Correlative

High confidence in results

Experimental (Randomized Controlled Trial)

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

19

slide-20
SLIDE 20

)-J f.

i

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Example research questions

  • All outcomes

– Does app A increase [fill in the blank] compared with app B?

  • Student performance

– Does assigning four hours of content per day increase assessment performance compared with assigning two hours

  • f content per day?
  • Student engagement

– Does having one check-in per day increase students’ engagement compared with having two check-ins per week? – Does delivering content synchronously increase students’ engagement compared with delivering content asynchronously? – Do letter grade policies increase students’ engagement compared with pass/fail policies? – Does using a mobile-friendly app increase students’ engagement compared with using a non-mobile-friendly app?

Phoenix's high school district under

  • ur member Chad Geston has

launched an ambitious “Every student, every day” effort that is exactly what it sounds like: every kid in the district having a check-in with an adult, every day. –EdWeek “We realize that if we tell kids today, ‘Hey, your grade can’t be any lower than it is now,’ or if we tell them we’re not going to grade them for the rest of the year, we’re going to have a big chunk of kids check out,” – Curtis Hicks, asst. superintendent, Salem City district, VA , via EdWeek

20

slide-21
SLIDE 21

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

A brief overview of the e2i Coach

21

slide-22
SLIDE 22

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Using evidence to your advantage

What works In your context For whom

22

slide-23
SLIDE 23

The Coach will reco1n1nend an approach to evaluate your program or practioe.

Institute of Education Sciences

The Coach will help you design an evaluation based on

the outcomes you are interested in and your

unique contex. . The Coach will use your data to create two groups - a

treatment group and a

si1nilar cornparison group. The Coach will

(.;7\

auto1natically

~

conduct your analysis and provide you with results. The Coach will compile I

u

your results and all ~f h the inforn1ation you

have entered into one succint repor .

Visit e2icoach.org

23

slide-24
SLIDE 24

New User NU

MY EVALUATIONS > P ILOT O F MAT H U

EVALUATION OVERVIEW 0 l ATEST TEAM ACT IV ITY

Pilot of MathU

e2 icoach. team@g m ... General Update 3/26/2020, 21 :35

The purpose of this evaluation is to determine whether the MathU program, a

  • nline platform , incr eases math achievement among students.

Team: e2icoach.team@gmail.com

EVALUATION STEPS

  • l. GETTING STARTED
  • 2. PLANNING YOUR RESEARCH
  • 3. PREPARING YOUR DATA

e GETTING STARTED

eTHE BASICS

e OUTCOME AND MEASURE e DETERMINE YOUR APPROACH

Are you ready to get started?

Here's what's comin

1g u1

p:

  • Provide information on the program or

practice you want to evaluate

  • Select the outcome you want to change
  • Determine how you will measure that
  • utcome
  • Use information about how the program or

practice is or wiU be implemented to identify

the right evaluation approach

  • 4. ANALYZING YOUR DATA
  • 5. SUMMARIZ ING YOUR FINDINGS

Wl hat y

1ou need:
  • Basic information on the program or practice

you're eva]uating

  • An outcome you want to change, and

information on how it will be measured

  • An understanding of

how the program or practice is or will be implemented

Institute of Education Sciences

The e2i Coach

24

slide-25
SLIDE 25

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

The e2i Coach

  • The e2i Coach takes you through a five-step evaluation process that puts

the questions you want to answer at the center. Use it to do the following:

– Investigate the implementation or impact of programs and practices you’re using – Craft your own research questions – Define your own metrics for success – Create research teams within your school or district – Share your insights with and learn from a community of educators

  • Check with the appropriate staff in your district about data use

restrictions.

– The e2i Coach does not store your data. – You should not upload personally identifiable information to the e2i Coach.

25

slide-26
SLIDE 26
  • 111'-•~~~~lllll!!!!!!!!!! ..
111111!!!!!!!!!
  • I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Evidence of effects

Participating in synchronous content Participating in asynchronous content Similar Not similar

26

slide-27
SLIDE 27

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Example research questions

  • All outcomes

 Does app A increase [fill in the blank] compared with app B?

  • Student performance

 Does assigning four hours of content per day increase assessment performance compared with assigning two hours of content per day?

  • Student engagement

 Does having one check-in per day increase students’ engagement compared with having two check-ins per week?  Does delivering content synchronously increase students’ engagement compared with delivering content asynchronously?  Do letter grade policies increase students’ engagement compared with pass/fail policies?  Does using a mobile-friendly app increase students’ engagement compared with using a non- mobile-friendly apps?

27

slide-28
SLIDE 28

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Summarizing your findings

There is a 78 percent probability that synchronous content increased assessment scores by 10 points compared with asynchronous content.

100 80 20 40 60

Probability

Outcomes Outcomes did not Outcomes meaningfully meaningfully change meaningfully decreased increased

28

slide-29
SLIDE 29

A

18

C

D

E F

I E

S ~I~

Institute of

  • -

"1tl~ Education Sciences

25

50

75

100

Ouffico es me nin1gfully decreased

Outcome did noffi

meaningfully change

Outcomes me nin1gfully

increased

Keeping equity at the forefront

29

slide-30
SLIDE 30

PCM m.ath achievement

P

11lot of MathU View Evaluation

Start a new evaluation or view all of your existing evaluations below.

ADD NEW EVALUATION V, IEW MY EVALUATIONS >

My Teams

B,

uilding evidence often requires a team

.. If

you are working with others, you can create a team for collaborating on evaluations

.. Add team

members nowi or once you get sta:rted on a new evaluation.

PllOT OF MATHU

e2icoach.

t ..

Gener, all U ... , 3/26/2020 .. ,, .

Community Findings Briefs

~

See findings from other e2i Coach users who have completed and shared their findings briefs.

VIEW Al l B,RIE~S

New User NlJ

Help us improve y 1our e2i Coach

..

1exp1er1ence

By lea:rning .

more about you and how you are

usi11g the e2i Coach, we can conti11ue to improve

this platform and make sure you are able to get the answers you need.

Share fieedback on the e2i Coach Add an feedback ou a e o 1 tile Coach

Institute of Education Sciences

30

slide-31
SLIDE 31

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

When to use the e2i Coach

You are here You will be here Prospective Retrospective

Visit e2icoach.org

31

slide-32
SLIDE 32

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

What you need to get started

32

slide-33
SLIDE 33

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Data requirements

  • Who used which tools or strategies
  • Previous performance or engagement
  • Characteristics likely to influence outcomes
  • Student group indicators
  • Students’ outcomes

33

slide-34
SLIDE 34

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Treatment group

0 0 0

nnn

Comparison group

A word of caution

  • If you are comparing two different

technologies, your outcome must be consistent across both technologies.

  • Avoid making comparisons that have only
  • ne student, classroom, or school in the

treatment group and one student, classroom,

  • r school in the comparison group.

34

slide-35
SLIDE 35

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Summary of key takeaways

  • We need new evidence on which remote learning

strategies are working to supplement the existing evidence base.

  • Technology provides many opportunities to

regularly collect detailed feedback on students’ progress and engagement.

  • A matched comparison approach will give us the

most confidence that one strategy is preferable to another.

  • Using a matched comparison approach is easy with

the e2i Coach.

  • You don’t have to do these analyses now, but you do

need to collect the data now.

One of our failures as a field—education researchers—is that we don't learn from failures…What I don't want to do is let all these natural experiments, these naturally

  • ccurring events that create these radical

transformations … just to come and go without trying to capture information about what worked, what didn't work, why. –Mark Schneider, director of the Institute

  • f Education Sciences

35

slide-36
SLIDE 36

s

(El

D El El

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Action steps

Collect good data on what remote learning strategies are being used, including which ed tech products are used as well as the pedagogy around those products. Talk to your ed tech vendors to make sure you’ll be able to access data for your students and how you’ll gain access. Appraise the types of outcome data that will be available to you and develop methods of collecting any additional data that might be needed.

36

slide-37
SLIDE 37

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

37

slide-38
SLIDE 38

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Closing

38

slide-39
SLIDE 39

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Thank you!

39

slide-40
SLIDE 40

I

ES ·

.. n E

L Regional Educational

  • MENU

.•

~

Laboratory Program

COVID-19: Evidence-Based Resources

The U.S. Department of Education's Institute of Education Sciences funds a network of 10 Regional Educational Laboratories (RELs). Each REL serves a designated region of the country and works with educators and policymakers to support a more evidence- based education system. In response to COVID-19, the RELs have collaborated to produce this series of evidence-based resources and guidance about teaching and learning in a remote environment, as well as other considerations brought by the pandemic.

UPCOMING WEBINARS

Ill

Supporting Educators in Implementing Reading Across the Content Areas This REL Pacific webinar will provide an overview of research on strategies to support educators in impl! ementing reading across the content areas for all students, including special populations such as English learners and students with disabilities.

Institute of Education Sciences

SPOTLIGH- PRACTICE~

This 30-minute teachers, instn highlights strat students in a v

  • 821 Coach

e

Mathematica solution for progress

Where can I find more resources?

Visit e2icoach.org https://ies.ed.gov/ncee/edlabs/projects/covid-19/ https://compcenternetwork.org/covid-19

40

slide-41
SLIDE 41

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

Contact us if you have any questions

  • Mikia Manley

– MManley@mathematica-mpr.com

  • Brian Gill

– BGill@mathematica-mpr.com

41

slide-42
SLIDE 42

I E

S ~I~ Institute of

  • -

"1tl~ Education Sciences

References

Common Sense Media. (2020). How teens are coping and connecting in the time of the coronavirus. Common Sense Media. https://www.commonsensemedia.org/sites/default/files/uploads/pdfs/2020_surveymonkey-key-findings-toplines-teens-and-coronavirus.pdf

  • EdWeek. (2020). Map: Coronavirus and school closures. EdWeek. https://www.edweek.org/ew/section/multimedia/map-coronavirus-and-school-

closures.html Hess, R. (2020). Chiefs for Change COO weighs in on challenges of coronavirus. EdWeek. https://blogs.edweek.org/edweek/rick_hess_straight_up/2020/04/straight_up_on_covid- 19_chiefs_for_change_exec_on_how_districts_are_responding_to_the_crisis.html Kuhfeld, M., & Tarasawa, B. (2020). The COVID-19 slide: What summer learning loss can tell us about the potential impact of school closures

  • n student academic achievement. NWEA. https://www.nwea.org/content/uploads/2020/04/Collaborative-Brief_Covid19-Slide-APR20.pdf

National Forum on Education Statistics. (2015). Forum guide to elementary/secondary virtual education data. (NFES 2016-095). U.S. Department of Education, National Center for Education Statistics. https://eric.ed.gov/?q=source%3A%22National+Forum+on+Education+Statistics%22&id=ED565838 Sawchuk, S. (2020). Grading students during the coronavirus crisis: what's the right call? EdWeek. https://www.edweek.org/ew/articles/2020/04/01/grading-students-during-the-coronavirus-crisis-whats.html

42

slide-43
SLIDE 43
  • MID-ATLANTl,

C

Regional Educational Laboratory

at Mmhematica

I E

S ~I~

Institute of

  • -

"1tl~ Education Sciences

Disclaimer

This work was funded by the U.S. Department of Education’s Institute of Education Sciences (IES) under contract ED-IES-17-C-0006, with REL Mid-Atlantic, administered by Mathematica. The content of the presentation does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. https://ies.ed.gov/ncee/edlabs/regions/midatlantic/

43