Participants STLF: project proposal, coordination etc. Build - - PowerPoint PPT Presentation

Can students taking DE or f2f versions

• f the same course have

equally effective / enjoyable learning experiences? ~

UBC FoS Supper Series, Nov 17, 2015

Francis Jones, with Louise Longridge

Dep’t Earth, Ocean & Atmospheric Sciences

*This slide-set licensed under Creative Commons, attribution non-commercial share-alike. Contact: Francis Jones, Science Teaching and Learning Fellow, EOAS, UBC, fjones@eos.ubc.ca

To get started ….

Grab desserts, etc…

• 1. First, a framework.
• 2. Compare challenges & opportunities in DE vs f2f.
• 3. Project context
• 4. Specific initiatives
• 5. One f2f  DE conversion example.

– Project process

• 6. Evidence of change:

1. Student products 2. Quantitative / Qualitative feedback 3. DE tool usage: groups, forums, online “hit‐rates”, workloads, etc.

• 7. Discussion

Participants

• STLF: project proposal, coordination etc.

– Build resources (activities, images, video, etc.) & pedagogy – Deploy to CONNECT (or alternative…) – Evaluation; data wrangling, analytics …

• Dr. L. Longridge: Lead DE instructor

– Taking the “risks” of deploying for fully DE course. – Fitting new tasks into existing course structure. – Handling all feedback and communication with students.

• Dr. S. Sutherland: f2f Instructor (on sabbatical)

– Configured original 50‐min. hands‐on lab experience – 50‐min group‐based whole‐class follow‐up with homework

• Dr. P. Smith: Original design of the exercise

– For 2nd year geoscience majors – Still used as a 2‐hr laboratory exercise with reporting.

Distance education vs face to face – DE vs f2f

• How many teach – or have taught – on line?
• How many are thinking of teaching a DE course?
• How many are involved with a course that has BOTH

DE and f2f versions?

• What differences between teaching / learning f2f and online?

First, frameworks …

Traditional …

• Content
• Instruction
• Practice
• Assessment

RBIS …

• Motivation
• Practice that is deliberate
• Solo and “social” learning
• Timely interactive feedback

Teaching = enabling RBIS by facilitating interactions:

• Student ↔ Content .
• Student ↔ Student .
• Student ↔ Expert

(Instructor / TA)

RBIS = Research Based Instructional Strategies

What are a few opportunities / challenges?

Face to face Distance education

(Restrict thinking to asynchronous)

Student . ↔ Content Student . ↔ Student Student . ↔ Expert

Instructor or TA

• pportunities

challenges

Compare: opportunities / challenges EOAS flexible learning project and courses

• Courses
• Specific DE to f2f translation project component

EOAS Flexible Learning project, 2014‐16

• EOSC 326, Earth and Life Through Time

– 3rd year elective for science students only. – Experiment with ideas in:

Active content; “Labs”; Small group work.

• EOSC 116, The Mesozoic Earth

– 1st year elective open to all students. – Secondary focus.

• EOSC 118, Earth's Treasures: Gold & Gems

– Later, apply “best” ideas from others. – Add a virtual museum activity.

All service courses, not core. ~ 450 1 f2f 3 DE ~ 350 1 f2f 3 DE ~ 600 3 DE Enrollments Sections

Stuff students did in 326‐DE

Originally ( ~ 2005)

• Readings  module tests; largely multiple choice (MC).
• 2 “labs”

– review content and resources, – answer MC questions.

• 2 Discussion board tasks:

– intro; – short essay + 1 response.

• Discussion board open forums for questions.

Introduced since 2014 Student  Content *

1. “Interactive” readings: instant feedback on questions.

• Tasks and questions embedded in basic content.
• Instant feedback; not necessarily ‘graded’ … but “instant”.

2. Interactive figures using image maps and JavaScript. 3. Labs: generate & share sketches and annotated figures. 4. Several low‐stakes, post‐activity “quizzing” opportunities

• MC, ranking, fill‐blank, matching, jumbled sentence, numerical, etc.
• “Blooms Dichotomous Key”; check q’n sophistication & set targets.
• Higher stakes testing familiar tasks and question types.

* Eg. Clark and Mayer, 2011

Introduced since 2014 Student  Student

• 1. Cooperative versus Collaborative1:

distribute work & ‘agree’ versus generate a whole bigger than the parts

• 2. Cooperative opportunities

– Semi‐structured discussion (“introduce yourselves and chat”) – Share results of solo work in groups – Generate group versions of: quizzes (eg. 2‐stage tests) or Cooperative products (eg. sketched problem solutions)

• 3. Collaborative opportunities

Not achieved this time around – but plans are afoot …  – Construction of knowledge and/or products (eg museum displays) – More autonomous than prescribed cooperative exercises – Blogs, journals, wikis, Google Docs, Google Earth ;

1 Cooperative vs collaborative: see eg. Panitz. 1999

Introduced since 2014 Student  Instructor

Expert  novice interaction is important and “precious”

• 1. Design / facilitate semi‐structured discussions.
• 2. Rubrics and exemplars
• 3. Feedback on intermediate work (may be automated)
• 4. Feedback on final work;

– Collected feedback about all student work; – Personalized by referring to collected items.

• 5. Implement – and act upon – student feedback

Outline ….

• 1. First, a framework.
• 2. Compare challenges & opportunities in DE vs f2f.
• 3. Project context
• 4. Specific initiatives
• 5. One f2f  DE conversion example.

– Project process

• 6. Evidence of change:

1. Student products 2. Quantitative / Qualitative feedback 3. DE tool usage: groups, forums, online “hit‐rates”, workloads, etc.

• 7. Discussion

Before progressing … the ‘skinny’

• 1. These supper‐series events always evolve

(devolve?) into great discussions, so …

• 2. Partial “conclusions” so far ‐ since project

evaluation steps are in progress. Can students taking DE or f2f versions of the same course have equally effective / enjoyable learning experiences? Yes, but with different types of instructional effort. We are encouraged by …

1. Interactive resources can be constructed WITHOUT particularly special skills. 2. Engaging tasks can be developed with care and attention to purpose and pedagogic detail. 3. Asynchronous small group interactions work with careful scaffolding.

We still need to do better at

1. Closing the feedback loop VISIBLY and productively 2. Assessments: a) align with tasks/activities & b) increase variety 3. Shift learning goals off “knowledge”  towards “skills”. 4. Incorporating analytics: Assessments & resource use or online behaviour.

Project progress so far …

Successes

• Engaging, effective learning tasks and resources can be facilitated

– We are still refining details. – Some simple 3rd party facilities are needed.

• Learning tasks first – resources second. Not vice‐versa.
• Learning goals tend to evolve “organically”.

Project Evaluation

• Analytics data are hard to get, often requiring processing of raw data.
• Tests evolve with innovations, hence comparing before‐after change is hard.
• Assessment sophistication can be gauged with Blooms Dichotomous Key.

Project progress so far (con’t.)

Limited to date

• Are there more RBIS evident? Interactions model can help.
• Are student  instructor (novice‐expert) more effective? Still a

challenge.

• Incorporate feedback and analytics to help students directly

(motivation, reflective practice & metacognition, etc.)

So far … mostly f2f  DE. Were any DE  f2f conversions effective?

• eosc116 homework activities
• Online resources can become available
• New museum activity: Virtual for DE118 first (January 2016), f2f after.
• However, based on consistency of feedback and analytics, we may be

“making” too many resources and not “assessing” / adjusting enough.

Implications

For instructors

• DE pedagogy is different. Experience with f2f is not enough. Example:

experience is needed developing questions in M.C. and other formats.

• Experience with Connect is vital – or close knowledgeable support.
• Awareness and moderate skill with web technology and resources is vital

for development, less so (but still important) for instructors themselves. For departments

• Need “official” points of contact for DE‐instructional support.
• Need facilities to host resources that are not Connect‐compliant.
• Converting course components DE  f2f is possible but non‐trivial.
• Can NOT improve a course with same resources used to “just” teach it.

For institution

• Analytics is necessary BOTH for improvement AND evaluation.
• Analytics for instruction has potential but needs development.
• I.T. support OUTSIDE the LMS is necessary, perhaps at Dep’t level.

data list

Outline ….

• 1. First, a framework.
• 2. Compare challenges & opportunities in DE vs f2f.
• 3. Project context
• 4. Specific initiatives
• 5. One f2f  DE conversion example.

– Project process

• 6. Evidence of change:

1. Student products 2. Quantitative / Qualitative feedback 3. DE tool usage: groups, forums, online “hit‐rates”, workloads, etc.

• 7. Discussion

Example f2f activity / Lab:

See video examples showing lab and classrooms in action

• One 50 minute lab examining fossil and rock samples

with TAs and instructor present.

http://blogs.ubc.ca/wpvc/watch‐look‐for/the‐laboratory‐experience/

• One 50‐minute structured group activity follow up a

week later.

http://blogs.ubc.ca/wpvc/watch‐look‐for/the‐subsequent‐whole‐class‐follow‐up‐activity/

• Some online quiz‐like homework.

Hands‐on f2f components

Lab time Handle specimens Follow-up group-work in class with paper worksheets

For DE – can we develop …

• Same learning goals ?
• Similar experiences ?
• Online data entry (after paper work) ?
• Online sketching and upload of results ?
• Online digitized resources emulating specimens ?

Virtual components for DE

Zoom-in high resolution lab space with clickable Hotspots. Hi-res specimen images + videos of “handling”

http://eos.ubc.ca/courses/eosc326/content/trilograpto‐lab/ ID, PW in notes.

Components for BOTH f2f and DE

Same specimens Same tasks (including sketching) Same goals. Same documents.

http://eos.ubc.ca/courses/eosc326/content/trilograpto‐lab/Sketching/SketchExercise.html ID, PW in notes

DE and f2f differences:

Week 1, F2F lab:

• 1. Manual / instructions
• 2. On paper: 21 fossil IDs and ages
• 3. Hands on:

‐ real specimens ‐ photos ‐ 1 hr in lab with instructor & TAs

• 4. Online questions about fossils

‐ all multiple choice.

• 5. Sketching on paper.

‐ Graded by TAs

Phase 1, DE “lab”:

• 1. Same …. add a scenario
• 2. 17 of 21 fossils, 3 exemplars

‐ Digital input & autograding of IDs / ages

• 3. Digitized specimens

‐ Interactive lab environment ‐ Images: high resol’n, zooming, multi‐view ‐ Videos: of handling specimens

• 4. Online q’ns (not all MC) about fossils to

address aspects of the scenario.

• 5. Digital sketch: annotate given figures.

‐ Sketch submission only graded by TAs.

DE and f2f differences:

Week 2, F2F lab:

• 1. Groups: Agree upon / re‐submit fossil

ID and ages.

• 2. Groups: answers to 2 point‐form

written questions.

• 3. Groups: Agree upon and re‐submit

sketched interpretation.

• 4. Graded by TAs.
• 5. Solution set: PDF provided online.

Phase 2, DE “lab” – Add team work: Going “live” early June.

• 1. Not done.
• 2. Questions done solo only.

Asynchronous work in small groups (5‐7)

• 3. Small groups: share, then agree upon and

re‐submit sketched interpretation.

• 4. Sketches graded by TAs
• 5. Solutions after grading.

NOTES: ‐ Groups are permanent. ‐ This is the 3rd of 4 small group tasks.

Questions? Comments?

• Framework

– Facilitate RBIS by addressing interactions. – S <‐> C S <‐> S S <‐> E

• Project context

– Initially: 2 service courses, 1 DE instructor, 1 f2f instructor. – Subsequently: 3rd DE service + 2nd DE instructor, and others.

• Specific activity conversion

– A hands‐on / classroom exercise for asynchronous DE.

Outline ….

• 1. First, a framework.
• 2. Compare challenges & opportunities in DE vs f2f.
• 3. Project context
• 4. Specific initiatives
• 5. One f2f  DE conversion example.

– Project process

• 6. Evidence of change:

1. Student products 2. Quantitative / Qualitative feedback 3. DE tool usage: groups, forums, online “hit‐rates”, workloads, etc.

Actions versus evaluation: An evolving project process …

• Evaluation details evolve based on

success (or not!)

• Ideally, learning goals are the primary

driving force

– BUT the process is not linear and takes iteration – Why? Because it is not initially obvious which new ideas will be practical and what won’t. – Requires an instructor who can confidently handle glitches. Implement Plan Experiment Evaluate Adjust

Evidence of change …

• We are 19 mths into 24mth project; Collecting / assessing data is ongoing.
• DE courses produce a richer data‐trail than f2f.

– BUT … accessing data is challenging (i.e. Connect is not helpful)

Evidence of …

• Student interactions.

– Content, Colleagues, Instructor / TAs

• Online and group‐work behaviors.
• Decisions, interpretations and “products” that students produce.
• Requested feedback.

Evaluation options – i.e. possible comparisons:

– Compare DE and f2f activity, engagement & outcomes – Compare prior‐ to current‐ interactions – Compare prior‐ to current‐ outcomes

Data sets Data contexts

• Tests, quizzes and checkup on activities.

– Analytics workflows are under development.

• Feedback results

– Added as a small part of activities or tests. – Active readings feedback – Whole course workloads and enthusiasm

• Hits or time in segments of CONNECT

– Course activity / Forums / Groups / single student – “Reports” and download formats are awkward

• Group work results

– Activity reports and thread listings (some manual analysis is possible)

• Image work results

– Annotated figures on uniform base‐images

• Quizzes, midterms, exams
• Lab exercises

– Our focus today – Includes sketching results

• Small group activities

– Introduction – Lab 1 – Lab2 (feedback) – Short essay and response

• Online “activity”

– Content – Hours – Hits on the course site

Questions? Suggestions? Or discuss & wrap-up

Exam / test results.

Not really ready yet; Why?

• Exams evolve with changes to learning goals and activities.
• Labour intensive since Connect is “stupid” about analytics.
• But – we can analyze for test sophistication.

Eg:

Compare earlier f2f and DE exams; “Blooms Dichotomous Key”

0% 10% 20% 30% 40% 50% 60% 70% 80%

Proportion of questions

DE f2f

Blooms Level, two final exams

data list

Data about the lab exercise

• Compare DE to original f2f exercise
• Sketches (solo and group)
• Discussion activity in groups
• Feedback about resources and workloads

Note: More here than we can cover, but we can pick and choose 

Solo and group results

• Solo tasks:

– Interpret 16 fossils from 4 “depths” at 3 locations, + 1 other unknown. – Determine ages from references. – Sketch lines bounding ages on geologic sections.

• Groups task:

– Agree on age boundaries.

data list

Lab 2 sketch result: eg. from one group

• Solo work varies in correctness and completeness
• Group work iterates towards correct & complete interpretation

data list

Lab 2 sketch result: eg. from one group

• Solo work varies in correctness and completeness
• Group work iterates towards correct & complete interpretation

Group version after disc’n Solution key 4 individual versions – done first

0% 10% 20% 30% 40% 50% 6 5 4 3 2 1 Prop'n of sketches with N errors N ‐ number of errors

Proportion of solo & group sketches with "N" total errors

solo (n=39) group (11)

Lab 2 sketching: solo and grp error rates

• Based on rubric components.
• Groups “better” than solo, but … could scaffolding be improved?

data list 0% 10% 20% 30% 40% 50% 6 5 4 3 2 1 Prop'n of sketches with N errors N ‐ number of errors

Proportion of solo & group sketches with "N" total errors

solo (n=39, avg=2.3) group (n=11, avg=2.9)

1 2 3 4 5 6 7 8 9 10 3 2.75 2.5 2.25 2 1.75 1.5 1.25 1 0.75 0.5 0.25

Solo Lab2 sketch score distrib'n

Lab 2 sketching: solo and discussion results

• Who posts … How often?
• 12 groups
• Wide range of active‐ness
• 2nd to post seems most active
• Very slight relation between

score and group post‐counts.

• Evidently, some room to improve

scaffolding and motivation.

2 4 6 8 10 12 14 16 8 7 6 5 4 3 2 1

• No. individuals with "n" posts each
• No. posts per person
• No. individuals with between 0 and 8 posts each.

Total no. students = 62 N contributors 1st 2nd 3rd 4th 5th 6th totals avg/pe Grp01 5 4 7 3 1 2 NA 13 3.3 Grp02 5 4 4 5 3 1 NA 13 3.3 Grp03 5 3 6 5 2 NA 13 4.3 Grp04 5 4 5 4 5 3 NA 17 4.3 Grp05 6 4 4 5 2 2 13 3.3 Grp06 5 2 1 1 NA 2 1.0 Grp07 5 3 2 6 3 NA 11 3.7 Grp08 5 5 7 7 4 2 2 NA 22 4.4 Grp09 5 4 1 6 6 4 NA 17 4.3 Grp10 5 4 7 7 6 4 NA 24 6.0 Grp11 6 6 4 8 1 3 2 2 20 3.3 Grp12 5 4 2 6 5 7 NA 20 5.0 totals 62 47 50 63 38 28 4 2 avg posts 4.2 5.3 3.2 2.3 0.3 3.8

data list

Lab 2 feedback

• “More of these would be great” …
• ‘14‐’15 diffs in mean not significant.
• “How many hours to complete solo part?”

– Very slight shift to more time. – Not significant, despite changes due to feedback.

0% 5% 10% 15% 20% 25% 30%

proportion of respondents

Self‐report time to complete

14wc (m=4.8 ±2.3) 15s (m=5.6 ±3.7) T.test: P=0.15

0% 10% 20% 30% 40% Proportion of respondents

More of these would be good

14wc 15s

data list

Lab 2 feedback

• Which resources were Most / Least useful?

– Video was “least” by smaller proportion of students in 15s. – Otherwise, similar.

14wc 15s 14wc 15s video 13% ‐65% zooming 72% ‐4% fixed 13% ‐17% multiple 2% ‐15% most least 14wc 15s 14wc 15s video 13% 12% ‐65% ‐44% zooming 72% 73% ‐4% 0% fixed 13% 12% ‐17% ‐38% multiple 2% 4% ‐15% ‐19% most least

0% 15% 30% 45% 60% 75% 90% yes no

• ther

Proportion of respondents

Did you use resources other than given? 15s 14wc

• Did you use other sources?

Which were most useful?

• More details next slide

Feedback from T/G lab 1st and 2nd iterations

Outside resources used?

0% 20% 40% 60% 80%

Yes No 2014wc: Did you use outside resources? (n=103)

No unspecified google wikipedia

• ther

disc'n 0% 20% 40% 60% 80%

Yes No 2015s: Did you use outside resources? (n=50)

No unspecified google wikipedia

• ther

disc'n

2nd iteration after some changes to questions and guidelines.

(Could apply χ2‐test, but ‘14 and ‘15 do seem quite different).

Feedback from T/G lab 1st and 2nd iterations

• Did you like the sketching app? Dislike it? Any suggestions?

0% 10% 20% 30% 40% 50%

too long suggestion confusing negative positive or OK

2014wc: Reaction to sketching app.

Other comments

2015s &

data list

Feedback from T/G lab 1st and 2nd iterations

• Any suggestions?

0% 10% 20% 30% 40%

• ther

have groups sketching app (negative) provide more practice time ‐ too long improve resources instruct'ns, concise, clarity none or positive

Any other suggestions

2014wc (105) 2015s (55)

data list

Feedback from T/G lab 1st and 2nd iterations

Any suggestions?

• Like it, Lab 2 sketching felt unnecessary but lab 1 sketching was really

helpful (especially when we compared with a group).

• I think activities like these are extremely useful and fun when you can

actually go into a lab with other students, but I found it difficult to make myself sit down and go through it on the computer; it isn't as fun or exciting seeing everything behind a screen and a lot of the time the interaction with other students is needed in order to figure things out.

• I liked this portion of the lab and really helped me with my understanding
• f the geological time columns and also helped me confirm my answers in

worksheet 2

• From First Version: It might be nice to somehow incorporate group work

into the activity (since in‐class labs often benefit from team work).

data list

Conclusions:

• Compare student feedback 2014 winter and 2015 summer
• 2014w = first attempt and NO group work
• 2015s = second attempt WITH group work
• Fall 2015 runs Nov 13th – 25th.
• Feedback suggests students reacted similarly.
• Conclusion – we need to pay closer attention to collective

feedback and address common issues and recommendations.

data list

General small groups data

• Compare prior to current DE course
• Recall – 4 small group activities

1. Introduction 2. Lab 1 3. Lab 2 4. Short essay and response

0% 5% 10% 15% 20% 25% 30% 800 750 700 650 600 550 500 450 400 350 300 250 200 150 100 50

Proportion of enrolled students Total # hits within discussion groups

Proportion of enrolled students versus total # hits in groups

2014wc

Hits within all groups

• Using Connect’s “report” data.

0% 5% 10% 15% 20% 25% 30% 800 750 700 650 600 550 500 450 400 350 300 250 200 150 100 50

Proportion of enrolled students Total # hits within discussion groups

Proportion of enrolled students versus total # hits in groups

2015sa 2014wc

data list

5 10 15 20 25 30 1 4 7 10 13 16 19 22 25 28 31 3 6 9 12 15 18 21 24 27 2 5 8 11 14 17 20 23 26 29 1 4 7 10 13 16 19 22 25 28

Average no hits per student

Day of the month during term.

Avg hits in groups per student on each day of term.

2014wc (N=133)

Hits within groups each day per student

• Interaction with colleagues is distributed across the term.

5 10 15 20 25 30 1 4 7 10 13 16 19 22 25 28 31 3 6 9 12 15 18 21 24 27 2 5 8 11 14 17 20 23 26 29 1 4 7 10 13 16 19 22 25 28

Average no hits per student

Day of the month during term.

Avg hits in groups per student on each day of term.

2014wc (N=133) 2015s (N=64)

data list

0% 10% 20% 30% 40% 50% 60% 51 ‐ 90 41 ‐ 50 31 ‐ 40 21 ‐ 30 11 ‐ 20 6 ‐ 10 3 ‐ 5 < 3 Proportion of all students per bin Bins: total number of disc'n board posts

Proportion of students binned by total number of discussion board posts

2014wa 2014wc 20015sa

Total discussion board posts

• Quantity of interaction with colleagues is increased.
• What about Quality???

0% 10% 20% 30% 40% 50% 60% 51 ‐ 90 41 ‐ 50 31 ‐ 40 21 ‐ 30 11 ‐ 20 6 ‐ 10 3 ‐ 5 < 3 Proportion of all students per bin Bins: total number of disc'n board posts

Proportion of students binned by total number of discussion board posts

2014wa 2014wc 20015sa

data list

How active are small groups?

• This is quantity not quality, but group behavior seems to vary.
• Room for improved scaffolding and motivation.

500 1000 1500 2000 2500 grp 11 (6) grp 2 (5) grp 5 (6) grp 1 (5) grp 8 (5) grp 12 (5) grp 10 (5) grp 7 (5) grp 9 (5) grp 3 (5) grp 4 (5) grp 6 (5)

Total # hits within each group

data list

Short essay + discussions: before/after small grps

2014wc and before

• Grps of 40‐50 for this assign. only
• Five forums (topics) per group.
• Short essay: post under 1 of 5

topics (ie. forums).

• Respond to one other.

2 required posts ~ 1 “thread” / student ~ 3 posts / thread 2015s and after

• Same small grps for 4 activities.
• One forum (per grp) for this assig.
• Short essay: post as a new thread,

any 1 of 5 topics.

• Reply to 3 others.
• Respond to 1 reply (i.e. “discuss”).
• Discuss further for bonus pts.

5 required posts ~ 1 “thread” / student ~ 7 posts / thread

data list

Activity in forums

Small groups introduced 2015s. ‐ Reduced unstructured engagement about content ‐ Increased structured engagement in 4 small‐group activities.

20 40 60 80 100 120

Course Grades Final Exam General/Open Mic How do I... Midterm Exam A Midterm Exam B Science in the News Study Groups Module A Module B Module C Module D Module E Introduce Yourselves Graded Discussion Activity 1: Fossil Identification and… Lab 1: Relative Dating Using Rocks… Lab 1: Part 2 (Group Portion) Lab 2: Trilobites/Graptolites Lab 2: Part 2 (Group Portion) Activity 2: Coast Fossils

Average hits per student in Forums

2014wa 2014wc 2015sa 20 40 60 80 100 120

Course Grades Final Exam General/Open Mic How do I... Midterm Exam A Midterm Exam B Science in the News Study Groups Module A Module B Module C Module D Module E Introduce Yourselves Graded Discussion Activity 1: Fossil Identification and… Lab 1: Relative Dating Using Rocks… Lab 1: Part 2 (Group Portion) Lab 2: Trilobites/Graptolites Lab 2: Part 2 (Group Portion) Activity 2: Coast Fossils

Average hits per student in Forums

2014wa 2014wc 2015sa

Structured Un‐structured

data list

Total activity: hours and “hits”

• Are students spending more time?

– DE may be able to explore this question more easily than f2f.

• Compare prior to current DE course

Total hours online

• Total hours is not significantly (Pr=0.206) different in three terms.
• However box plots and stdev do suggest increased variability.

Summary :

ANOVA result:

year N mean sd se ci 1 14wa 93 22.7 15.2 1.6 3.1 2 14wc 130 26.8 21.0 1.9 3.6 3 15s 77 28.5 30.0 3.4 6.8 Df Sum_Sq Mean_Sq F_value Pr(>F) yr 2 1569 784.5 1.59 0.206 Resid’s 297 146525 493.4

data list

10 20 30 40 50 60 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113

hits per person Days from first day of first month

Average total hits to Connect per day per person

2014wc (avg 11.0 hits / day /pers) 2015sa (avg 10.9 hits / day /pers)

Total hits in Connect each day per student

• Total hits did not change after adjusting to 4 small group tasks.
• However, engaging in the course became less sporadic.

10 20 30 40 50 60 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113

hits per person Days from first day of first month

Average total hits to Connect per day per person

2014wc (avg 11.0 hits / day /pers) 2015sa (avg 10.9 hits / day /pers)

data list

One more idea …

• Correlate “total submissions” versus final grades.

R = 0.46 (or R2 = 0.213) – Not particularly convincing. – But consistent with other “time‐on‐task” vs “success” studies.

data list

Workloads, enthusiasm and experiences

• Compare “new” DE to prior DE and f2f.
• Uses results of SLES (Student Learning Experiences Survey)

0% 5% 10% 15% 20% 25% 30% much more little more same little less much less Proportion of course comparisons Workload and enthusiasm for EOSC 326 compared to other courses workload, n=146 enthusiasm, n=136 0% 10% 20% 30% 40% 50% 60% 70%

much more little more same little less much less

Proportion of course comparisons 326DE 2015s (‐0.39, ‐0.14 ) workload, n=103 enthusiasm, n=103

Workloads and enthusiasm (SLES)

0% 5% 10% 15% 20% 25% 30% 35% 40%

much more little more same little less much less

Proportion of course comparisons 326DE 2014s (‐0.31, ‐0.23 ) workload, n=94 enthusiasm, n=95 0% 5% 10% 15% 20% 25% 30% 35% 40%

much more little more same little less much less

Proportion of course comparisons 326DE 2014a (‐0.04, 0.33 ) workload, n=230 enthusiasm, n=230

DE: fall / winter terms largely similar. f2f: “more work”, “more enthusiasm” DE: 2014 / 2015 summer: similar overall but distinct in detail

data list

SLES for DE and f2f

• Student Learning Experiences Survey (SLES);

– 45 Likert‐scale questions in f2f courses – 55 Likert‐scale questions in DE courses

• For how many strategies do mean responses vary significantly?
• No open questions analyzed yet.

“Significant” differences between f2f and DE

Padj < 0.05 Padj < 0.01 N Between 4 DE sections 5 4 48 Between 4 DE and 1 f2f 5 12 22

data list

SLES for DE (and f2f)

• Other homework exercises

(Not quizzes; eg. problem sets, etc.) P = 0.0241

• Projects you did with other students

(written, oral, poster, etc.) P = 0.0002

No specific “group” questions, BUT – 2015 is 1st to use small groups

f2f | 2014 | 2015 f2f | 2014 | 2015

data list

 “helpfulness”   “helpfulness” 

SLES for DE (and f2f)

• The text book
• Feedback on completed work

Interesting f2f – DE comparisons:

f2f | 2014 | 2015 f2f | 2014 | 2015

data list

 “helpfulness”   “helpfulness” 

Activated readings feedback

• Early “active content”. Questions asked to recommend changes.
• Resulting adjustments improved activities in subsequent terms.
• Time on task changed little.
• “Appreciation” changed little.

0% 10% 20% 30% 40% 50% purely negative ‐ drop it etc. more time, or timing closer to modules with background readings drop or change a portion

• ther

more feedback about why right or why wrong no or none more readings or pre‐readings or explanations of info/figures provided positive only viewing or cosmetic or clarity problems or "explain how" longer and/or more sophisticated

Suggested improvements ‐ coded. Sorted by 2014wa priorities

2014wa 2014wc 2015s

Open feedback from coast fossils activity

Effect of adjustments due to feedback in term 1.

Gray – first feedback Colour – subsequent feedback ‐ Mostly “none” ‐ Term 1 priorities reduced

0% 10% 20% 30% 40% 50% purely negative ‐ drop it etc. more time, or timing closer to modules with background readings drop or change a portion

• ther

more feedback about why right or why wrong no or none more readings or pre‐readings or explanations of info/figures provided positive only viewing or cosmetic or clarity problems or "explain how" longer and/or more sophisticated

Suggested improvements ‐ coded. Sorted by 2014wa priorities

2014wa 2014wc 2015s

data list

Time on coast fossils activity

• Time on this task seems little different.
• Possibly shifted to a more uniform 1.5‐2.0 hrs. by summer 2015?

‐ But summer and fall/winter terms have many reasons to be “different”.

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% > 3.0 2.5‐3.0 2.0 ‐ 2.5 1.5 ‐ 2.0 1.0 ‐ 1.5 0.5 ‐ 1.0 < 0.5

Proportion of students in each bin

Hours spent working on this activity

2014wa 2014wc 2015s

data list

Likert‐scale feedback on coast fossils

Consistent from term to term.

0% 10% 20% 30% 40% 50% 60% 70% 80% Strongly Agree Agree Neither Disagree Strongly Disagree

helped improve abilities to relate fossils to living environment.

2014wa 2014wc 2015s 0% 10% 20% 30% 40% 50% 60% 70% 80% Strongly Agree Agree Neither Disagree Strongly Disagree

helped improve abilities to apply geologic & paleontological info.

2014wa 2014wc 2015s 0% 10% 20% 30% 40% 50% 60% 70% 80% Strongly Agree Agree Neither Disagree Strongly Disagree

helped integrate knowledge and skills from earlier modules.

2014wa 2014wc 2015s 0% 10% 20% 30% 40% 50% 60% 70% 80% Strongly Agree Agree Neither Disagree Strongly Disagree

It would be great to have more of these activities.

2014wa 2014wc 2015s

data list

Some references and resources

• http://eos.ubc.ca/about/faculty/F.Jones.html
• http://eos.ubc.ca/research/cwsei/
• http://www.eos.ubc.ca/courses/Dist‐Ed/DE326.html
• http://ctlt.ubc.ca/distance‐learning/courses/eosc/eosc326/
• Bernard, Robert M., Eugene Borokhovski, Richard F. Schmid, Rana M. Tamim, and Philip C. Abrami. 2014. “A

Meta‐Analysis of Blended Learning and Technology Use in Higher Education: From the General to the Applied.” Journal of Computing in Higher Education 26 (1): 87–122. doi:10.1007/s12528‐013‐9077‐3.

• Clark, Ruth C., and Richard E. Mayer. 2011. E‐Learning and the Science of Instruction: Proven Guidelines for

Consumers and Designers of Multimedia Learning. John Wiley & Sons.

• Kennepohl, Dietmar Karl, and Lawton Shaw. 2010. Accessible Elements Teaching Science Online and at a
• Distance. Edmonton: AU Press. http://www.doabooks.org/doab?func=fulltext&rid=14423.
• Kerton, Charles1, kerton@iastate.edu, and Cinzia2 Cervato. “Assessment in Online Learning‐‐It’s a Matter of

Time.” Journal of College Science Teaching 43, no. 4 (April 3, 2014): 20–25.

• Michaelsen, L. K., M. Sweet, and D. X. Parmelee, eds. 2009. Team‐Based Learning: Small Group Learning’s Next

Big Step: New Directions for Teaching and Learning, Number 116. San Francisco, CA: Jossey‐Bass.

• Panitz, Theodore. 1999. “Collaborative versus Cooperative Learning: A Comparison of the Two Concepts Which

Will Help Us Understand the Underlying Nature of Interactive Learning.” http://eric.ed.gov/?id=ED448443, retrieved 15/04/02.

• Pashler, H., P. Bain, B. Bottge, A. Graesser, K. Koedinger, M. McDaniel, and J. Metcalfe. n.d. “Organizing

Instruction and Study to Improve Student Learning. IES Practice Guide.” National Center for Education Research, Institute of Education Sciences.

• Singer, Susan R., Natalie R. Nielsen, and Heidi A. Schweingruber, eds. 2013. Discipline‐Based Education Research:

Understanding and Improving Learning in Undergraduate Science and Engineering. Accessed July 25. http://www.nap.edu/catalog.php?record_id=13362.