the robotics design lab Peer- Review Easily with Confidence : A Look - - PDF document

1/09/2017 1

Peer-Review Easily with “Confidence”: A Look at Replication and Randomization as tools for Statistically-Significant Assessment via Platypus

the robotics design lab

UQ Robotics 2

1/09/2017 2

UQ Robotics 3

What is Platypus?

Many Things Put Together!

• (Yet another) Electronic Submission System
• A Question-based Online Peer Assessment Tool
• An “Anonymous” Review + Student Progress Tool
• A Tool With a Nuanced (Data-centric) View of Assessment

UQ Robotics 4

1/09/2017 3

Question-Based Peer Review: Workflow Model

A(1) - Q(1) A(1) - Q(2) A(1) – Q(N)

. . .

Assignment 1 A(2) - Q(1) A(2) - Q(2) A(2) – Q(N)

. . .

Assignment 2 A(M) - Q(1) A(M) - Q(2) A(M) – Q(N)

. . .

Assignment M

. . .

• I. Collect Assignments (one per student)
• II. Randomly Shuffle Questions between assignments to create kM “Papers”

(Where k is the peer review factor, or the number of papers a student needs to review, eg 3)

Paper 1 A(1) - Q(1) A(2) - Q(2) A(M) – Q(N) Paper kM

. . .

A(1) - Q(2) A(2) – Q(N) A(M) - Q(1)

• III. Each student grades k papers (may be assigned randomly or via an alternative metric)

UQ Robotics 5

Why a Question-Based Shuffle?

• Replication & Randomization
• Assessment is an Observer

∴ We would like N(0, σ)

• Less Numbers Needed for Analysis

– A Project Class is not Super Large

• Summative Assessment by parts
• Faster feedback

Paper 1 A(1) - Q(1) A(2) - Q(2) A(M) – Q(N) Paper kM A(1) - Q(2) A(2) – Q(N) A(M) - Q(1)

UQ Robotics 6

1/09/2017 4

Platypus

UQ Robotics 7

Platypus

UQ Robotics 8

1/09/2017 5

It’s Open…

UQ Robotics 9

SQL + Laravel

UQ Robotics 10 10

1/09/2017 6

SQL: Sometimes it’s handy

SELECT users.`first_name`, users.`last_name`, users.`student_id`, users.`id` FROM ((users INNER JOIN answers ON users.`id` = answers.`user_id`) INNER JOIN text_blocks ON text_blocks.`id` = answers.`answer_text_id`) WHERE text_blocks.text LIKE '%fbcdn.net%'

UQ Robotics 11 11

Benefit: Focus on Learning Outcomes

UQ Robotics 12 12

1/09/2017 7

Peer Feedback Research

UQ Robotics 13 13

Question-Based Peer Review: Where it Works & Doesn’t

UQ Robotics 14 14

1/09/2017 8

“Statistics”: Tools to “Assess” Learning?

UQ Robotics 15 15

Basic Counts…

• Scores Histograms…
• Grade Correlations…

UQ Robotics 16 16

1/09/2017 9

Interesting Finds: Dunning-Kruger Effect?

• Correlation Between Students' Actual & Estimated

& Final Grades

UQ Robotics 17 17

Trustworthiness of Peer & Tutor Marking

UQ Robotics 18 18

1/09/2017 10

• Total Student Feedback Output Characters vs.

Total Student Marks Received

Longer answers better?

UQ Robotics 19 19

• Tutor Feedback Length and Final Course Grade

Longer feedback better?

UQ Robotics 20 20

1/09/2017 11

Positive Δ : Positive Outcome?

UQ Robotics 21 21

Worldwide Access

UQ Robotics 22 22

1/09/2017 12

Future Work

• Richer Inputs

– [Matlab] Figures – Simple Diagrams/Sketches  Rich media in general

• Alternate Question Types

– Lab Reports, etc. User-Defined?

• Team Submissions/Review
• Visualization Tools

F(x)

UQ Robotics 23 23

Will it Replace the Teacher?

Today we piloted Gradescope 's AI-assisted grading. So here we are, CS 188 all done with 650 18-page exams after just three hours of grading! AI graded AI 

UQ Robotics 24 24

1/09/2017 13

Platypus: Available For Your Class Too

🌐 http://openplatypus.org/

UQ Robotics 25 25

Students will be students

UQ Robotics 26 26

1/09/2017 14

Collaboration: More than Just for Robots

UQ Robotics 27 27

The Goal: Create Change 

UQ Robotics 28 28

1/09/2017 15

Does Your Job Make the World a Better Place?

Source: A. McCann, BusinessWeek, 8/8/13

UQ Robotics 29 29

For More Information:

🌐 🌐 http://openplatypus.org/

UQ Robotics 30 30

1/09/2017 16

UQ Robotics: Dynamic Systems in Motion

Planning Under Uncertainty & Decision-Making Mechanics

• f Motion

Aerial Robotics Systems Hanna Kurniwati (NUS/MIT) Paul Pounds (ANU/Yale) Surya Singh (Stanford/Syd)

Diverse international research group

UQ Robotics 31 31 UQ Robotics 32 32