Expert Code Review and Mastery Learning in a So f ware Development - - PowerPoint PPT Presentation

▶
expert code review and mastery learning in a so f ware
SMART_READER_LITE
LIVE PREVIEW

Expert Code Review and Mastery Learning in a So f ware Development - - PowerPoint PPT Presentation

Nov 24, 2023 146 likes •513 views

Expert Code Review and Mastery Learning in a So f ware Development Course Sophie Engle Sami Rollins sjengle@cs.usfca.edu srollins@cs.usfca.edu CCSC Southwestern Region Conference Sophie Engle and Sami Rollins April 5-6, 2013, San Marcos, CA

slide-1
SLIDE 1

Sophie Engle and Sami Rollins Department of Computer Science

Expert Code Review and Mastery Learning in a Sofware Development Course

Sophie Engle sjengle@cs.usfca.edu Sami Rollins srollins@cs.usfca.edu

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-2
SLIDE 2

Sophie Engle and Sami Rollins Department of Computer Science

INTRODUC INTRODUCTION TION

Why Try Mastery Learning?

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-3
SLIDE 3

Sophie Engle and Sami Rollins Department of Computer Science

Motivation

  • Some students made it to upper-division courses,

but unable to pass those courses on first try

– Most delay graduation and retake – Many upper-division courses offered once a year

  • Ofen have weak but broad level of programming
  • Somewhere between start and end of degree, not

quite preparing these students for harder classes

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-4
SLIDE 4

Sophie Engle and Sami Rollins Department of Computer Science

Sofware Development

  • Bridges lower division programming courses and

upper division programming-heavy courses

  • Provides student project experience
  • Goal is to produce well-designed large sofware

project, approximately 2k lines of code

  • Promotes iterative development
  • Undergraduates already have two introduction to

programming courses (Python and Java)

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-5
SLIDE 5

Sophie Engle and Sami Rollins Department of Computer Science

Issues Identified

  • Possible to gain enough partial credit to pass the

course without mastering all of the core concepts

  • Easy to test for correctness, difficult to test design

– Unit tests and scripts for correctness – Code review for design

  • Assigning partial credit to code design tricky

– Teacher assistants unfamiliar with code review – Unwilling to give low grades for functional code

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-6
SLIDE 6

Sophie Engle and Sami Rollins Department of Computer Science

Issues Identified

  • Originally believed iterative projects would lead to

iterative development

– Students loath to refactor "working" code – Students not sure of issues and how to fix them

  • Only a certain core of projects were really critical

for upper division courses

– Needed multithreading, code design mastery – Did not need mastery of web-related topics

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-7
SLIDE 7

Sophie Engle and Sami Rollins Department of Computer Science

Hypothesis

  • Use homework and quizzes to address breath, and

projects for depth

  • Use mastery learning to force students to master

core concepts necessary for upper division courses

  • Use expert code review to enforce mastery

learning for code design

  • Force students to refactor code until passes both

unit tests and code review

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-8
SLIDE 8

Sophie Engle and Sami Rollins Department of Computer Science

BA BACK CKGROUND GROUND

Sofware Development Course

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-9
SLIDE 9

Sophie Engle and Sami Rollins Department of Computer Science

Course Basics

  • Semester-long course
  • Hour-long classes meeting three times a week
  • Approximately 10 to 30 students per section
  • Offered every semester
  • Mostly undergraduate majors (part of core)
  • Also included minors and new graduate students*

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*We revisit this later on in the talk.

slide-10
SLIDE 10

Sophie Engle and Sami Rollins Department of Computer Science

Environment

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

http://cs.usfca.edu/facilities.html

slide-11
SLIDE 11

Sophie Engle and Sami Rollins Department of Computer Science

Traditional Approach

  • Lecture

– Hour long twice weekly – Slide-based or live code walkthroughs

  • Lab

– Hour long once weekly – In-class homework and quizzes*

  • Exams

– One midterm and one final exam – Closed-book closed-note except Java API

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*These were added in later versions of the course.

slide-12
SLIDE 12

Sophie Engle and Sami Rollins Department of Computer Science

Traditional Approach

  • Seven large iterative programming projects

– Word Count – Inverted Index – Partial Search* – Multithreading – HTML Parsing – Web Crawler – Search Engine

  • Assigned throughout semester

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*Used to include a redesign component.

slide-13
SLIDE 13

Sophie Engle and Sami Rollins Department of Computer Science

Student Experience

  • Very popular among students

– Helpful in future courses – Helpful for finding sofware development jobs

  • Very motivated by search engine project
  • Reputation for being fairly easy to pass

– Kludge together something before deadline – Get partial credit and move on to next project

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-14
SLIDE 14

Sophie Engle and Sami Rollins Department of Computer Science

APPRO APPROACH CH

Mastery Learning and Expert Code Review

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-15
SLIDE 15

Sophie Engle and Sami Rollins Department of Computer Science

Approach

  • Traditional Approach (Breadth)

– Lectures – Homework – Quizzes – Exams

  • Mastery Learning (Depth)

– Projects

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-16
SLIDE 16

Sophie Engle and Sami Rollins Department of Computer Science

Projects

  • Reduced from 7 to 5 projects

– Keep inverted index, partial search, multi-

threading, web crawler, and search engine

– Assign word count and HTML parsing projects as

homework instead

  • Two-stage project submission

– Teacher assistant runs tests for correctness – Instructor performs code review for design

  • Must continue to refactor until both stages pass

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-17
SLIDE 17

Sophie Engle and Sami Rollins Department of Computer Science

Projects

  • Unable to start next project until current passes
  • Provide suggested deadlines to try and keep

students on track

  • Accelerated deadline schedule to promote an agile

approach and provide time for resubmission

– Students instructed to expect to submit twice – Students must also master time management

  • Cutoff deadlines given to ensure enough time for

resubmission and still pass

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-18
SLIDE 18

Sophie Engle and Sami Rollins Department of Computer Science

Project Grading

  • Projects are worth majority of grade
  • Project grade based on how many projects passed

– Must pass multithreading project to pass course – If perform poorly on exams, must also pass web

crawler to pass the course

  • Small penalty deducted if students resubmit

project too many times*

  • Small extra credit added if students submit by

suggested deadlines*

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*Again, this reflects how it is done currently. It was slightly different in previous semesters.

slide-19
SLIDE 19

Sophie Engle and Sami Rollins Department of Computer Science

Code Review

  • Performed by instructor, not teacher assistant

– Instructor has code review experience – Instructor more strict on design and style

  • Performed interactively with student*

– Each session maximum 20 minutes – Specific criteria evaluated for each project – Comments made directly in students' code and

committed to their svn repositories

  • Result is either pass, conditional pass, or resubmit

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*This is how we are doing it this semester, which has evolved since when we wrote the paper.

slide-20
SLIDE 20

Sophie Engle and Sami Rollins Department of Computer Science

Review Criteria

  • Assume once criteria passed in one project, will be

correct in following projects

– Not ideal, but necessary due to time constraints

  • Inverted index (first project) criteria

– Proper code style (e.g. comments, names) – Proper use of keywords (e.g. static) – Proper generalization (e.g. reusable code) – Proper encapsulation (e.g. no passing references

  • f private mutable members)

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-21
SLIDE 21

Sophie Engle and Sami Rollins Department of Computer Science

RESUL RESULTS S

Grades, Submissions, Lessons Learned

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-22
SLIDE 22

Sophie Engle and Sami Rollins Department of Computer Science

Measurements

  • Compared two semesters

– Fall 2011 using traditional approach – Spring 2012 using expert code review

  • Evaluated student performance

– Number of submissions – Average project grades – SLOC per project

  • Evaluated student experience

– Conducted survey

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-23
SLIDE 23

Sophie Engle and Sami Rollins Department of Computer Science

Caveats

  • Difficult to get statistically significant results!

– Small classes sizes to begin with (≤ 30) – Did not include graduate students – Did not include minors – Did not include ghost students*

  • Only 9 students for Fall and 12 for Spring semester
  • Different types of students in Fall versus Spring

– Fall had separate section for graduates – Spring combined undergrads and grads

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*Some students never showed up, or dropped before the first project suggested deadline.

slide-24
SLIDE 24

Sophie Engle and Sami Rollins Department of Computer Science

Results

1.0 1.5 2.0 2.5 3.0 3.5 1 2 3 4 5

Project Average Submissions

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

Provided for Spring semester (with mastery learning and expert code review) only.

slide-25
SLIDE 25

Sophie Engle and Sami Rollins Department of Computer Science

Results

#S SLOC Grade Project SP FA SP FA SP 1: Inverted Index 3.1 0186 0218 58% 100% 2: Partial Search 2.8 0341 0418 76% 090% 3: Multithreading 2.0 0494 0706 93% 076% 4: Web Crawler 1.2 0686 0914 84% 038% 5: Search Engine 1.0 2360 1781 96% 016% Average: 2.5 0569 0590 81% 082%

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

See discussion in paper.

slide-26
SLIDE 26

Sophie Engle and Sami Rollins Department of Computer Science

Comments

  • "I had fun with the projects and they made me work

hard."

  • "Projects were more challenging because there would

usually be significant refactoring that had be done afer each grading session."

  • "The hardest thing was the grading process of the

projects, it takes way too long for the resubmission process to take place."*

  • "I really understand the idea of object oriented

programming afer CS212."

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

*A very common complaint.

slide-27
SLIDE 27

Sophie Engle and Sami Rollins Department of Computer Science

Conclusions

  • Mastery of code design and refactoring improved

– Supported by decreasing number of submissions

  • Mastery of complex concepts improved

– Supported by higher grades of first three projects

  • Project progression was slower

– Supported by lower grades of last two projects

  • Time required for the code review was reasonable

– Maximum 30 students, 20 minutes per review – Only subset of students need review each week

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-28
SLIDE 28

Sophie Engle and Sami Rollins Department of Computer Science

Drawbacks

  • Time Issues

– Time consuming to setup and get process down – Must coordinate verification and code reviews to

avoid major delays

– Students must wait longer for grades

  • Attrition

– Better prepares majors, what about minors? – Students still fail due to poor code, but now also

poor time management

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-29
SLIDE 29

Sophie Engle and Sami Rollins Department of Computer Science

Drawbacks

  • Progression

– Difficult to reset student expectations – Difficult to acclimate students to new process – Disbelief that I will force them to resubmit when

they already have "working" code

– Difficult to convince students they are running

  • ut of time for submissions
  • Evaluation

– Difficult to calculate student grade mid-semester

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-30
SLIDE 30

Sophie Engle and Sami Rollins Department of Computer Science

CONCL ONCLUSION SION

Summary and Final Thoughts

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-31
SLIDE 31

Sophie Engle and Sami Rollins Department of Computer Science

Summary

  • Some students failing upper division courses
  • Focus on bridge course between lower and upper

division courses

  • Keep traditional approach for breadth on lectures,

homework, quizzes, and exams

  • Use mastery learning enforced via expert code

review for projects and code design

  • Some initial success of approach

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-32
SLIDE 32

Sophie Engle and Sami Rollins Department of Computer Science

Adaptation

  • Requires low faculty to student ratio
  • Requires space in schedule for resubmission

– Difficult on quarter schedules

  • Requires incremental projects

– Otherwise difficult to justify refactoring

  • Requires appropriate subset of topics for mastery
  • Requires mid-level course

– Expert code review less necessary for lower levels

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-33
SLIDE 33

Sophie Engle and Sami Rollins Department of Computer Science

Future Directions

  • Before Class

– Watch slide-based lectures – Watch short code walkthroughs

  • Class Time

– Lab exercises and quizzes – Longer code walkthroughs

  • Code Reviews

– Every other week despite project status

(30 students, 20 minutes, 5 hours weekly)

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-34
SLIDE 34

Sophie Engle and Sami Rollins Department of Computer Science

MOOC Comparison

  • Low enough faculty/student ratio for more one-on-
  • ne interaction than possible with MOOCs
  • Better assessment of student status

– Automatic assessment of homework and quizzes – Manual assessment of projects and exams

  • Still get benefit of recorded videos

– Students can easily re-watch videos – Frees up class time for other activities

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference

slide-35
SLIDE 35

Sophie Engle and Sami Rollins Department of Computer Science

Questions?

  • Sophie Engle

sjengle@cs.usfca.edu http://sjengle.cs.usfca.edu/

  • Sami Rollins

srollins@cs.usfca.edu https://sites.google.com/site/srollins/

  • Course Website

http://cs212.cs.usfca.edu/

April 5-6, 2013, San Marcos, CA CCSC Southwestern Region Conference