Udacity And San Jose State See Improvement In Their Online Education Experiment [Updated]

Experimentation is a roller coaster. San Jose State University suspended its controversial experiment in online courses last spring after disappointing results. But the university and its platform partner, Udacity, bounced back on their second try, improving students' outcomes in four of five summer courses, compared to their traditional online counterparts.

SJSU's pilot with Massively Open Online Courses (MOOCs) is supposed to usher in an era of super-low-cost courses for the masses — without sacrificing quality of instruction. In fact, the theory is that the self-paced learning style of online video lectures is more tailored to individual needs than a one-size-fits-all classroom. Students consult with tutors and their peers whenever they desire, and they can listen to lectures as many times as they need to in order to master the material.

SJSU was the first to offer college credit, which caused a torrent of online higher education experiments all over the country. So, when SJSU's Spring semester fell short of expectations, it was potentially a huge setback. Turns out, the failure was premature.

According to SJSU Provost and Vice President for Academic Affairs Ellen Junn, they ramped up orientation efforts for new students, improved email communications during the course, and added ways to encourage students to finish the course. "Everyone needs a little encouragement to stay on track. So we've added tools that help students gauge their progress and we're checking in with individual students more often," Junn said a statement.

One major issue plaguing the experiment is that Udacity and SJSU have opened up the course to high schoolers and part-time college students. According to the Wall Street Journal, "20% were high school students, 62% of students in the pilot were not regular San Jose students, and all of the matriculated ones had failed a remedial math class before." In other words, students with sub-par performance dominated the class, driving down the scores.

I should note that there's an additional level of complexity to the story. The problem of different student populations between online and offline is  compounded by poor reporting methods used by SJSU. Both during the Spring and Summer classes, SJSU reported simple percentages of student outcomes. Contemporary education researchers, however, usually group students by age, gender, race, socioeconomic status, IQ, and/or educational experience. Instead, SJSU just releases the average scores of all the students  (the graph above), even though the average MOOC student is much different from the average on-campus student.

Using modern techniques, instead of simple percentages, we'd likely see that the MOOC was both less of a failure in the Spring and less of a home-run in the Summer–saving everyone a big headache.

Update: Udacity spokesperson tells me that they will be releasing more sophisticated statistical analyses of the courses on September 4th, which is good, because The Chronicle Of Higher Education found out the online student population over the Summer was much different than in the Spring, which further complicates the issue.

Regardless of the outcomes (and the methods of assessment), this is all one big experiment, so we'll have to patient with the outcome. I maintain, however, that SJSU will fundamentally change higher education as we know it.

[Image Credit: Flickr user roarofthefour]