E-learn Weblog of Willem van Valkenburg

Last week Harvard and MIT published an interesting report about four years of MOOCs. They explored 290 Harvard and MIT online courses, a quarter-million certifications, 4.5 million participants, and 28 million participant-hours. That is a bit more than TU Delft, we currently have 1.2 million enrolments of 821,283 unique accounts, 75 courses, 31 thousand certificates, but our first MOOCs didn't start until September 2013. In this blog post I guide you through their findings and have tried to add some comparisons with our DelftX data.

Exhibit 1: Cumulative trends in certification and participation

They see a steadily growth over the 4 years. On average, 1,554 new, unique participants have enrolled per day, over 1,523 days. The leave out the accounts (3.89M) that only enrol and never enter the courseware. We have a weekly growth between 6,000 to 10,000 enrolments per week. Around one third (31%) of unique participants have participated in multiple HarvardX and MITx courses. For just TU Delft we have 27% of the unique accounts that have participated in multiple TU Delft courses. There are almost 6,000 accounts that enrolled in more than 10 courses.

Exhibit 2: Course participation and demographics

Most of their number are the similar to ours. The only big difference is the percentage from the USA. In our MOOCs this is on average 19% as seen in the image below.

Exhibit 3: Certification rates

The difficult issue here is that in the beginning of 2016 edX stopped to offer the free honour certificates. Since than you have to pay to receive a certificate. They state that:

Certification rates are statistics that, properly contextualized, can describe and evaluate MOOCs. The naïve certification rate of 5.5% (245K certificates/4,449K participants) is a useful reminder that not all participants earn certificates, but it is a poor evaluative metric for MOOCs. Not all MOOCs offer free certificates, and not all participants desire certificates.

 They have median of 60% of the payers received a certificate. For DelftX this is 61%.

Exhibit 4: Enrollment trends for repeated and modularized courses

They created an interesting graph on the enrolment trend for repeated courses:

I reproduced this for our DelftX courses. The second run looks similar, but the third run is different. In our data set this is the effect of going to self-paced courses that stay open much longer.

My data set is of 23 parent courses, 4 have repeated 3 times, 6 have repeated two times and 13 have repeated once.

Exhibit 5: Flow of participants between courses 

They made an interesting graph to visualise the flow between courses. Central in their network are the computer science courses. Unfortunately this is not something that I can easily recreate, so I can't give you the DelftX results.

Exhibit 6: Curricular and institutional breakdowns

An interesting finding is the difference between courses from different curricular areas. They distinguise 4 areas:

• CS = Computer Science
• STEM - Science, Technology, Engineering, Mathematics
• HHRDE = Humanities, History, Religion, Design, Education
• GHSS = Government, Health, Social Science

The first question that pops up in my head is, why separate CS? Isn't that part of STEM?

I categorized our courses in the same categories and included the data have available:

* only included the courses that offer honour certificates

So in our data we also see the difference between CS courses and the other courses, although our difference is much smaller in comparison to the data set of Harvard/MIT.

Exhibit 7: Intent

In their MOOCs they deploy a survey and ask the intent of the learner. There are four responses possible:

• Unsure: Have not decide whether I will complete any course activities
• Browse: here to browse the materials, but not planning on completing any course activities
• Audit: planning on completing some course activities, but not planning on earning a certificate
• Complete: Planning on completing enough course activities to earn a certificate

54% of responding participants state an intention to earn a certificate. Of these participants, 16% ultimately earn a certificate. Interesting is their remark about CS50x:

These percentages are influenced considerably by CS50x, which accounts for 1/3 of those who intend to complete but has a certification rate around 1%.

The certification rates of Browsers and Auditors is around 5%, so although these participants stated that they are not planning on earning a certificate.

In our surveys we have a similar questions with these four options:

• I want to complete the course (we don’t ask about the certificate, as courses now don’t have honor certificates anymore, and before we haven’t asked about intentions)
• I will only do specific parts of the course that interest me
• I am just checking this course out
• Other

Currently, 87% of learners in instructor-paced MOOCs say they want to complete the course, and 91% in self-paced (altogether a little less than 90%). We also ask them specifically about their intentions regarding watching videos, doing assignments, forums, reading materials. 98% intends to watch videos, 89% do assignments/quizzes, 94% read reading materials, but only 43% intends to participate in forums (and of them, usually less than 50% actually does in the end, at least for the courses we have analysed so far). This is for instructor-paced, but numbers are similar for self-paced (perhaps small difference in terms of forums, where 38% intend to participate).

Exhibit 8: Teachers

An interesting finding is that there is a high level of participation by those with educational background, including teachers of the course content that is taught.

An interesting follow up question for me, would be if those teacher would reuse the materials of the MOOCs in their own courses. Off course the copyright owner of the course should allow this, which is often not the case for Harvard and MIT MOOCs.

Exhibit 9: Comparing MOOC time online with on-campus credit hours

A very interesting graph is the result of the analysis of the log data. The graph shows the estimated time spent for the all the participants, counting any online activity as continuous as long as less than 30 minutes pass between server logged events. The graph shows that many participants browse courses for mere seconds while many others spend days.

They did the same for the participants that passed the course. On average they spend significally less than 50 hours. The 50th percentile is at 29 hours, 10% take less than 5 hours. Actual times may be higher, because this doesn't include offline activities.

Conclusion

It is an insightful report with some interesting finding. Kudos that they share the data behind the report on their website. I would be interested in some more of their queries extract the data from the edX data set.

Reference

• Chuang, Isaac and Ho, Andrew Dean, HarvardX and MITx: Four Years of Open Online Courses -- Fall 2012 - Summer 2016 (December 23, 2016).  Available at SSRN: https://ssrn.com/abstract=2889436