E-learn Weblog of Willem van Valkenburg

Last month researchers from the University of Washington published an article about MOOC usage for professional workforce development outcomes in Colombia, the Philippines, & South Africa. There were a couple of news sites that used this article to say that students from developing countries have a high completion rate in MOOCs. To me that didn't seem like a logical conclusion, so I asked our very smart data analyst Sara Topolovec if we could see the same behaviour in our data set of MOOC enrolments.

There are two sets of data to look at:

• our surveys include questions that are similar to those of the UW research
• platform data

Survey data

If we look at the survey data (for 10 courses), the results are:

• 80% of both groups of students (developing and developed countries)  reported they had taken an online course before;
• 12% of students from developed countries that had taken an online course before, never completed any course, and 16% from developing, which means that
• 88% of students from developed countries that had taken an online course before state they completed at least one course, and 84% of students from developing countries.

Here we see that many students actually completed at least one course in both groups, but overall percentages are still slightly “in favour” of students from developing countries.

Platform data

If we look at the platform data (example of one course):

• The average grade of developing countries is 3,73% vs. 5,55% for developed, and the passing rate is 2,93% vs. 5,26%.
• There are more students from developing countries that hadn’t really started the MOOC (i.e. had a grade = 0), 91% vs. 87%, but even among those that did (grade > 0), the average grade is lower (36% vs 40%), as well as passing rates (32% vs. 47%).
• Only when you look only at people who received a certificate, the average grade is basically the same.

Because Sara is very good, she also had some comments about shortcomings of the original research:

• It tries to compare the result of their survey about completing any course ever to per-course completion rates. While it acknowledges that they don’t really have the actual data for comparison, it is still misleading to mention it alongside it, because in reality, it doesn’t tell us anything. They have no idea how many students in general completed at least one course already in developed countries.
• The comparison is problematic even further because of very different sources of information. While the completion rates are actual, true numbers, the survey is an estimate, that can be largely biased. We know that more students that receive a certificate in the end complete even the pre-survey (compared to the actual passing rates), which may also be true in this case, i.e. more engaged students completing their survey.
• Also, they compare their “completion” to per-course “completion rates”. But, our completion rates are actually certification rates, while their “completion” is completing the course with not necessarily receiving a certificate. Furthermore, students may understand very differently, what does it mean to complete a course, possibly connected to their intentions. We have no idea how many students in our courses would say they completed the course. So this is another point why these numbers can hardly be compared to the regular “completion rates”.
• On a per-course basis, the number of “registrants” is rather high (usually only around 50% of students does anything in our courseware), which is very far from the 2% of “registrants” they identified in their sample, which further shows that their completion numbers and course completion numbers can hardly be comparable.
• Their study is conducted only on people between 18-35 years old. As we know there are many students above 30 (30 is usually the median) in our courses, this is hardly representative of all MOOC users.

Conclusion

The researchers based their conclusion solely on self-reported survey data, but tried to compare their result to actual per-course completion rates, which creates a false sense that students from developing countries actually complete more courses than their peers from developed countries. While the high completion rates among students from developing countries may still be a surprise, it is important to keep in mind what we are actually looking at. Both platform data and survey data of ours revealed that somewhat more students from developed countries complete courses, or receive a certificate.  In most of our research we combine the survey data with the platform data to get more accurate results and less user bias.