If massive open online course offerings from Harvard University and the Massachusetts Institute of Technology could be described as a city, then computer science would be its vibrant downtown core, surrounded by less densely populated but no less characteristic neighborhoods of STEM, humanities and social sciences courses.
That city continues to grow, researchers at the two institutions are finding, but the challenges of taking MOOCs beyond the experimental stage will require more work than improving a single metric, be it the completion rate, proportion of female learners or bachelor's degree holders, they say.
Harvard and M.I.T. on Wednesday released  what researchers there called “one of the largest investigations of massive open online courses (MOOCs) to date” -- an analysis of 68 MOOCs, 1.7 million learners, 10 million hours of activity and 1.1 billion logged events. The report covers MOOCs offered by the two institutions between July 24, 2012, and Sept. 21, 2014, through edX, a MOOC provider they co-founded.
The courses have faced an intense backlash in the years since The New York Timescrowned 2012 the Year of the MOOC . What was once a promise to transform higher education has since developed largely in three directions: for faculty, MOOCs provide a laboratory to experiment with digital learning; for researchers, a source of learner data; and for users, an outlet for continuing-education credentials.
Critics of MOOCs have attacked the courses’ single-digit completion rates and the fact that most learners who enroll already hold at least a bachelor’s degree. But as the Harvard and M.I.T. report shows, learner demographics and intent vary by the courses they take. Institutions that are interested in future MOOC research should be mindful of those differences, the researchers write.
“There’s no grand unifying theory of MOOCs,” said Justin Reich, the Richard L. Menschel HarvardX Research Fellow, who served as a member of the team behind the report. “Even though there are a lot of these courses, there’s an awful lot to be gained by courses and clusters on their own terms.”
The report groups the MOOCs offered by Harvard and M.I.T. into four “curricular content areas” -- one each for computer science, STEM, humanities and social science courses -- to visualize those differences (see above).
Computer science courses, for example, “continue to put the 'massive' in MOOCs,” the report reads. Although only totaling 9 of the 68 MOOCs offered, computer science courses have enrolled 611,564 learners -- more than half of the total enrollment. That tops the 448,837 learners who enrolled in STEM courses, of which there were 24.
STEM courses, meanwhile, enrolled both the youngest learners -- a median age of 26 -- and the most learners from outside the U.S., at 74 percent. While 68 percent of the roughly 1.7 million total MOOC learners were men, nearly half of those enrolling in humanities courses, or 45 percent, were women. Social science courses (a category that included government and health courses) were most likely to attract learners with bachelor’s degrees, at 81 percent.
"One concern that people have had is about the proportion of students who have a bachelor’s degree or advanced degrees," Reich said. "In intro to computer science, that’s a concern that definitely should be considered. I think very differently about those concerns in public health classes."
The study only included learners who actually accessed the course content, meaning the researchers discarded data from the 1.3 million learners who registered but never showed up.
One goal of Harvard and M.I.T.’s research, Reich said, is to give university leaders data to inform their decision making about MOOCs -- particularly when it comes to ways to measure student performance.
“We’re also trying to tell people we don’t want to optimize any single outcome measure independent of any other,” Reich said. “Different kinds of courses are attempting to do different kinds of things and probably should be evaluated differently.”
MOOC providers launched with individual courses created by high-profile academics. Since fall 2013, one increasingly popular strategy has been to bundle related courses into sequences . The dozens of MOOCs now offered by Harvard and M.I.T. have also given researchers an opportunity to see which pathways learners themselves follow. The popular computer science MOOCs, in particular, serve as hubs for other content, both in computer science and other disciplines, according to the report.
Those pathways could also help researchers determine what makes a MOOC successful, Reich said. “There may be some classes where people are way more likely to register for other classes,” he said. “That may be an indication they’re having a good experience. It would also be really neat to find out if people who do well in one class do well in lots of classes. We now have the data to explore that.”
Despite fluctuating levels of MOOC hype, Harvard and M.I.T.'s courses have seen steady enrollment growth -- an average of 2,200 new learners a day. But the stability also means a large number of learners are choosing not to re-enroll, Reich said. That's why the report ends with a different metaphor, describing the various metrics used to evaluate MOOCs as "interacting ingredients in a recipe" to improve both online and face-to-face learning.
“If those numbers were exponential growth, then every year would be the Year of the MOOC,” Reich said. “If those numbers had fallen to zero, we would be closing up shop. With linear growth, you have to start asking -- if our population learners will conceivably stabilize -- are institutions happy with that? What should we do with it? Knowing that we have a revolving population of learners doesn’t answer any questions, but I think it makes the realm of possibilities a lot clearer.”