Study: U.S. Colleges Ain’t What They Used To Be

Educational achievement has been stagnant for three decades. Can online learning help?

Loading the podcast player...

Steven Cherry: Hi, this is Steven Cherry for IEEE Spectrum’s “Techwise Conversations.”

Here’s a rather striking fact about the United States. Quote: “Levels of educational attainment in this country have been stagnant for almost three decades, while many other countries have been making great progress in educating larger numbers of their citizens.”

That’s from a new research study published by ITHAKA, a not-for-profit organization that describes itself as dedicated to helping the academic community use digital technologies.

In other words, 60-year-old Americans are better educated than 60-year-olds from other countries, but 25-year-old Americans are not better educated than 25-year-olds elsewhere.

Can the U.S. regain its leadership in higher education? Some experts see the recent push toward online learning as America’s ace in the hole, while others see it as yet another way in which it’s let its key institutions—universities, in this case—fall into decline.

My guest today to help us decide which view is closer to the truth is one of the coauthors of the ITHAKA study, Matthew Chingos. He’s a Fellow at the Brookings Institution’s Brown Center on Education Policy. He has a Ph.D. in government from Harvard University, and he previously served as research associate and project manager at the Andrew W. Mellon Foundation.

I should mention that his coauthors in the study were William Bowen, Kelly Lack, and Thomas Nygren.

Matthew, welcome to the podcast.

Matthew Chingos: Thanks so much for having me.

Steven Cherry: Matt, the title of the study is “Interactive Learning Online at Public Universities: Evidence From Randomized Trials.” So you compared an introductory statistics course taught at six different public institutions. Why public institutions, and why statistics?

Matthew Chingos: Well, public institutions because that’s where the students are. Something like two-thirds to three-quarters of undergraduates in this country are educated at public institutions, so we wanted to do this study, this experiment, at a set of institutions that are typical of the kinds of places that a lot of students go. And we chose statistics because it’s a broad introductory course taken by students in a lot of disciplines, in political science, in economics, in sociology and psychology, and because it’s a course that we think is particularly conducive to these models of online learning that we were interested in experimenting with. It’s a course where there’s usually a right answer, which I think is a lot easier to do in an interactive, machine-guided format, than, say, a course like a Shakespeare seminar, where it’s more discussion oriented.

Steven Cherry: One of your goals was to avoid some problems with earlier studies about online learning. What were wrong with them?

Matthew Chingos: Well, a lot of the earlier studies, sometimes they were on a very small group of students; a lot of times it wasn’t a whole semester, it was just maybe an hour in a computer lab; and in a lot of cases, they weren’t randomized. Students could choose to take online or in person. And once you let people make that choice, it compromises the validity of the study because you don’t know—are the people choosing one different than the people who chose the other? So by recruiting students who are willing to be randomized and basically having a flip of a coin determine who got online and who got traditional, that enabled us to say with a high degree of confidence that any difference in outcomes or lack of difference in outcomes, as it turned out to be, could be attributed to the way they took the course, not to something else.

Steven Cherry: Very good. So, you were looking to find out four different things. One was whether online courses were equally effective in community colleges and four-year colleges, but I gather you weren’t able to get good data for the community colleges. So let’s look at the other three. The big one was, Can online courses improve learning outcomes? Can they?

Matthew Chingos: Well, what we found is that they can produce the same outcomes. Of course, when you’re trying something new and innovative, it would be nice if it produced even better outcomes, but that wasn’t what we found. We found it produced similar outcomes as traditional modes of instruction, and students did it in less time, and in the long run, we think this online—well, actually a hybrid model—can save money. So you could expand access to more students, because the idea is that if the computer is doing a bunch of the instruction that was previously done by instructors, now those instructors can now serve a larger number of students.

Steven Cherry: Yeah. So maybe you could just describe this a bit—what you mean by a hybrid model.

Matthew Chingos: So, what we did was...the traditional model is what you think of as a college course. You take—you go three to four hours a week, you listen to an instructor teach you the material. What we did instead was students learn the material, but the lecture’s replaced by this statistics course developed at Carnegie Mellon University where students go online and they read a lot of material and answer questions and get feedback, and then instead of going for three to four hours in person, they go for one hour in person, and that’s an opportunity for the instructor to answer their questions. And the great thing about this is in a traditional class, the instructor doesn’t usually know how well students have learned the material until the first exam. In this interactive system developed at Carnegie Mellon, the instructor actually has access to a data dashboard where they see exactly who’s been doing the work, how well students have been getting at different concepts. So they can go into that face time knowing, “Well, my students really understand the concept of mean, but they had trouble with standard deviation, so I’m going to focus on the subject they struggled with.”

Steven Cherry: We should point out that besides Carnegie Mellon, this does seem to be the wave of the future. The big Stanford online courses last year, the courses for Stanford students, were the hybrid model, and MIT has been experimenting with a hybrid model as well. So, adding an online component didn’t improve learning outcomes, but it didn’t lessen them either, and I gather the students got to the same point somewhat quicker as well. You also looked at whether online courses work as well for unprepared students as prepared students. What did you find, and why did you care?

Matthew Chingos: So, while you could imagine going into this that someone who is less well prepared, they might need a little more hand holding, they might not do as well online—or maybe it’s the reverse, maybe it’s the students who are really advanced are going to get bored by online and are going to need the personal interactions. And we actually found no differences. The impact of doing online versus traditional was the same for students who came in with lower grades than with students who came in with higher grades—you know, from their previous college courses. So that was one thing we were really surprised. We thought different groups of students might respond differently, and we found no evidence of that. The sort of do-no-harm, similar-outcome sort of finding applied to all the groups of students that we looked at in our study.

Steven Cherry: You also looked at the effectiveness of online learning specifically for minority groups.

Matthew Chingos: That’s right. So, just as we could look for different groups in terms of preparation, we were able to look at different racial and ethnic groups. And same thing there: We didn’t find that the online impact was different for different groups of students.

Steven Cherry: Now, despite these lack of differences, which are striking and interesting, the feedback from students didn’t exactly square with those actual results in some interesting ways.

Matthew Chingos: That’s right. So at the end of the semester, in addition to looking at, you know, did students complete the course? How’d they do on the final exam? How’d they do on the standardized statistics test we gave them? On all those measures, they did the same. But when we gave them a survey and said, “How did you like the course?” you know, on average the students in this hybrid online format gave somewhat lower marks, on average. And one reason for that may be that we view the statistics course we use as something of a prototype. It has a lot of great features, it’s interactive, it has these feedback loops built in—but at the same time, it’s not very engaging and maybe not quite as addictive as you might imagine a more sophisticated, further developed course being. So our hope is that future versions of online courses will have some of those engaging, addictive features that will make learning a little bit more fun, if it’s ok to describe it that way.

Steven Cherry: So, specifically, hybrid students reported feeling that they learned less. I mean, they didn’t learn less, but they felt that they did, and they also found the course more difficult.

Matthew Chingos: Right, right. So they did report learning less, and I think that my guess is that it’s just sort of a general set of perceptions. If you don’t like something as much, you probably associate a lot of other things with it, such as “Oh, maybe I didn’t learn as much.” But I would take the actual evidence of learning outcomes with a little bit more credibility than someone’s self-report of how much they think they learned.

Steven Cherry: Do you think that’s because they spent less time with their peers and so they didn’t get that sense of, you know, “I’m average” or “I’m doing even better than average,” even when they were?

Matthew Chingos: Yeah, it’s certainly possible, but it’s hard to say for sure with the data we have.

Steven Cherry: Right. So, stepping back from this specific study, at the top of the show I mentioned differences between the U.S. and other countries, and for that I guess you relied on data from the Organisation for Economic Co-operation and Development. What did it show?

Matthew Chingos: So, it’s really interesting. You mention this comparison of older versus younger generations. The U.S. is one of the few countries in the OECD where the younger generations are not more educated than the older generations. If you look at Americans age 55 to 64 in 2009, 32 percent of them have finished the equivalent of a bachelor’s degree or better. You look at Americans age 25 to 34, same exact number—32 percent. Now, pick a country like, say, Korea: The 55- to 64-year-olds, it’s 12 percent, and the 25- to 34-year-olds, it’s 38 percent. So they’ve more than tripled the share of their population with a college degree, and in the U.S. it’s stayed exactly the same. So what you look at is, you see the U.S. kind of moving down in the rankings list as you go from older generations to younger generations. So basically, in another generation we’re going to go from being the most educated country in the world to one of the least educated countries—you know, among other developed countries.

Steven Cherry: So, I guess the question that I posed at the top of the show is whether online teaching—or at least the hybrid version of it—will save higher education in the U.S. or its rank among other countries. Do you think it will?

Matthew Chingos: I mean, on it’s own I don’t think anything can save higher education in the U.S. Obviously there’s a lot that needs to happen, but I think it can be an important part of the solution, particularly if you want to think about expanding the access mission, making higher education available in a high-quality way to a larger number of students. So one of the things we identified in the study was students did the same in the hybrid format but potentially more students could be served for the same cost. So, a lot of people don’t like to think about cost and productivity, but if budgets are going to stay the way they are right now and you want to expand access, you need to have a way to provide education at a lower cost per student. So I think certain interactive online models have the potential to do that.

Steven Cherry: Yeah. And I guess the most cost-conscious are the community college students, so it’s a shame you weren’t able to come to any conclusions about that.

Matthew Chingos: Yeah it is, we’re hoping in future work to be able to learn more about community colleges.

Steven Cherry: Very good. Well, Matt, higher education seems to be undergoing the same kind of massive tectonic shifting that we’ve seen in manufacturing, retail, journalism, and entertainment, so thanks for bringing some actual data to the table, and thanks for taking the time with us today.

Matthew Chingos: You’re welcome. Thanks so much for inviting me.

Steven Cherry: We’ve been speaking with Matthew Chingos of the Brookings Institution about a new study that looks at whether adding an online learning component can improve college classroom learning.

For IEEE Spectrum’s “Techwise Conversations,” I’m Steven Cherry.

Announcer: “Techwise Conversations” is sponsored by National Instruments.

This interview was recorded 21 August 2012.
Segment producer: Barbara Finkelstein; audio engineer: Francesco Ferorelli
Read more Techwise Conversations or follow us on Twitter.

NOTE: Transcripts are created for the convenience of our readers and listeners and may not perfectly match their associated interviews and narratives. The authoritative record of IEEE Spectrum’s audio programming is the audio version.