0 Grading as the New Battlefield

This guest post is authored by R. Thomas, a graduate student at the University of Wisconsin-Madison.


Grading might not be the most exciting aspect of a teacher’s career but it might be one of the best sources of job security - particularly in light of proposed online courses with automated grading that aim to put professors out of work or into unappealing, low-wage, jobs.

Of course we’ve heard a lot about online education, MOOCs, merit badges, and competency assessments. As Clayton Christiansen describes it, universities save money with online education simply because there is a glut of PhDs. For years graduate schools have been over producing the amount of PhDs. Consequently talented instructors are more than willing to teach courses at low wages, with no benefits, and with essentially no job security. As he writes in The Innovative University,

“Adjunct instructors give the online educators two advantages. Rather than receiving an annual salary, as full-time faculty at traditional universities do, online instructors are paid by the course. This means that the online university can match teaching supply to student demand - the instructor is hired, or contracted for, only when it class is likely to have enough students to generate an operating profit.  Also, an online instructors teaching performance is easily monitored, and an underperformer has no contractual right to further employment.” (page 213)

According to this view, it will be a lower class of underpaid instructors that will help bring higher education to the masses at a cheaper cost. It is a Wal-Mart version of higher education- a vision that combines both inexpensive education and legions of poorly paid instructors.

One might hope that this is just a minority view, a freakish nonrepresentative example. In fact you might say that there are structural barriers to the spread of online education: instructors are simply needed to grade students’ essays and exams. Unfortunately widespread changes might not be so unlikely. Yesterday a New York Times article described an effort to go beyond multiple-choice and actually automate the grading of essays. The vision is that a few superstar lecturers will deliver content to hundreds of thousands of students, whose grades will be established by a computer. To say nothing of whether the computer can actually grade an essay, one can imagine the impact on jobs for PhD’s. Another article from the Chronicle of Higher Education described a similar vision. The State of California has proposed a new university that requires no coursework whatsoever. Students simply need to take multiple-choice exams to receive a degree.

American higher education is commonly characterized as unique among industries in having missed the technological revolution. Universities have simply not garnered the productivity gains that other industries have received, mainly because teaching cannot be automated. Consequently college is exceptionally expensive. Advocates for greater access and cheaper prices maintain admirable positions; however the question remains who will pay for greater access and cheaper tuition. Many students cannot afford to pay for it, the private sector will not pay for it, and the states will not pay for it. Such efforts as described in the New York Times and the Chronicle suggest a technological solution. Many professors are to be replaced by machines, just as factory workers were replaced by robots. Maintaining control of grading is one way of preventing this change. Indeed, ownership of grading may soon become a major battleground for American professors and it seems that at least some of the American professoriate is rising to the challenge.

0 The MOOC Industrial Complex (Part II)

About a month ago I began writing about the MOOC Industrial Complex.  As expected, it's growing rapidly.  Today's NY Times tells us that soon computers will be grading essays at the college level.

Not only will those brainy professors be replaced by technology, but heck even better, the computer will give you a "retake" so you can try and improve your grade! Gee, how lovely.  The creator even emphasizes the importance of that immediate feedback and instant gratification, since (oh my goodness!) students currently have to wait days-- or even weeks!!-- for their essays to be graded.

It won't be long now, we're told, as the system is "nearing the capability of human graders."

Watch out you graduate students and professor-- the machines are coming...

Just wait, as one of my favorite Twitterers Siva Vaidhyanathan writes, for the students to line up at the computers' office hours (??) to protest and renegotiate their grades.

0 Those Selfish PhD Candidates

One of the most stunning moments in the recent Ed Talks Wisconsin came towards the end of a Friday night discussion about MOOCs. The room was heavily populated with graduate students, many of whom were asking about the implications of these online courses for their employment prospects.  With the decoupling of teaching from their future responsibilities, many were (rightly) worried about how they'd be trained, funded, and what they'd do post degree.

As one student put it,"What's the incentive for the next generation of scholars to pursue a PhD?"
 
In response to that question, the Chancellor of the UW Colleges, Ray Cross, responded this way:
"Is your goal to get a PhD, or is your goal to change education?"

Many in the room looked up, confused about whether he was serious.

Well, an email that just arrived from UW-Madison suggests he was.  The newsletter it included contained the following key blurb:

"New program trains students to create online courses.  With the advent of online courses and degrees, a new program will teach students how to provide online course support. Student Online Course Support (SOCS) will help students find campus tech-related jobs, expose them to new career paths, and save the University funds by hiring student support. Tell your students about this opportunity."

Got that? It's full-speed ahead in support of the Flex Degree and the MOOCs here at Madison; above I linked to the course description that explicitly references these programs. Want a campus job? Get on board. And be sure, of course, to save the university money!

Will this next become a mandated part of our graduate training? Or will we no longer need those folks, as our younger Badger are trained and ready to serve?


ps. The video of Cross starts around the 1:32 minute mark.

0 How to Evaluate a MOOC

Since we're already doggie-paddling around the deep-end of the MOOC phenomenon, it seems high time to begin thinking about how to assess MOOCs.  This is far from straightforward, since they are not typical courses and might best be conceived of as extension or continuing education activities.

Very few colleges and universities are revealing their assessment practices for MOOCs; Duke is one exception.  This report explains that selective university's experiences with one course, and demonstrates how their administrators and faculty thought about measuring success. In my view, some of it is good, and some of it is exactly what not to do.

So here is my overview of an initial evaluative framework for thinking about whether or not a MOOC was a "success." I'm posting this with the explicit intention of generating discussion to refine this framework and build it out, so please add your two cents.

The framework has three components: an assessment of costs expended to generate and implement the MOOC, an assessment of its intended impacts in multiple domains, and as assessment of unintended consequences.  Only by considering all three can one make an informed decision about whether a MOOC was worthwhile, yet reporting on MOOCs suggests that many institutions are focusing on only one or two of the three.

1. COSTS
1. How many total hours of effort were required to build the MOOC?
1. How many of these were on the part of the faculty member?
2. How many involved TA time?
3. How many involved staff time?
2. What technological costs were incurred?
1. How much was spent to get the MOOC off the ground?
2. What are the regular maintenance costs?
3. What are the costs of upgrading the MOOC?
3. How many hours of administrative time were devoted to meetings about the MOOC? 
1. How much time and money is spent advertising the MOOC?
2. How much time will be required to regulate it?
3. How much time will be required to assess its outcomes?
2. IMPACTS
1. On Students.  
1. How many students initially enrolled in the MOOC?
2. How many of these were students were not already enrolled in the university?
3. How many countries are represented among MOOC users?
1. How many of these countries were previously not served by the university, or represent new relationships?
2. How many of these countries are home to under-educated populations?
4. How many of these students had no college experience prior to the MOOC?
5. How many of these students already held a college degree of any kind?
6. What percent of the total initial enrollment completed the first week of the course?
7. What percent of the total initial enrollment completed the first assessment?
8. What percent of the total initial enrollment completed the entire course?
2. On Faculty.  
1. What was the professor's level of satisfaction with the course, as compared to satisfaction of teaching a usual university course?
2. What new contacts for the professor resulted from the MOOC, related to his/her research agenda?
3. How did teaching the MOOC inform the professor's research agenda?
4. How did teaching the MOOC change how the professor approaches his/her regular university courses?
5. Is there any evidence that the professor and/or his/her research gained more recognition in the community where MOOC students came from? (e.g. media mentions, etc)
3. SIDE EFFECTS
1. How many course releases (from campus teaching) were required for the professor to teach the MOOC, and how many students on average are taught per course (typically)?
2. How many press mentions occurred regarding the MOOCs?
1. How many were positive?
2. How many were negative?
3. What did state legislators say about the MOOCs?
1. How many felt the MOOC was a welcome activity?
2. How many felt the MOOC was an unwelcome activity?
4. What is the dollar value of the activities incurred by the MOOC contracted out to the provider rather than performed "in-house"?

As I said, this is just a rough start.  Write in, and help me flesh this out!

0 Performance Funding, MOOCs, and Public/ Private Distinctions

The discourse around higher education on two key topics-- MOOCs and performance funding-- is unfortunately trending toward a generic approach to institutions.  Lumping public and private universities together is always unwise, because not only do their funding sources differ, but so do their missions and masters.

Public institutions still lean heavily on taxpayer support to provide undergraduate education, and that funding is subject to rules developed by externally, democratically controlled governance units.  They are explicitly responsible for expanding access, growing the proportion of the public that is educated at the postsecondary level.  These conditions represent both benefits and constraints to public colleges and universities.

In contrast, at private instituions tuition and endowments provide most of the resources for undergraduate education, and rules for that funding and spending are mainly developed internally.  Their missions vary widely and are internally determined.  These conditions also come with benefits and consequences, but quite different ones than those faced by public schools.

Given these differences, let's consider the following questions:

(1) What are the effects of holding both sets of institutions to the same metrics/outcomes?  They are not equally in control of the inputs critical to achieving outcomes. Also,  the impacts of achieving an outcome like graduation comes as a cost for other outcomes --such as access- for which public institutions alone are truly held responsible.

(2) What are the differential effects of implementing MOOCs in public and private schools?  They fit within institutional mission and resources in very different ways in these spaces.  Private institutions can decide to devote resources to outreach and brand recognition, but public institutions must first excel at their core efforts-- expanding access and providing high quality education at a low price point.  Private institutions must answer to their boards for such activities, while public institutions must answer to taxpayers.   As my colleague Chuck Rybak at UW-Green Bay recently asked via Twitter, "Is someone going to ask if it's ethical for state schools to offer MOOCs/coursework to non-Wisconsin taxpayers for free?"  I bet they are now...

Too often those inside higher education focus on the activities of their elite, private "peers" without acknowledging that these aren't our peers at all.  It reminds me of high school. Why are the really smart, devoted, activist kids who are great at so many things but don't invest time putting on make-up trying to pander to the whims of the homecoming kings and queens who invest in dating and shopping for cars?  Why are those capable of changing the world by helping the truly disadvantaged gain their footing instead spending their time chasing technological innovations that eventually will mainly serve to line the pockets of the already-wealthy?  And when will we stop them from imposing their technocratic "rules" on our hard work?

0 The Illusory Appeal of the UW-Madison MOOCs

It was only a matter of time.  The anticipation was palpable when a group of about 50 concerned faculty, students, and staff from around UW-Madison gathered last summer to talk about the governance crisis at UVA.  The president of that flagship university had just been ousted for failing to quickly embrace MOOCs-- a sign some UVA board members thought meant that she was failing to embrace the "disruptive innovation" that will purportedly transform higher education.  While President Teresa Sullivan was reinstated some weeks later, after a period of alumni outrage, the writing was on the wall.  MOOCs were established as an especially hot new trend with broad appeal among the powerful who find "shock and awe" scenarios the best way to promote change-- and it wouldn't be long before UW-Madison lept into the quicksand.

This morning UW-Madison is going public with the news that we're joining with Coursera to offer four pilot MOOCs.  Our administrators have apparently decided MOOCs are "compelling"enough to jump on the bandwagon.

This is a pretty remarkable turn of events, for several reasons. Let's set aside for a moment that relatively little open, public discussion of MOOCs has occurred on campus, especially discussion involving faculty and students.  But frankly, I can't even pretend surprise about that anymore. As usual, the University Committee and "numerous" faculty were engaged in discussions with administrators, but obviously that's small potatoes on a campus of 2,000+ professors.  If surveyed, I'm willing to bet that 50% or more of the professors at Madison don't know what a MOOC is, despite the presence of this "thought piece" crafted "for considering if MOOCs are right for UW-Madison at this point in time" and placed on a website most faculty don't know about.

Instead, I'm a little more surprised at the apparent willingness of the Madison administration to engage in the illusions/delusions of grandeur posed by the MOOC movement. Sure, this is just four courses, a small step forward, and it's framed as a pilot effort to "learn by doing."  It's not as if the entire campus has gone for MOOC madness.  But "learning by doing" is often akin to a gateway drug--once you've done a little, you might as well do more-- after all, you've already make the investment.  Furthermore, if you're really learning by doing, you build in continuous assessment of both intended and unintended consequences-- and the administration's documents don't mention of this sort of work at all. (Fingers crossed that the evaluation plan is among a detailed set of documents we'll soon get to see.)

The rhetorical flourishes of MOOCs abound, with talk of great public service and institutional branding. But in some key areas, the reality seems far grimmer.  Consider this:

1. MOOCs cost money and apparently generate little in return. We are constantly told that financial resources are scarce on campus, and hard to come by.  But as Madison's administrators know very well, MOOCs "require upfront investment without directly generating new revenue."  The business model for MOOCs is simply unclear.  What even the administration suspects is that it likely threatens the employment of our teaching assistants.

2. MOOCs take faculty time. There's rampant complaints of "overwork" among Madison faculty and yet at the same time demand by our students for access to more courses with us.  MOOCs require time to develop and teach.  If course releases are provided to do this, it pretty much has to come at a cost to UW-Madison students-- or involve further overloading the faculty.  This leads me to wonder, why would we release our professors,  paid with state appropriations and student tuition, to teach students elsewhere?  While it's grand and laudable, expanding their influence and potentially improving our rankings, how is this any different than funding a course release for a professor who flies off to teach at Harvard or in Shanghai?  We don't do that, so why do this?  Professors can and do engage in such things on their own, but they aren't paid by us for it. Why should this be different?

3. MOOCs don't address the core problems we face as a state institution.  MOOCs will neither strengthen our relationships with the Wisconsin legislature or rural communities or help us deliver greater economic development returns to the state.  While these are not all we are tasked with doing, or have the ambition to do, by investing energy in MOOCs rather than those other activities, we are revealing our priorities.  There are opportunity costs to every effort, and this one is occupying our time.

4. Despite their name, there is little chance that MOOCs will serve the masses truly in need of free educational opportunities. The Madison document suggests that MOOCs will "improve lives" and increase "access to education" and thus help fulfill the Wisconsin Idea. This is the grandest claim of all, and yet it's the most far-fetched.  There's no evidence to suggest that the people taking advantage of learning from MOOCs are the under-educated-- but rather they are the highly-educated seeking to learn a bit more in the process of lifelong learning. An assessment of one of Duke's MOOCs found that just 11% of the MOOC enrollment came from students who hadn't attended college-- and the tiny percent of course completers usually held a BA or higher.  MOOCs aren't a substitute for postsecondary education; they are a continuation of it.  They represent not our core mission but the "extras" that we hope to provide at Madison-- if only we can afford to do so.

So let me conclude with this. I love UW-Madison and I love it when we try new things as a university.  Pushing the envelope is a sign of bravery, and our faculty are nothing if not fearless.  But jumping onto an already-moving train just because our peers are onboard, at a time when we have scarce resources and much to lose, well, it looks a bit insane.  And it's really scary to think that some of our administrators subscribe to a shock doctrine approach to educational change, awaiting a "sufficient sense of urgency" to reach the campus to get it engaged in MOOCs.

Let's really debate the MOOCs, sifting and winnowing our way through the process. Let's talk about how to assess this pilot effort, and what sorts of outcomes must be measured.  Let's figure out a plan to put an end to these pilots if they are costing us more than they're worth.  As my colleague Kris Olds has put it, "There are political and economic machinations associated with the stirring of interest in, and coverage of, MOOCs. Given this, and given the stakes at hand, it is important to address the MOOCs phenomenon is a serious, sustained, and reflective way, not in a knee jerk fashion, one way or the other."  Agreed. And now Olds is creating one of Madison's 4 new MOOCs.

With that,  I invite you to a conversation about MOOCs taking place during ED TALKS Wisconsin, Friday March 15 at 7 pm at the School of Education.  Our keynote speaker is Anya Kamenetz, and Kris Olds will be responding to her, along with Chancellor of UW Colleges Ray Cross.  It should be a really good time-- hope to see you there!


ps. Need a laugh? Check out what North Korea's up to regarding MOOCs. (H/T Gary Rhoades)

0 The Higher Education Lobby Comes to UW-Madison

This morning, the UW System Board of Regents heard from a prominent speaker: Molly Corbett Broad, President of the American Council of Education.  Then, around noon, she joined a group on the UW-Madison campus to share a similar talk, but this time with an audience of faculty, staff, and students. Both talks focused on the theme of "higher education at a crossroads."

I had the honor of introducing President Broad to the second audience, in my role as Senior Scholar at the Wisconsin Center for the Advancement of Postsecondary Education.  I also moderated the discussion portion of the conversation.

As I'm grateful to Broad for joining us, I feel it's among the most respectful acts to fully engage with her comments and offer my thoughts and questions here.  Simply receiving information from a talk without vigorously considering and debating the ideas is inconsistent with the spirit of the Wisconsin Idea.  So, with that in mind, here are my thoughts.

First, let's begin with 10 key points Broad made in her talk:

1. Today's undergraduates are quite different from yesterday's. In particular, they are often older, have fewer resources, and are especially interested in getting a return on their college investment.
2. Students' reactions to loan burdens are often inappropriate, given the sizable return to college degrees.  Broad noted that it's better to have a college education than to own a home. She also clearly stated that students who take on too many loans are greatly in need of financial education.
3. Many  colleges and universities are struggling financially to stay afloat.  Broad posited that this was an inevitable trend, unlikely to change.  She showed a slide indicating that 50% of universities are reporting lower enrollment growth.
4. The federal government, said Broad, is a minor player in higher education and it's best that it remain that way.  A growing role of the federal government, she said, would be unwelcome.  She told the audience to remember the old adage that those with the gold do the ruling (or something similar, implying that the federal government will try to control whatever it funds).
5. It is therefore problematic, noted Broad, that a growing number of undergraduates are "dependent" on the federal Pell Grant. 
6. There is a lot of variation among institutions, Broad pointed out. To illustrate this, she showed data indicating that the selective private universities admit just 13% of applicants and spend $75,000 on each student, while less selective privates admit 67% of applicants and spend $21,000 per student.
7. It is important, said Broad, that we provide better counseling to explain to students (especially those without family resources) that "not every student can afford to attend every institution."  
8. We have done a great job expanding access to college, say Broad. 
9. The classic iron triangle of higher education (which I've dissected many times, including in a blog for the Board of Regents) is now outdated, according to Broad, and must be rethought with a focus on innovation.
10. The key innovation lies in online courses and services.  Broad said "isn't it time for us to take lessons from private business?" and pointed to MOOCs.  Udacity, Coursera, and EdX "will help higher education meet the forces of change," she reported. Of course, Broad noted the usual concerns about academic quality, and then said that we could be confident that ACE was helping ensure quality control. Specifically, she said that ACE had been "retained" by the MOOCs to assess quality.

Before providing my own comments, here is a taste of what the audience at WISCAPE had to say in response. First, a graduate student asked about the implications for academic labor, noting his concern that graduate student training opportunities would be diminished if MOOCs were the main mechanism for course delivery.  Broad responded that graduate student experiences with regard to teaching were already subpar.  A staff member asked how we might alter tenure incentives to focus more on innovative teaching, and Broad said she didn't know.  A faculty member asked about how MOOCs generate money for their schools-- and Broad said that remained to be seen.  And finally, the Chief Information Officer of UW-Madison noted that broadband access, critical to online education, is woefully insufficient in some parts of the country, and asked Broad to do the lobbying to help shore it up. In response, Broad suggested that he contact the Obama Administration and let her know what they said.

Ok.  I was struggling to stay silent throughout all of this for one major reason:  There is a abundant gulf between the (mainly accurate) trends among undergraduates and institutions that Broad described and the solution she offered.  So I asked her: "If on the one hand we know that the number of economically disadvantaged students in higher education is growing, and many colleges and universities have fewer resources with which to serve them, how can we expect a solution (MOOCS) which provide less student contact (particular the hands-on kind important to success of first generation students) and no additional revenue to help?"  

Her response was that our current model is broken, and that she would not want to be the person to "underestimate the potential of disadvantaged students to benefit from MOOCs" as much as any other student.

Hmm.  Well.  

First, this must be the new version of the "soft bigotry of low expectations" for higher education.  Now it's the skeptic of MOOCs who doubts the potential of poor students, rather than the potential of the proposed educational plan? This was especially remarkable since the comment was directed at a researcher who spends inordinate amounts of time with the same Pell recipients she was discussing-- who, by the way, would take offense at being termed "Pell dependent."  (Recall when welfare receipt became welfare dependence, folks?)  What I know, from spending time on the campuses of community colleges and less-selective institutions, is that their students prize their time with people-- they are happy to have technology be a part of instruction, but it cannot replace personal interaction.  If anything, perhaps MOOCs are best suited to the sorts of students increasingly in the minority/nontraditional category of higher education-- like the artificial intelligence students with whom Sebastian Thrum discovered his love for teaching online.

Second, it is uncommon for academics like Broad to uncritically accept and repeat claims that a system is simply "broken," without questioning how and why it broke and for whose benefit it is broken. To her credit, Broad alluded to politics when questioning the right of the federal government to be involved in the work of colleges and universities.  But she did not address the broader trends resulting in the defunding of education and research, the push towards "innovation" generated in the private sector, or the increasing focus on the deficits of students rather than the institutions serving them.  When declaring the push for access a "success," she failed to note that the chances of attaining a bachelor's degree for students born in the bottom income quintile is less than one in ten.  Moreover, instead of pointing to states for diminishing appropriations for higher education and driving up tuition, and institutions for catering to the rich students rather than keeping college affordable, she blamed students for their poor choices with regard to loans.

In the end, the picture Broad painted today was not so much of higher education at a "crossroads," but rather a disturbing vision of colleges and universities frantically trying to pull up the drawbridge and create a new moat for their protection. To keep out those unwashed masses of unkempt nontraditional students, and prevent federal "intrusion" colleges and universities can no longer simply raise tuition-- the public will not stand for it. Instead, they must shift to protecting the elite survivors (the A institutions, I think she called them) by generating MOOC courses that can be launched into the cloud to create a virtual wall -- satisfying those new degree-seekers whom colleges and universities will never adapt to serve in person the way their administrators and professors will serve their own kids.  Such efforts will only thrive at the most elite places, since as Broad noted, MOOCs require course release time to develop-- and the reality, despite her statement to the contrary, is that such time is incredibly rare these days. 

Will "quality" postsecondary education survive?  Does is now exist?  Color me unimpressed by the fact that ACE has been "retained" by the MOOCs.  A look at ACE's website and reports suggests close ties between the two, and tight relations with the Gates Foundation-- not what you want to see when looking for independent assessment.  And more importantly, it seems that the persistence of elite status among those powerful institutions fueling ACE's fire depends on the success of the MOOCs.  For without these educational alternatives, a real revolution might erupt-- with the masses actually demanding the same types of rich college experiences that American undergraduates are famous for enjoying.