0 Vouchers and College Attendance: Puzzling Findings Deserve Much Caution

Several months ago I described the problems in a study that seemed to have great policy relevance, but little empirical support for its contentions.  Sadly, examples of studies like these abound in education, and another is currently making headlines.  "Vouchers Boost Blacks' College Enrollment Rates," claim the stories-- and boy do the effects seem large! A "24 percent increase" in college attendance among black recipients of those vouchers-- what a dream. And it must be an accurate statement, right, since this was an experiment?

Well, not necessarily.

Too many practitioners, policymakers, and even researchers are far too inclined when they hear the words "randomized trial" to ignore the usual concerns about the reliability and validity of estimated program effects.  After all, it's "gold standard," and touted by the Institute of Education Sciences as being the most valid to get a sense of how well programs work. Unfortunately, its usefulness is a bit more limited than that-- first, experiments don't always work as planned in creating equivalent initial groups for later comparison, and second, they often tell us only how well the intervention worked under a set of very specific conditions and circumstances, that are often crucial but rarely described in detail.  Moreover, unless they are really carefully planned in advance, their post-hoc analyses can get particularly squirrelly when it comes to estimating different effects for different people.

For these reasons, I'm not sharing in the wild enthusiasm over the new Brookings study by Paul Peterson and Matt Chingos that purports to show that vouchers provide a big boost to college attendance to a very at-risk group: African-Americans.

I started laying out these concerns a few days ago via Twitter, but am restating and summarizing them here, in case it's useful to those who don't spend all of their time obsessing about methodology and need to know what really works in education.

Here are three reasons why the findings don't pass my sniff test:

(1) The estimated average treatment effect of offering the voucher is null.  Since the effects of receiving the voucher is positive and large for one group- African-Americans-- this implies that the effects must be negative for another group, and yet this is never mentioned.  Why? It's rather unusual to show effects for only selected groups, and not for all of them. Most importantly, it goes against best practices.

(2) The only subgroup with effects, African-Americans, is a group that doesn't seem to have equivalent treatment and control groups before the offer of the voucher.  If anything, the treatment group students seem more inclined to college attendance independent of the voucher, given that more of their parents have bachelor's degrees (while other factors are also imbalanced, this one is a known drive of college attendance, among the most important).  While the authors attend to this issue a bit, and try one kind of sensitivity analysis to adjust for it, in their text they fail the potential flaws all of the cautions they deserves-- even going so far as to making this finding the main highlight of the paper.

(3) In the paper and the press the authors stress the effects of receiving a voucher but voucher receipt is not randomly assigned.  So if you are excited about the experimental component-- in a study that claims to be "The first to measure the impact of school vouchers on college enrollment" -- you need to know that the main result (for example, see paragraph 1 here) isn't experimental. This is a quasi-experimental approach and is subject to the usual kinds of errors.

Are these flaw par for the course, and thus no big deal? I don't think so.  There was an evident PR effort behind this report, and it's led to widespread reporting of a study that really needs more vetting.  Words like "the Brookings Institution at Harvard" (sidenote: huh?) give it more credibility than it deserves at this stage, and the term "experiment" makes folks feel overly confident in the results.

Now, all that said, I do understand how these things can happen.  Since they suggest differential responsiveness to programs (and thus the potential for cutting costs while increasing program effectiveness), subgroup analyses are quite seductive and compelling, as are randomized trials themselves. Last year, my colleagues and I wrote about some tentative findings from our study of financial aid that suggested effect heterogeneity. Prior to the release, we extensively vetted those findings with colleagues, and ran at least five different sensitivity analyses.  After publication of the working paper, which we were careful to describe as "in progress," we sought even more feedback and advice-- and got a crash course in the enormous difficulty in disentangling effect heterogeneity from heterogeneous treatments. Truth is, the work is still ongoing.  And that's an incredibly important and valuable part of the research process, and one we all should wish and allow for-- it makes the work better.

So, here's to hoping that this is what will happen next for this voucher study.  Instead of rolling full steam ahead thinking vouchers will magically boost college attendance for black students everywhere, let's support the authors working through all potential alternative explanations for these odd results, and then replicating their experiment.  Again, my own experience suggests replication is critical, revealing the processes and contexts under which effects occur and are more reliable.  We should all demand it, especially from high-profile studies like these.

0 A New Walker Report on Wisconsin Higher Education

Don't you just love last-minute breaking news when you're trying to head out the door on vacation? Come on.....! 

THIS JUST IN.

Tim Sullivan, businessman, has issued a blueprint prepared for Governor Scott Walker that includes some significant plans for higher education.

Among its highlights:

• The skills gap demonstrated by highly-esteemed economists, as well as leading Wisconsin organizations with sizable expertise in business and higher education is apparently a "myth."  Writes Sullivan, "there are opportunities available if people know where to look and can see the value." (p.101) Sure, he admits his is no scientific study-- after all, he is doing policy analysis by anecdote, drawing on his experiences at his own company-- but gee, he's sure confident there's a myth out there to be busted!
• The costs of Wisconsin Technical Colleges are too high because they are doing too much--namely, wasting time on liberal arts college transfer, "spending millions" before "technical education even comes into the picture."
• The new online initiative in UW System is expected to "change the face of education in Wisconsin."  Boy, and we just thought it was an addition, not a replacement....
• UW System, including UW-Madison, is "failing" to produce enough new start-ups, and needs to be more focused on business collaborations.
• We should open the Wisconsin Higher Education Grant (which already runs out of money every year compared to demand among degree-seeking students) to non-degree seeking students. Yes, he said "open" the grant-- not add funding to the grant. Hmm.
• UW System should pay the additional tuition if students don't finish their degrees in 4 years.  That's right-- UWS-- not the state, even though economists like John Bound and Sarah Turner convincingly demonstrate that time-to-degree rises because states cut funding to universities! 
• The publics should act more like the privates and give out more financial aid.  The latter give out a wider range of aid.  Again, duh....wonder why.

Ok, enough. Read the darned thing for yourself, and write in and tell us all about your favorite parts.  Tim Sullivan, businessman, here to save higher ed.

0 Making the Pell Grant Memorable

In a new policy brief just released by the Scholars Strategy Network at Harvard University, I make the case that the emotional meanings of financial aid can and should be enhanced to promote student success.  Sociologists and psychologists have long known that money has social value and that this can be increased through social connections.  The creators of the GI Bill understood this and took advantage of it by ensuring all recipients understood where funding for that program came from and what it meant. The same must now be done for the federal Pell Grant.  In fact, it could be done for all grant programs.  Governors, mayors, legislators, and yes, presidents, should get involved in conveying a strong, supportive message to the millions of needy yet promising students struggling every day to make it through college.

PS. Just got the following response on Twitter.

@saragoldrickrab TRUTH. as #Pell Recipient, I had no idea why I received 3K for my education.. just thankful w. no one to thank
— Dan S. (@destatter) August 7, 2012

0 Wisconsin Needs to Educate, Not Incarcerate

Yet another policy brief highlights what realists know:  Wisconsin policymakers are presiding over poor policy decisions that threaten to undermine taxpayers' decades-long investment in the state's human capital.

Far from saving our children from lifetimes of debt, those on the neoliberal Left and the conservative Right advocating for either "freeing" state universities from the limitations of state funding in pursuit of market models, or diminishing state spending in a time of austerity, are accomplishing the same goal:  driving up the costs of college attendance and reducing the overall educational attainment of our state's workers.

Forty years ago our grandparents elected officials who invested $14 per $1000 of personal income in higher education.  Today, we elect jokers who put in just $5.  What happened?

Figure courtesy of Tom Mortenson, Postsecondary Education Opportunity

Let's admit it: we aren't leaders anymore, we're laggards. Yes, Wisconsin pays taxes, but we throw away far too much of it on other things.  According to Figure 4 in the new report I referenced above, we rank 32nd thanks to the policy choice displayed above-- relative to per capita income, we are outspent by the likes of Mississippi, Alabama, and West Virginia, not to mention our neighbors Illinois, Indiana, Iowa, and Minnesota.

Where is that money going instead? One simple word answers the question: corrections.  To paraphrase Ronald Reagan, we fought a War on Drugs, and drugs won-- but heck, we are still throwing our money at the problem.  Legacy spending, you might call it.   Over the last 10 years, spending on corrections went up 9%, while spending on k12 dropped by 6% and spending on higher education dropped 20%. Right, because clearly the goal of Wisconsin taxpayers is not to help educate our children, but rather to lock 'em up and shut 'em up.

For those who manage to avoid prison and get into college, instead of investing in their future, Wisconsin taxpayers seem to want their families to foot the bill. How's that working for us? Well, enrollment in our public institutions is lagging behind those in other states.  We have experienced far slower growth in fall enrollment as measured over both 5-year and 10-year periods, compared to the national average (see Table 6 here). Perhaps most startling is how little enrollment in our 2-year colleges has changed-- there was practically no change at all in enrollment there over the last 5 years (0.6%) while the national average was 16.7%! Perhaps not coincidentally, during that time, tuition and fees at the 2-years (already higher than the national average 5 years ago) rose by 20%.

I have to admit being persistently perplexed at how other parents throughout Wisconsin can sit idly by while we pour money intended for our kids into pits of despair like the state's correctional facilities.  It is far more cost-effective to educate rather than incarcerate.  It's time to make our policymakers do right by the limited dollars we have. Let's re-instate a real early release plan, and rollback the ridiculous "truth in sentencing" guidelines that lengthened parole time, greatly increasing the likelihood of being returned to prison. As UW-Madison expert Walter Dickey notes, there are numerous hidden costs to incarceration, and as state we simply can't afford to be in the corrections business.   

The best solution is to treat education as the crime-fighting technique it really is.  Providing young people with truly viable opportunities later in life gives them something to really aim for, helping keep them off the streets and on the job.  A recent UW-Madison graduate, economist Ben Cowan, finds that a $1,000 reduction in tuition and fees at two-year colleges is associated with a 26% decline in the number of sexual partners an adolescent has, and a 23% decline in number of days in the past month he used marijuana.  Policies that support affordable higher education may simultaneously support reductions in the costs of incarceration, in a virtuous cycle that is win-win for all.

This is pure common sense and we all know it.  It's simply time we demand that our "leaders" catch up.