0 Live By The Sword, Die By The Sword?

The problem with Jay Mathews' defense ("Measuring Progress At Shaw With More Than Numbers") of a Washington, DC school principal who did not demonstrate student learning gains at his school after one year is that the principal operates within an accountability system that demands such a result. In this case, both Mathews -- and DC Schools Chancellor Michelle Rhee, as described in Mathews' WP column -- are right not to have lowered the boom on Brian Betts, principal of the DC's Shaw Middle School at Garnet-Patterson, based on a single year's worth of test scores.

The state superintendent of education's Web site says Shaw dropped from 38.6 to 30.5 in the percentage of students scoring at least proficient in reading, and from 32.7 to 29.2 in math.

But those were not the numbers Rhee read to Betts over the phone.

Only 17 percent of Shaw's 2009 students had attended the school in 2008, distorting the official test score comparisons. Rhee instead recited the 2008 and 2009 scores of the 44 students who had been there both years. It didn't help much.

The students' decline in reading was somewhat smaller; it went from 34.5 to 29.7. Their math proficiency increased a bit, from 26.2 to 29.5. But Shaw is still short of the 30 percent mark, far below where Rhee and Betts want to be....

Despite the sniping at Rhee, the best teachers I know think that what happened at Shaw is a standard part of the upgrading process. I have watched Betts, his staff, students and parents for a year. The improvement of poor-performing schools has been the focus of my reporting for nearly three decades. The Shaw people are doing nearly everything that the most successful school turnaround artists have done.

They have raised expectations for students. They have recruited energetic teachers who believe in the potential of impoverished students. They have organized themselves into a team that compares notes on youngsters. They regularly review what has been learned, what some critics dismiss as "teaching to the test." They consider it an important part of their jobs.

That's how it's done, usually with a strong and engaging principal like Betts.

Mathews' take -- including consideration of contextual factors, such as the fact that only 17% of the school's students had attended the prior year and the contention that school turnaround requires more than a single year -- is how the education world should work. Embrace the complexity of learning and trying to measure it! To do so would disallow the use of single-year changes in test scores for making high-stakes decisions about schools and individual school personnel. It would also remove the unrealistic pressure on school turnarounds to bear fruit in a single year. Test scores would be used responsibly in combination with other data and evidence to paint a fuller picture about individual school contexts and inform judgments about school leadership and student success.

But Michelle Rhee and other education reform advocates have publicly argued that student performance as measured by test scores is basically the be all and end all. According to this Washington Post story ("Testing Tactics Helped Fuel D.C. School Gains"), Rhee supports strengthening No Child Left Behind to "emphasize year-to-year academic growth." Such a stance creates a problem for such reformers when they are leading a district and staking their leadership on uncomplicated test score gains. Others will assess their leadership and judge their success by this measure -- an ill-advised one in its simplest form.

I would argue that, in addition to doing the right thing (as happened in this instance), reform advocates and school leaders like Rhee also have a responsibility to say and advocate for the right thing. They have a responsibility to be honest about the complexity of student learning and the inability of student assessments to accurate capture all of the nuance going on within schools and classrooms. While the reformers' challenge of the adult-focused policies of the educational status quo is often warranted, some reforms -- accountability, chief among them -- have been taken too far. Student learning, school leadership and teaching cannot be measured and judged good or bad based on a single set of test scores. Test scores must be part of the consideration -- and supporting systems such as accountability, compensation and evaluation must be informed by such data -- but they should not single-handedly define success or failure.

The complexity as presented by Mathews in his article -- and, more importantly, by existing research (such as by Robert Linn, Aaron Pallas, Tim Sass, and embedded within Sunny Ladd's RttT comments) about year-to-year comparisons of both overall test scores and test score gains -- strongly suggests that educational accountability systems should be designed more thoughtfully than they have been to date, but unfortunately that does not seem to be the direction that policymaking is headed at either the federal or state levels. Part of being more thoughtful is moving away from NCLB-style adequate yearly progress and toward a value-added approach, but thoughtfulness also requires not making high-stakes decisions based exclusively on volatile student data. Do I hear "multiple measures"? Sure, but Sherman Dorn offers some provocative thoughts on this subject in a 2007 blog post.

With regard to educational accountability, policymakers first should do their homework -- and then they clearly have more work to do in creating a better system and undoing parts of the existing system that aren't evidence-based and accomplish only in simplifying a truly complex art: learning.

-------------------

For those of you that have gotten this far, there's a related post on the New America Foundation's Ed Money Watch blog discussing a new GAO report that analyzes state spending on student assessment tests -- $640 million in 2007-08.

The increasing cost of developing and scoring assessments has also led many states to implement simpler and more cost-effective multiple choice tests instead of open response tests. In fact, although five states have changed their assessments to include more open response items in both reading and math since 2002, 11 and 13 states have removed open items from their reading and math tests, respectively over the same time period.... This reliance on multiple choice tests has forced states to limit the content and complexity of what they test. In fact, some states develop academic standards for testing separately from standards for instruction, which are often un-testable in a multiple choice system. As a result, state NCLB assessments tend to test and measure memorization of facts and basic skills rather than complex cognitive abilities.

------------

And here's a new story hot off the presses from Education Week. It discusses serious questions raised about New York City's school grading system.

Eighty-four percent of the city’s 1,058 public elementary and middle schools received an A on the city’s report cards this year, compared with 38 percent in 2008, while 13 percent received a B, city officials announced this month.

“It tells us virtually nothing about the actual performance of schools,” Aaron M. Pallas, a professor of sociology and education at Teachers College, Columbia University, said of the city’s grades.

Diane Ravitch, an education historian at New York University, was even sharper: She declared the school grades “bogus” in a Sept. 9 opinion piece for the Daily News of New York, saying the city’s report card system “makes a mockery of accountability.”

But Andrew J. Jacob, a spokesman for the New York City Department of Education, defended the ratings, even as he said the district’s demands on schools would continue to rise next year....

The city employs a complex methodology to devise its overall letter grades, with the primary driver being results from statewide assessments in reading and mathematics, which have also encountered considerable skepticism lately.

The city’s grades are based on three categories: student progress on state tests from one year to the next, which accounts for 60 percent; student performance for the most recent school year, which accounts for 25 percent; and school environment, which makes up 15 percent.

Mr. Pallas of Teachers College argues that one key flaw with the city’s rating system is that it depends heavily on a what he deems a “wholly unreliable” measure of student growth on test scores from year to year that fails to account adequately for statistical error.

0 Teacher Residency Requirements

Apart from being marginally good local politics to require city employees (including teachers) to live within city boundaries, why would an urban district create barriers that make it more difficult to attract the highly effective teachers that it needs?

Ask Chicago and Milwaukee. (Boston, too, has a residency requirement for city employees, but it excludes teachers.) Any others out there we should be aware of?

From the Chicago Tribune (9/11/2009):

The city, for its part, maintains that teachers should be contributing to the tax base that funds their schools, among other reasons.

From the Milwaukee Journal-Sentinel (1/24/2008):

The residency rule has been controversial for years. Some say it is unfair and MPS needs good teachers too much to restrict the pool of possible teachers. Others say it doesn't actually have much effect on who teaches overall and it's good for the city to have employees live within the city line. Efforts in the state Legislature to repeal the residency rule recently have not succeeded.

0 Making it Easier to Get Aid

Efforts are underway to simplify the complex, byzantine system we've created to administer financial aid, and it's about time. Streamlining the process holds promise-- take a look at the recent H&R Block study if you have any doubt. Where policymakers are starting, by reducing the complexity of the application, is a good place to begin, but we could do more. There are some basic facts about individual decision-making which are neglected in the design of the current system-- and remedying those oversights could go a long way towards enhancing participation.

I don't want to argue over who figured out first that humans aren't highly rational beings. Maybe it was the behavioral economists, maybe the psychologists, maybe the sociologists. In any case, it's clear that we tend towards inertia, confusion over too many choices, and that we're highly influenced by what those around us do. And as Richard Thaler and Cass Sunstein nicely articulate, the right kind of policy "nudge" can get us moving in a better direction.

In today's New York Times, Thaler makes a strong case for changing the organ donation system from an "opt in" model to an "opt out" one. He cites evidence that in countries where people have to go out of their way to decline to donate their organs, a far greater percentage choose to donate. For example, in Germany, the opt-in system results in a positive donor consent rate of just 12%-- in Austria, the opt-out system has the opposite effect-- 99% of adults donate.

Thaler also highlights the benefits of a system where some kind of choice is required-- where it's impossible to skip the step of making a decision. So in Illinois, you have to decide whether or not to donate when you signup for your driver's license-- and 60% of people decide "yes."

The benefits are clear-- plenty of people need organs, and this is a way to make sure more get them,without resorting to a scary open-market system where we need people to sell and buy organs.

I think this could work for financial aid. Right now we have a system that leaves many eligible students out-- simply because they do not fill out the required form-- the FAFSA. We can fix this. Instead of having students submit a separate financial aid application in addition to their college application, why not require it? If students want to decline to apply, make that the decision that needs to be reached-- rather than the other way around. Right now you have students being admitted to college who forgot to apply for aid-- let's flip that, and make sure the processes are integrated, and the only ones not considered for aid are the ones who took actions not to be considered.

This model would require ramping up aid counseling, for sure, to make certain that students who wanted to apply could figure out the process. Hopefully, with the simplification of the FAFSA pending, this will become less burdensome. But it ought to pay off in the end-- getting more eligible students access to the dollars they are entitled to. Money left on the table is money wasted-- and if we can identify a fairly simple, elegant solution to the problem then heck, shouldn't we try?

0 Musical Elective of the Month

The Musical Elective of the Month is 7 Worlds Collide: The Sun Came Out.

Neil Finn is one of the great, under-appreciated singer/songwriters of the last 30 years. A New Zealand native, Neil came of age and then into the musical forefront through the one-of-a-kind New Wave band Split Enz, founded and fronted by his older brother Tim. He penned and sang the band's biggest #1 hit (in Canada, Australia and New Zealand), 1980's "I Got You." He went on to form Crowded House (1985-1996), fashioning it into an internationally renowned band (re-formed in 2006) and a frequent visitor to critic's 'best of' lists, with hits including "Don't Dream It's Over," "Something So Strong," "Weather With You," and "Distant Sun." And Neil has recorded solo albums in 1998 and 2002 and two albums, in 1995 and 2004, with his brother under the moniker The Finn Brothers.

It has been eight years since Neil's first 7 Worlds Collide project which gathered a stellar group of musicians (including Eddie Vedder, Johnny Marr, and members of Radiohead) to play a series of live shows in Auckland, New Zealand. All proceeds from the resulting live album and DVD went to Medecins Sans Frontieres. Around Christmas 2008, Neil assembled an even larger group of musicians -- with international development organization Oxfam chosen as the beneficiary this time -- to collaborate and record a studio album of original songs. All of the songs were written and recorded over the course of three short weeks. Pretty amazing.

To be released on September 29, 2009 in the United States, the two-disc 7 Worlds Collide: The Sun Came Out features: the Finn clan (Neil, Tim, sons Liam (solo artist) and Elroy (drummer for The Tricks), and wife Sharon); Johnny Marr (The Smiths, Modest Mouse): Ed O'Brien and Phil Selway (of Radiohead): Jeff Tweedy, Glenn Kotche, Pat Sansone and John Stirratt (of Wilco); KT Tunstall; Lisa Germano; Don McGlashan; Glenn Richards (of Augie March); Bic Runga; Sebastian Steinberg (of Soul Coughing), Johnny's son Nile Marr (AKA Kid 4077), and Jeff's son Spencer Tweedy.

There are some real gems on here and fans of good music and any of the artists represented should check this out. The album features tracks such as "True Blue" co-written by Johnny Marr and Jeff Tweedy, "Hazel Black" co-written by KT Tunstall and Neil Finn, and "The Ties That Bind Us" written (and sung) by Radiohead drummer Phil Selway.

Order through amazon.com here and support Oxfam. Music samples are available now on the album's web site and you can check out some of the limited live performances from January 2009 on YouTube.

When your seven worlds collide
Whenever I am by your side
And dust from a distant sun
Will shower over everyone
--Distant Sun," Crowded House, Together Alone (1993)

Visit here for past Musical Electives, including Guster, Gomez, and Travis.

0 Research: Attracting New Teachers to Urban Schools

New research led by Tony Milanowski of the University of Wisconsin-Madison provides more evidence that increasing teacher pay may not be the best approach to attract new teachers to high-need, hard-to-staff urban schools. A key finding of the study -- published in the International Journal of Education Policy and Leadership -- which explored job factors important to pre-service educators was that "working conditions factors, especially principal support, had more influence on simulated job choice than pay level."

'Policy implications' include:

• "[M]oney might be better spent to attract, retain, or train better principals than to provide higher beginning salaries to teachers in schools with high-poverty or a high proportion of students of color."
• "[I]nduction programs and curricular flexibility are important to new teachers. The finding that induction programs are attractive, combined with evidence that such programs can be
  effective in reducing teacher turnover (e.g., Ingersoll and Kralick, 2004; Smith and Ingersoll, 2004), suggests that urban districts may want to implement high-qualityinduction and mentoring programs, especially for new teachers in schools with high proportions of poor students or students of color."

0 Superteacher To The Rescue!

Given the recent spate of federally-funded studies showing no effect of a variety of educational innovations and interventions, my predicted answer to the question ('Can Teachers' Talent Translate Elsewhere?') posed in this Houston Chronicle story is "no."

I worry, however, that the basic premise of the federally funded Talent Transfer Initiative is faulty and builds upon the notion of teaching (as reinforced by popular culture) as an individual rather than as a collective pursuit. Can 'superteachers' walk into dysfunctional school cultures and work magic that can result in a quantifiable impact on student learning? Some surely can. (It's too bad we can't clone Jamie Escalante and Frank McCourt, isn't it?) More important to ask is, should we expect them to?

What is more desperately needed than an expensive scheme to redistribute 'superteachers' is a serious attention to teaching and learning conditions. My New Teacher Center colleague, Eric Hirsch, spearheads assessment of school culture and the training of school administrators to more effectively shape it. His and independent research (here and here) has identified that teacher effectiveness is facilitated by a positive school context, including support from leadership, the existence of a collaborative working environment, and time for professional learning.

It doesn't appear that the Talent Transfer Initiative envisions teaching and learning conditions as part of the solution, and that's terribly unfortunate. I wonder if the TTI is even collecting such data to investigate the relationship between these variables and teacher success, or lack thereof? Until we address these contextual issues in low-performing and hard-to-staff schools, we're not going to get the results that we expect and students deserve.

UPDATE (9:35 p.m.) -- Claus von Zastrow offers an excellent blog post on Public School Insights about this study as well.

0 House Passes Historic Community College Legislation!

Today the U.S. House of Representatives voted 253 to 171 to pass the Student Aid and Fiscal Responsibility Act. It knocks private lenders out of the student loan business, and uses the savings to make transformational investments in the nation's community colleges, as well as increase the Pell grant. Some, including La Guardia Community College president Gail Mellow, have called this the most important piece of higher education legislation since the G.I. Bill.

Let's hope the Senate soon follows on the House's class act!
(at least the House had at least 1 class act this week...)

0 Community College Legislation Moves Forward

A quick update on the proposed legislation affecting community colleges-- HR 3221. Today, an online JAM in support of the American Graduation Initiative took place, drawing a virtual crowd of around 400 folks. Lots of good discussion happened, especially about issues of how to make the competitive grant process work well without leaving the more disadvantaged colleges behind, and questions about that required match for the construction funds. Even more exciting, the House of Representatives is poised to take a full floor vote tomorrow, after the rule governing floor debate passed, 241-149. The Senate is expected to begin taking action next week. Stay tuned...

0 I'm hiring (again)!

Doug Harris and I are seeking a Project Manager for our ongoing study of financial aid, the Wisconsin Scholars Longitudinal Study.

This person will help us with administrative tasks (e.g. handling travel and meeting planning) but also become involved in the daily work of the interviewing portion of the study, and have a chance to learn alot about the research process.

The position is based at the University of Wisconsin-Madison, and you should learn more and apply by clicking here.

Thanks!

Sara

0 The Perils of Group-Think

With the President imploring more Americans to go to college and not give up on themselves or their nation, it's understandable that many policymakers want to join in and do their part. Making college more affordable is a very good place to start.

And free tuition-- heck, free anything--sounds great, especially in a recession. Free tuition provided without changes to the state's operating budget or cuts to any other programs, or by digging a deeper deficit, sounds even better. That's probably why Michigan state representative Fred Durhal Jr. thinks he's got a good thing going, proposing to use a new lottery and casino profits to provide free tuition to more than 160,000 students. It's a win-win right? Says Durhal, "You can feel a little better about losing money to the house if you know it's going to go to children."

Except that the money you are losing is being taken from your own children, and given to someone else's. And, since only students who've lived in Michigan continuously for 5 years and earned a 2.5 GPA or better are eligible, the person losing the money is far more likely to be relatively poor, while the kids getting the money are likely to be relatively rich. Let's think-- who moves around a lot and is far less likely to meet that GPA requirement...? Hmmm...

A basic tenet of good social policy-making is targeting. If you want to help students who right now can't afford college, then you need to set the requirements such that they are the (only) ones who are eligible. Not doing so means spreading the wealth among those who don't really need it, diluting the potential impact. And if you really want to help disadvantaged families then don't use a funding mechanism that draws the cash from the program right out of their pockets.

Michigan can look to many other states for better ideas, and for lessons on why this one is a poor one. Take a look at who has benefited from Georgia's lottery-funded HOPE scholarship, which also includes a GPA cutoff. According to expert Don Heller, over 90% of the expenditures have gone to students who would've attended college even without the financial assistance. As long as a scholarship is tied to a GPA cutoff, and family income is correlated with GPAs, then such programs won't effectively reach their intended audience. As long as poor folk dominate those playing the lotto and visiting the casinos, then that funding mechanism doesn't work either.

I'm as pleased as punch that people want to help more poor kids afford college. That's an idea worth glomming on to. But policy proposals require more than good ideas, they require carefully thought out details and a strong theory of action. This one just doesn't cut it.

0 Premature Conclusions: More Money, No More Grads?

Some members of the media are covering the release of a new Canadian study, associated with the Educational Policy Institute, that examines the effects of a financial aid program on college-going and completion among low-income students. Researchers at the Measuring the Effectiveness of Study Aid Project tried to isolate those effects by examining what happened following a change in student aid policy in Quebec that increase aid eligibility and decreased reliance on loans. By comparing student outcomes both before and after the policy change, and comparing the outcomes of similar student in Quebec to those in other provinces (where such reforms did not occur), analysts attempted to establish a causal effect of aid.

They conclude that the policy affected access (increasing overall enrollment among students from families making less than $20K per year by 4-6 percentage points), and persistence (increasing retention rates by 6 percentage points) but did not affect graduation rates--at least within the 4-year window of time during which graduation was measured.

While noting that the null findings may stem from that short period of observation, the researcher still goes on record with this conclusion: "These results therefore cast doubt on the efficacy of this reform in particular, and of needs-based grants in general, to improve graduation rates."The headline over at Inside Higher Ed reads "More Money Doesn't Equal More Graduates."

This is a distinctly premature and irresponsible conclusion. First, as one of my graduate assistants James Benson pointed out, "if the percentage of college-eligible students that enrolled in college increased by 5 percent, and the persistence and graduation rates remained entirely static, then the program produced a net gain in the proportion of young adults completing semesters and degrees."

Furthermore, there are many reasons why an effect might not be estimated properly in this study. As my colleagues Doug Harris, Phil Trostel, and I explained in a recent paper, a simple correlation between aid receipt and college success is likely to be negative because students from low-income families, in the absence of aid, are for a variety of reasons less likely to succeed. Unless researchers can convincingly account for all of those reasons – and we argue that very few do – the estimated effects of aid are likely to look smaller than they really are. This study is not very convincing and really doesn't move far beyond a correlation, for many reasons. For example, as another graduate assistant, Robert Kelchen, indicates:

1. The comparison groups (Quebec vs. other provinces) have very different rates of financial aid take-up prior to the reform. This calls the validity of the comparison into question. It's also too bad the researcher didn't see fit to post his tables on the website, since we cannot see whether the differences post-treatment are significant.

2. Quebec saw increases in the enrollment rates of high-income students following the reform, in addition to increases in the enrollment rates of low-income students. If financial aid was the real driver, it shouldn't have affected the (ineligible) high-income students.

These are but a few examples-- if a full research paper (such as would be submitted for academic review) was available, I bet we'd have more concerns.

This is a case of the press jumping the gun and running with a story, and a headline, not supported by the empirical work done by the researchers. We're in a recession, and aid programs cost a lot of money. We do need to know if they work, and in particular if they are cost-effective. But the estimation of impacts should be done more carefully, and results discussed in a much more responsible manner. Sexy but un-informed headlines will do little good-- perhaps even casting a shadow on an effective program, reducing its ability to maintain funding. All of us studying financial aid have an obligation to do much, much better.

0 Abandon All Hope (For Reform) Ye Who Enter Here!

At first glance, one might dismiss a recent policy brief authored by a former Bush Administration official as a partisan diatribe against the American Recovery and Reinvestment Act (ARRA) and the Obama Administration. After all, a chief conclusion of the brief authored for the American Enterprise Institute by Andy Smarick (former Deputy Assistant Secretary in the Spellings-era Education Department and in 'W's White House with the Domestic Policy Council), is: "It appears all but certain that the ARRA’s $75 billion in formula-based education programs are a lost cause for education reform. These funds have been used almost exclusively to fill budget holes, and cash-strapped states and districts will likely use what remains of these funds for similar, reform-averse purposes."

Abandon all hope (for reform) ye who enter here!

That quoted summary language in the paper *is* perhaps a bit over the top. A "lost cause"? Really? And that's certainly been the takeaway of some blog accounts of this paper (such as this). But that's not really what Smarick is saying nor is it the most important part of this AEI brief. And, as much as he is making that point, his 'lacking in reform' criticism is directed more at the 50 states than at the federal government.

Economic stimulus and a minimization of a short-term funding cliff were among the main aims of ARRA and its education-focused formula dollars. I don't think anyone seriously expected differently. If you read the ARRA web page, it largely spells this out. Now, the Education Department did envision that State Fiscal Stabilization Funding would be used to promote reform as well, and despite an initial look by the GAO, some dollars may accomplish reform, but how on earth could there yet be any real evidence of reform let alone impact when the 2009-10 school year has just begun in most places?!? In addition, as Smarick notes, the economic downturn and its effect on state budgets was far worse than anticipated at the time that ARRA was enacted in early 2009, which lessened the likelihood of these dollars doing anything less than filling holes.

Smarick's take on the competitive aspects of ARRA -- the Race to the Top and the Investing in Innovation (I3) funds -- is generally fair and balanced. He raises important questions about the general risks to any reform push, and specifically to ARRA. Smarick identifies several factors that may reduce the likelihood that competitive dollars will further education reform: on-going state budgetary challenges, resistance to specific reform components, and lack of faithful and vigorous implementation. He warns of "Trojan horse" applications where states will seek the money, but won't use it for reform. Of course, unmentioned are a whole host of other potential roadblocks, such as resistance from school districts, lack of buy-in from teachers and school administrators, lack of capacity to implement reforms, consultants and subcontractors who can't deliver promised expertise or technical assistance, data systems that cannot accurately match student and teacher data, etc.

Read the brief. Or check out a summary at Flypaper.