Historical Disconnect - Cut Scores and NAEP Achievement

The School Administrator June 2008 Number 6, Vol. 65

Cut Scores, NAEP achievement Levels and Their Discontents

The attempts of political bodies to bludgeon public schools with arbitrary performance standards

 

[selected excerpts]

 

In 1988, Congress created the National Assessment Governing Board and charged it with establishing standards. NAEP [then] became prescriptive, reporting not only what people did know but also laying claim to what they should know. The attempt to establish achievement levels in terms of the proportion of students at the basic, proficient and advanced levels failed.

The governing board hired a team of three well-known evaluators and psychometricians to evaluate the process — Daniel Stufflebeam of Western Michigan University, Richard Jaeger of the University of North Carolina at Greensboro and Michael Scriven of NOVA Southeastern University. The team delivered its final report on Aug. 23, 1991. This process does not work, the team averred, saying: “[T]he technical difficulties are extremely serious … these standards and the results obtained from them should under no circumstances be used as a baseline or benchmark … the procedures used in the exercise should under no circumstances be used as a model.”

NAGB, led by Chester E. Finn Jr., summarily fired the team, or at least tried to. Because the researchers already had delivered the final report, the contract required payment.

The inappropriate use of these levels continues today. The achievement levels have been rejected by the Government Accountability Office, the National Academy of Sciences, the National Academy of Education, the Center for Research in Evaluation, Student Standards and Testing and the Brookings Institution, as well as by individual psychometricians.

I have repeatedly observed that the NAEP results do not mesh with those from international comparisons. In the 1995 Trends in International Mathematics and Science Study, or TIMSS, assessment, American 4th graders finished third among 26 participating nations in science, but the NAEP science results from the same year stated that only 31 percent of them were proficient or better.

At the press conference announcing the [U.S. Chamber of Commerce/Center for American Progress jointly developed report “Leaders and Laggards” in February 2007], an incensed John Podesta, president and CEO of the Center for American Progress, declared: “It is unconscionable to me that there is not a single state in the country where a majority of 4th and 8th graders are proficient in math and reading.” He based his claim on the 2005 NAEP assessments.

Podesta could have saved himself some embarrassment had he read the recent study by Gary Phillips, formerly the acting commissioner of statistics at the National Center for Education Statistics. Phillips, now at the American Institutes for Research, had asked: “If students in other nations sat for NAEP assessments in reading, mathematics and science, how many of them would be proficient?”

Because we have scores for American students on NAEP and TIMSS and scores for students in other countries on TIMSS, it is possible to estimate the performance of other nations if their students took NAEP assessments.

How many of the 45 countries in TIMSS have a majority of their students proficient in reading? Zero, said Phillips. Sweden, the highest scoring nation, would show about one-third of its students proficient while the United States had 31 percent. In science, only two nations would have a majority of their students labeled proficient or better while six countries would cross that threshold in mathematics.

NAEP reports issued prior to the Bush administration noted that the commissioner of education statistics had declared the NAEP achievement levels usable only in a “developmental” way. That is, only until someone developed something better. But no one was or is working to develop anything better. When I wrote an op-ed piece for The Washington Post (“A Test Everyone Will Fail,” May 20, 2007), an indictment of the achievement levels, I got feedback that officials at the National Assessment Governing Board were quite satisfied with the levels as they are. That can only mean NAGB approves of the achievement levels used as sledgehammers to bludgeon public schools. They serve no other function.

 

Posted via email from concernedabouteducation's posterous