Special thanks to Ze'ev Wurman for sharing his observations on Dr. William Schmidt's presentation last week. In an event co-sponsored by Achieve, Chiefs for Change and the Foundation for Excellence in Education, Dr. Schmidt presented a briefing on his work: Common Core State Standards Math: The Relationship Between High Standards, Systemic Implementation and Student Achievement.
I'd like to share this information posted by Ze'ev last August, in order to highlight his work experiences and expertise in ed policy as well as the technology sector.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Last week, Bill Schmidt, of Michigan State University, held a highly publicized national press event, the “key conclusions” from his recent research. We can’t see any of the underlying research, as Schmidt did not publish it; its supposed findings, however, already got so much uncritical exposure and praise that it is important to put Schmidt’s words in context. That context seems more problematic than organizations like Achieve, or Chiefs for Change, who sponsored this research, would like us to believe. I have reviewed Schmidt’s presentation and these are some of my observations.
1) First, we should note how carefully Schmidt hedges his bets. His first (and last) slide says that the Common Core Standards for Mathematics "[c]an potentially elevate the academic performance of America's student," with the emphasis on the “potentially” in the original. It is hard to imagine a more sweeping disclaimer—almost anything can "potentially" elevate academic performance. More money; more professional development; more unionization; more school choice; more selectivity in choosing teachers; better textbooks; better parent education via public campaigns; better movies from Hollywood that will improve character education and discipline of youth; and so on.
2) Schmidt repeats in multiple slides that parents and teachers support them and claim to be familiar with them. A large fraction of teachers even believes it is prepared to teach them. I can believe that teachers have heard about them, but I doubt many have any real basis for liking them, or for claiming to be prepared to teach them. Other surveys found most teachers and parents don't really know or understand the actual content of the standards and the implications of teaching them. After teachers actually try teaching them in the classroom and we see the assessments, maybe we could put more trust in these surveys.
3) On slide #2 Schmidt, gives the old picture (from TIMSS '95 and '99) of topics progressions based on the so-called "A+" countries (TIMSS very high achievers). In the next slide (#3) he shows state "averages" generally following this pattern (due to his own influence a decade ago, to some degree). But then, on slide #4, he shows the Common Core mapping. It looks, at first glance, similar to the previous overall shape. Upon closer examination, however, we see that the order of the topics has changed, and a few new ones show up!
This sleight of hand is doubly troublesome because Schmidt, during the press conference, repeatedly referred to “looking at the pattern” that emerges and the “coherence” it implies. He never mentioned anything about rearranging old topics or adding new topics for the Common Core categorization. Clearly, the “coherent pattern” that is supposed to emerge is completely different if one re-arranges the sequence and the nature of the topics.
To observe how misleading—and, frequently, simply wrong—Schmidt is, one should closely compare slides #2 and #5. For example, the “3D Geometry” topic is taught in the A+ countries in grade 7 and 8. In contrast, Common Core teaches this topic starting in grade 1(!) and until grade 8, with a two-year break in grades 3 and 4. To hide this, Schmidt moved this topic from the bottom in slide #2 to close to the top in slide #5. With respect to “Measurement Estimation & Errors” and “Number Theory,” not only were these topics significantly moved between slides #2 and #5 to hide the fact that Common Core starts teaching them in grade 2 (while the A+ countries teach them only in grades 7 or 8), but also slide #5 wrongly (and misleadingly) marks the A+ countries as if they teach them starting already in grades 4-5, further minimizing the visual difference between them.
To convince yourself simply check those same topics on slide #2. Another example is “Functions” that are handled by the A+ countries in grade 8, while Common Core topic (called “Patterns, Relations, & Functions” in slide #5) teaches them starting in grade 4 for a full five years, up to grade 8. There are many more examples of such mislabeling between slides #2 and #5, such as “Properties of Whole Number Operations,” “Fractions,” “Percentages,” and more. All this does not speak kindly of the focus and coherence of the Common Core, and does not show a progression closely similar to that of the A+ countries.
Specifically, here is what Schmidt says (minute 14 of the press conference):
"Using the very same TIMSS methodology of coding we took the documents from the Common Core and coded those. And I just want to make a one really important point, all of which is saying it is not somebody’s opinion whether the Common Core’s good or bad or indifferent. They really simply were graduate students given the job of coding and finding out what topics were taught at what grade levels. … Here is the graph that represents the CCSSM and you can see a very similar sort of shape to this to what you saw with the earlier slide with the A+ or the top achieving countries so this alone suggests a great deal of conformity between the two sets of standards."
Note his misleading use of “very same TIMSS methodology.” This means that the methodology of assigning standards to topics is the same, but not the order of the topics, or the topics themselves. Yet it is clearly his implication that they are identical when he urges us to “see a very similar sort of shape to this to what you saw with the earlier slide with the A+ countries.”
What Schmidt is doing here borders on the dishonest. He switches the underlying topics and their order and then urges us to watch “a very similar” pattern, never mentioning that the pattern represents completely different underlying topics on the different charts—just as a magician makes sure the audience watches his face and not his hands.
If Schmidt’s misleading way of reorganizing topics and mislabeling what A+ countries do was not enough, one cannot even trust his underlying categorization of the standards. For example, he marks “Constructions Using Straightedge and Compass” as taught in grade 7 by the Common Core. Yet the Common Core clearly does not teach it before high school geometry. In grade seven, the Common Core only expects students to draw shapes with ruler and protractor, not with straightedge and compass. This major difference between informal and formal geometrical constructions somehow escaped those “graduate students” who coded the standards “using the very same TIMSS methodology.”
4) In slide #5 we see that Schmidt, even with the tabulating errors, finds about 15% (18 of ~130) of the A+ standards in different grades than Common Core, sometimes two grades apart; some of the Common Core topics are not in A+ countries and vice versa. But not all standards are made equal—not every topic is as important as the next, as any mathematician will easily tell you. Except that Bill Schmidt, a statistician, does not go there—he’s after simplistic statistical correlation of badly classified standards.
5) An even bigger issue with the new list of categories is that we don’t know how these categories were put in place, and we still have no idea of the relative importance of topics. Was the categorization created long ago as the result of Schmidt’s research on curriculum over the last decade, or was it custom made for the Common Core? If it’s the latter, the whole comparison is meaningless and serves effectively only as an advocacy research. It is easy to “tune” the categories so that they will show excellent alignment between standards of high-achieving countries and Common Core, or show a pseudo-coherent progression pattern by rearranging the topics. I don’t know the answer to that, but at his talk Schmidt did not mention what caused him to make the changes, or when.
6) Slide 6 shows a number-of-topics table, and other than observing general reduction among the states between 1995 and now, there is little to say -- it doesn't show that Common Core is significantly "slimmer" than the state average, and certainly doesn't show that the Common Core is slimmer than any given state. All this -- even if we were sure that reducing number of topics is of cardinal importance -- which we are not![1] One can easily accept that having a huge number of topics per grade is a problem, but at the same time once you get to below 20-25 it is quite unclear that 21 is better than 25 and worse than 17. Could be just the opposite! This slide is yet another example of Schmidt's statistical leanings rather than his understanding of content -- it pretends to say something meaningful yet throws mostly meaningless data up in the air.
7) Slide #14 is probably the most indicative of the overall weakness of this whole story. Schmidt’s research is supposed to show “alignment” between Common Core and standards of high-achieving A+ nations, and hence “conclude” that Common Core will lead us to higher achievement because of that alignment. Yet here we see California at the top of the scale being Common Core-like, and Massachusetts being somewhere in the middle of the pack. One does not need a PhD in statistics, however, to realize that Massachusetts had made extraordinary progress with their “mediocre” (by his measure) standards, while California made only a mediocre progress with its “exceptionally-aligned” standards. This, yet again, brings up Tom Loveless’s recent argument that the correlations (standards=>achievement) are small and the causality argument is highly problematic.
Schmidt tries to address this visible problem in his story by developing a new measure of congruence that adds the rigor of state’s cut-scores, arguing that, in some mysterious way, cut-scores are reflective of the quality of the standards themselves. With this correction, Massachusetts suddenly moves up on his scale of congruency, and this is supposed to explain why it achieves so well. Yet adding cut-scores to a measure of the quality of the standards is unsupported by logic or reason. Cut-scores may indicate the seriousness that the state attaches to student achievement, its level of expectations from them, and possibly its quality of curriculum implementation. But cut-scores have little to do with the quality of the standards themselves. This seems yet another example of creating a fancy statistical “measure of congruence” to “prove” a statistician’s point, while hiding the fact this measure has neither educational nor policy logic behind it.
(As an aside, it is interesting that Schmidt studiously avoids mentioning cut-scores of the Common Core; we don’t know how high—or low—they will be set, and this is potentially a huge area of contention; it’s hard to see how Massachusetts and Mississippi will agree on common cut scores.)
To conclude, Bill Schmidt centers his argument around two themes: that the Common Core standards are similar to those of the A+ countries; and that states with standards more congruent to the A+ countries show bigger progress on the NAEP. To make the last claim work, Schmidt redefines “congruency” to include cut-scores for no logical reason. Both claims are unsupported by his own data and, in addition, his own data is riddled with errors. Yet the Chiefs for Change already tout that “Dr. Schmidt’s research shows that state leaders are on the right track. Common Core State Standards have the potential to raise student learning and performance across America. Most importantly, they are competitive with the standards found in the highest achieving countries.” What Dr. Schmidt presented is just another piece of misleading advocacy research, brought to you and paid for by the Bill & Melinda Gates Foundation and channeled through the friendly services of Achieve (which received a recent $375K grant for advocacy from the Gates Foundation), the Foundation for Excellence in Education (which received a recent $1M grant for advocacy from the Gates Foundation), CCSSO (which received $9.5M last year from the Gates Foundation to promote the Common Core), and Chiefs for Change (funded by the Foundation for Excellence in Education).
Ze'ev Wurman
