The CDE's August SARC newsletter carried news of an odd addition to the SARC: a comparison of California's fourth and eighth graders' results on the National Assessment of Educational Progress (NAEP) with those of their peers in the country. You may know this test as the nation's report card. It is the only test that enables us to see how students in each state compare with students in every other state. This is no doubt useful to researchers, legislators, and business people. But why is it in the SARC?
This simple question reveals a tale of twisted tongues and puzzling policies. It reveals why the SARC has become a textbook example of why one should refrain from comparing apples with oranges. It also reveals how a good and simple idea—annual reports that show how schools are doing—has been stretched and strained beyond recognition.
Should this stop you from communicating the facts that you want your public to see? Of course not. The SARC is not a prison for information. It was never intended to limit you from communicating more. It simply defines the irreducible minimum amount of information you must share with your staff, parents, and community. If you want your public to be more aware of the impact of cuts on their kids' schools, simply open up the SARC and put those facts and your interpretation wherever you like.
TWISTED TONGUES
The results of NAEP give us one way to see the relative value of each state's measures of proficiency. All states use the word "proficiency"—they have to. Since the passage of No Child Left Behind in 2001, each state has had to use its own tests to measure how well students are learning their own state's content standards. The states are free to decide what their students should know and be able to do at each grade level. They are equally free to draw a line separating those kids they believe to be "proficient" from those who are not.
No surprise, the states draw that line at very different points. In effect, California happens to draw it relatively high, because we have high content standards. Our tests, then, are tougher than most at each grade level, so a California fifth grader who scored at the basic level in math is likely to do better on the state tests if he moved to Mississippi.
The value of a dollar in San Francisco is lower than the value of a dollar in Selma, Mississippi. No one expects a dollar to be worth the same thing in these two places. So it goes with test results. NAEP simply gives you one way to view the relative value of test scores from state to state.
PUZZLING POLICIES
The NAEP results are important, no doubt, but they belong in the newspaper, in research papers, and in policy reviews. The NAEP results certainly belong in the state’s highest-level accountability report, but the NAEP results do not belong in the SARC because the SARC is a school accountability report card.
The CDE higher-ups tried to resist the U.S. Department of Education’s policy to push NAEP results into the SARC. Despite their opposition, the federal officials prevailed.
NAEP IS NOT THE ONLY ILLOGICAL ELEMENT OF THE SARC
The federal policy that led all state education officials to include NAEP results is hardly the first policy that eroded the integrity of this school annual report, nor is it the worst of the bunch. In fact, it only dilutes the report's value by adding two pages to its length and half a page to the executive summary. Sadly, the SARC has suffered from many attacks of illogic over its 21 years of life. Consider just three instances of comparing apples with oranges.
| 1. |
Comparing elementary schools' student results with students in grades 2 through 11. This flaw was built into the SARC at the beginning and has never been corrected. If you were an elementary school principal, you might like this, since middle and high school students’ scores in English/language arts and math are generally lower, but it pollutes what would otherwise be clearer and more fair: comparing elementary school students only with others at the same grade level.
|
| 2. |
Comparing the scores of students who take more rigorous courses with those who take easier courses. The SARC takes all students’ math scores, mixes them together, and pours out those students who score Proficient or Advanced. That's the percentage that is reported in the SARC. If your middle school puts all eighth graders into algebra, and another middle school puts only 10 percent of its eighth graders into algebra, the SARC doesn't care. It compares them anyway, in effect punishing schools that pursue more rigorous course work for their students. (The same punishment is dished out to high schools whose students pursue tougher math and science course work.)
|
| 3. |
Comparing the scores of students in a school with the average scores of students in the district. This is most flawed for the 564 districts with four schools or less. It is flawed in three ways. First, the unit of comparison is too small to be meaningful. The larger the benchmark, the more valid it is as a comparative yardstick. Second, the school itself is included in the district comparison. If three elementary schools, each with 300 kids, make up a small district, then each school makes up one-third of the comparison group. It should be excluded from the comparison group. Third, in a unified district, schools are compared with schools that are not at the same grade level. Why would you compare elementary students with middle school students, or even worse, with high school students?
|
Could the SARC benefit from a more reasonable and logical approach? Yes. Could the SARC benefit from offloading some dead weight? Certainly. NAEP adds nothing to the reader’s understanding of the school or the district. Could the SARC be more useful to more people if it only compared apples with apples? Sure.
The good news is that you are free to put all three of these improvements into practice. Here's our advice about how you might do just that.
RECOMMENDATIONS
| 1. |
Sequence the report your way. Personalize the SARC by deciding what matters most. Relegate the things that matter less to the back of the report. This is a traditional publishing convention, and entirely consistent with CDE’s rules and regulations.
|
| 2. |
Calculate school-level comparisons on the CSTs. You can filter the CST results of elementary schools, middle schools, and high schools so you can compare each school's results at the right school level. This would enable you to avoid comparing elementary school students’ math results to the results of students in grades 2-11, for example. (It’s not easy, but it’s possible. If you want help, call us for guidance at 415 337-7971.)
|
| 3. |
Create a one-page summary report that's aligned to your district's goals. You are free to improve upon the executive summary that sits at the front of the CDE's SARC template. If you align this important summary to your district's LEA plan and your board’s strategic plan, you’re bound to make both parents and leaders happy.
|
REFERENCES
The CDE has published the 2008-2009 SARC template in Microsoft Word format.
The CDE’s Frequently Asked Questions section explains that districts are free to walk away from the SARC template and publish the data in their own way.