How I will evaluate CAASPP Data

Let’s face it. The public data we get back on the CAASPP tests is awful.

I don’t mean the numbers are dismal and depressing. That is probably true too. But the actual type of data that we get tells us very little about how schools are helping students.

The primary data that the California Department of Education releases has to do with the percent who “met” or “exceeded” standards on the test. This is a measure of proficiency, not growth. Proficiency is a very poor measure of how a school is performing because it is not measuring value added. Schools should be measured primarily on how much students improve, rather than proficiency.

So I am going to bend the data to make it into a growth measure.

Here we have two schools: Luther Burbank Middle School (my school) and Endeavor College Prep. Both schools have 7th grades. Both schools have 8th grades. Therefore, we can look at the 2016 data for the 7th graders and the 2017 data for the 8th graders and see how thosestudents improved or regressed.

The scores here are a little confusing so let me explain: The numbers you see are how many scale score points away from “meeting standards” the students are. This measure is called the “distance from 3” and is what is used in the new California school rating system.

This means that at Endeavor, the average 7th grade student was 21.6 points under meeting standards, while at Burbank, the average 7th grade student was 57.3 points away. In other words, Endeavor students finished 7th grade in 2016 far ahead of Burbank Students. 

However, by 2017, the Burbank students (now 8th graders) had caught up, improving to 40.7 points away from proficiency. Meanwhile, the endeavor students slipped to 37.1 points away from meeting standards.

The Endeavor students are still scoring higher, but when you look at this data, it is clear which achievement is more impressive: One school improved student outcomes, while the other allowed students to fall further behind. 

In order to compare all schools against one another, I create a percentile for every school. Burbank’s 8th grade math department receives a score of 82, meaning that we outperformed 82% of LAUSD schools in student growth. Conversely, Endeavor received a score of 28, meaning they outperformed 28% of LAUSD schools in student growth.

There are several sources of error in this process. I recognize these errors, and yet I persist, because I am so hungry for a fairer way to evaluate schools. Here are a few of those errors:

  • The scale scores I am using are averages, so they do not really reflect individual student growth.If I had microdata on individual student achievement levels, we could do a lot more damage, but this is what we have.  Still, this is a decent proxy, and it will have to do.
  • The 7th grade test is different from the 8th grade test. It is problematic to compare student achievement on different tests. However, the common core standards are built on each other, so even though the tests are different, their content is sequential.

So, now that I have acknowledged the faults in my methodology,  lett’s start evaluating some student growth. Next week, I will have a nice dashboard with which to evaluate improvement from grades 4 through 8. Get ready….

4 Replies to “How I will evaluate CAASPP Data”

  1. I like the idea and look forward to next week’s analysis. Somewhat surprising CA doesn’t have individual student growth data where many other states (FL comes to mind..) do. How might you adjust for schools that are already in the 90th+ percentile and don’t have room to grow? It would seem hard for them to score well in these rankings..

  2. Another topic for you: how to evaluate HS SBAC scores when students aren’t tested in grades 9 and 10.

Leave a Reply