Bias in Texas Growth Measure and Implications for Austin and IDEA Charter

Posted on December 18, 2012

1



One of the often over-looked factors that influence achievement are peer effects. There is a growing body of literature that concludes peer effects have fairly substantial effects on individual achievement. However, it is incredibly difficult to disentangle peer effects from all the other factors influencing student achievement. (1) The preponderance of the research is pretty clear in this area–students placed in classrooms with peers with greater achievement will likely benefit from positive peer effects while those placed in classrooms with students of lower achievement will likely suffer from negative peer effects. Moreover, lower performing students benefit more from positive peer effects than other students while higher performing students suffer more from negative peer effects than other students.

This plays out in assessments of teachers and schools when judgments are made without controlling for the incoming ability of students. And, even when incoming ability is controlled for in the statistical analysis, peer effects are often so difficult to capture in a statistical model that the results end up being biased with respect to incoming ability. Bias simply means that the end result of the analysis is still associated with some characteristic of the teacher or school not under the control of the teacher or school personnel. For a great example on this, see Bruce Baker’s blog at: http://schoolfinance101.wordpress.com/2012/12/12/its-good-to-be-king-more-misguided-rhetoric-on-the-ny-state-eval-system/

Let’s look at the state’s own Financial Allocation Study for Texas (FAST) that is being used by many of the state’s witnesses in the current school finance trial. The student growth model is pretty darn sophisticated and created by some top-notch people in the field. Certainly far better than I could do. However, the end results are biased in certain ways. The greater the percentage of economically disadvantaged students, the lower the growth. More disturbingly, the greater the prior scores of students entering middle and high schools, the greater the growth. Both of these factors are out of the control of schools.

As shown in Figure 1, the greater the 5th grade TAKS math scores of incoming 6th grade students, the greater the FAST growth measure. In other words, schools with incoming students that have higher test scores are more associated with greater student growth on the TAKS tests. This is true even after controlling for a large number of individual student factors and some school factors (See http://www.fastexas.org/overview/methodology.php).

Figure 1: Relationship between the 5th Grade TAKS Mathematics Scores of Incoming 6th grade Students (2008, 2009, and 2010) and Average School-Level Student Growth on TAKS Mathematics Test (2008, 2009, 2010)

Incoming TAKS math and FAST growth TAKS math

So, all other factors being equal, schools are more likely to exhibit greater growth rates if high achieving students enter the school. While other factors such as the non-random distribution of effective teachers or principals may explain this phenomenon, the academic achievement of incoming students would increase the likelihood that a school could recruit and retain effective teachers. Thus, peer effects and the influence that having higher performing students in the school has on recruitment and retention efforts are likely causes–and these causes are out of the control of educators unless there is some recruitment or application mechanism such as those in charter schools or magnet schools.

Table 1 below documents the 5th grade TAKS mathematics achievement of students entering the 6th grade for Austin ISD and IDEA charter schools. As one can see, certain schools tend to enroll higher performing students while other schools enroll lower performing students. What is surprising is that even though IDEA schools are located in the Rio Grande Valley and enroll very high percentages of poor and minority students, students entering IDEA middle schools look very similar to the Austin middle schools on the west side of town that enroll large percentages of White and not economically disadvantaged students. I find it not very surprising that students in IDEA charter schools have higher performance than Austin middle schools on the east side that serve poor, minority, and lower-performing students (for non-Austinites, these schools are essentially all those below IDEA Alamo.

Table 1: Fifth Grade TAKS Mathematics Performance of Students Entering the 6th Grade for Austin ISD and IDEA Charter Middle Schools (Incoming Classes of 2010 and 2011)

AISD and IDEA incoming 6th gradersPolicymakers need to start paying attention to these finer-grained details about the characteristics of students entering schools and how the composition of the student body affects the academic culture of the school and the detrimental effects of segregating kids based on race/ethnicity, poverty, and academic performance. Many of the Austin school baord members are very concerned about the trend to segregate kids based on academic performance and they are right to be concerned. The current trend in education policy is to simply segregate kids on past academic performance and parental involvement and then simply write off the kids who with low prior achievement and low parental involvement. This is simply an unethical and unacceptable approach. This country is supposedly a meritocracy, but that certainly is not how the current band of education reformers are operating.

NOTE: The Austin school board voted to terminate the contract with IDEA based, in part, on data that showed the charter continually lied and mis-led policymakers and the public about the academic outcomes of the charter organization.

(1) This is one reason why reliance on value-added methodology is so perilous. Systems that do not account for peer effects at the classroom level may be assigning positive or negative effects to a teacher when, in fact, peer effects were the real influence.

Advertisements
Posted in: Uncategorized