This plays out in assessments of teachers and schools when judgments are made without controlling for the incoming ability of students. And, even when incoming ability is controlled for in the statistical analysis, peer effects are often so difficult to capture in a statistical model that the results end up being biased with respect to incoming ability. Bias simply means that the end result of the analysis is still associated with some characteristic of the teacher or school not under the control of the teacher or school personnel. For a great example on this, see Bruce Baker’s blog at: http://schoolfinance101.wordpress.com/2012/12/12/its-good-to-be-king-more-misguided-rhetoric-on-the-ny-state-eval-system/

Let’s look at the state’s own Financial Allocation Study for Texas (FAST) that is being used by many of the state’s witnesses in the current school finance trial. The student growth model is pretty darn sophisticated and created by some top-notch people in the field. Certainly far better than I could do. However, the end results are biased in certain ways. The greater the percentage of economically disadvantaged students, the lower the growth. More disturbingly, the greater the prior scores of students entering middle and high schools, the greater the growth. Both of these factors are out of the control of schools.

As shown in Figure 1, the greater the 5th grade TAKS math scores of incoming 6th grade students, the greater the FAST growth measure. In other words, schools with incoming students that have higher test scores are more associated with greater student growth on the TAKS tests. This is true even after controlling for a large number of individual student factors and some school factors (See http://www.fastexas.org/overview/methodology.php).

Figure 1: Relationship between the 5th Grade TAKS Mathematics Scores of Incoming 6th grade Students (2008, 2009, and 2010) and Average School-Level Student Growth on TAKS Mathematics Test (2008, 2009, 2010)

So, all other factors being equal, schools are more likely to exhibit greater growth rates if high achieving students enter the school. While other factors such as the non-random distribution of effective teachers or principals may explain this phenomenon, the academic achievement of incoming students would increase the likelihood that a school could recruit and retain effective teachers. Thus, peer effects and the influence that having higher performing students in the school has on recruitment and retention efforts are likely causes–and these causes are out of the control of educators unless there is some recruitment or application mechanism such as those in charter schools or magnet schools.

Table 1 below documents the 5th grade TAKS mathematics achievement of students entering the 6th grade for Austin ISD and IDEA charter schools. As one can see, certain schools tend to enroll higher performing students while other schools enroll lower performing students. What is surprising is that even though IDEA schools are located in the Rio Grande Valley and enroll very high percentages of poor and minority students, students entering IDEA middle schools look very similar to the Austin middle schools on the west side of town that enroll large percentages of White and not economically disadvantaged students. I find it not very surprising that students in IDEA charter schools have higher performance than Austin middle schools on the east side that serve poor, minority, and lower-performing students (for non-Austinites, these schools are essentially all those below IDEA Alamo.

Table 1: Fifth Grade TAKS Mathematics Performance of Students Entering the 6th Grade for Austin ISD and IDEA Charter Middle Schools (Incoming Classes of 2010 and 2011)

Policymakers need to start paying attention to these finer-grained details about the characteristics of students entering schools and how the composition of the student body affects the academic culture of the school and the detrimental effects of segregating kids based on race/ethnicity, poverty, and academic performance. Many of the Austin school baord members are very concerned about the trend to segregate kids based on academic performance and they are right to be concerned. The current trend in education policy is to simply segregate kids on past academic performance and parental involvement and then simply write off the kids who with low prior achievement and low parental involvement. This is simply an unethical and unacceptable approach. This country is supposedly a meritocracy, but that certainly is not how the current band of education reformers are operating.

NOTE: The Austin school board voted to terminate the contract with IDEA based, in part, on data that showed the charter continually lied and mis-led policymakers and the public about the academic outcomes of the charter organization.

(1) This is one reason why reliance on value-added methodology is so perilous. Systems that do not account for peer effects at the classroom level may be assigning positive or negative effects to a teacher when, in fact, peer effects were the real influence.

IDEA puts students on path to succeed in an increasingly competitive global marketplace by providing a rigorous college preparatory education and preparing our low-income, Hispanic and minority students in under-served communities across Texas to apply, matriculate and succeed in a the four year college or university of their choice. To date, 100% of IDEA graduates have been accepted to a four-year college or university and our student (sic) are demonstrating remarkable staying power: 92% are either still in college or have graduated.[2]

Further, on page six of the report, the claim is made that, “ . . . for the fifth year in a row, 100% of IDEA graduates enrolled in four-year colleges and universities, fulfilling IDEA’s mission of College For All Children.”[3]

Yet, these claims are demonstrably false, the report fails to cite any data sources or studies that substantiate the claims, and the report fails to report publicly available data on the performance of students in four-year universities. The remainder of this short report substantiates my claims made about statements included in the IDEA annual report and provides data from the Texas Higher Education Coordinating Board about the Performance of IDEA Public School graduates in Texas four-year universities.

Claim #1) 100% of IDEA graduates have been accepted to a four-year college or university

While 100% of graduates from IDEA may ultimately enroll in four-year universities, data from the Texas Higher Education Coordinating Board (THECB)[4] directly contradicts this claim for every cohort with available data.

Table 1 shows the number and percentage of IDEA graduates in various settings in the fall immediately after high school graduation. Some students are not trackable due to issues in the student identification number and some students are designated as “not found.” Students not found could have entered a post-secondary institution in another state or simply not entered any post-secondary institution. As shown in Table 1, 58% of IDEA 2007 graduates entered a Texas four-year university while 75% of 2011 graduates entered a Texas four-year university. These rates could be greater if non-trackable and not found students actually entered four-year universities either within Texas or in another state. However, even if we assume that all non-trackable and not found students actually entered four-year universities, the rate would still not be 100% since between 6% and 11% of graduates entered two-year colleges. While the rate of students entering four-year universities is impressive, the rate is not 100%. Thus, data from THECB clearly contradicts the claims of IDEA CEO Tom Torkelson.

This decision was made in December 2011, a full 6 months after students completed the 2011 TAKS testing and at least 5 full months after the state released student-level data.

Now, given that AISD has some top-notch researchers, one would have suspected that AISD would have had internal researchers examine the student growth at Allan relative to other AISD schools and schools across the state after controlling for student demographics, school size, etcetera. At the very least, one would have though the district would have hired some outside consultants to examine all of the relevant information about the schools before making such a decision. They did not do this either other than hire a physics professor with no training in research methodology and who does not believe in any type of statistical analyses that controls for student background characteristics.  The physics professor did not have access to student-level data (the district could have provided him the data) and, evidently, no one in AISD ran any statistical analysis to determine the academic trajectory of Allan elementary.

The Texas Comptroller hires some statistical consultants with a great wealth of experience and expertise to create growth estimates in math, reading, and math/reading combined for all schools in Texas. This data was released within the past few months and I just got around to looking at it for another project. And what did that data tell us about Allan?

The data shows that Allan had made some pretty solid growth in math during 2009-10 and made some STELLAR growth in reading during 2010-11. The results are z-scores of student GROWTH. Negative numbers indicate growth that was below the state average and positive numbers indicate growth that was above the state average. The further the score is from zero, the worse the score (if negative) or the better the score (if positive).

Did IDEA Academy have greater gains? Most definitely. But that is not the point.* There will always be schools that outperform or under-perform other schools. What is important is that Allan was making PROGRESS–indeed, making pretty significant progress.

Why would the superintendent and school board throw a school showing such improvement under the bus?

Shouldn’t the superintendent and school board have recognized this positive trend and rewarded the educators at the school and supported them to make further growth?

Or were there other reasons for why a charter operator that will remove democratic control of a local school?

—————————————————-

The FAST methodology focuses only on students taking a regular TAKS test (as opposed to special needs students who take an alternate for modified form of the test), does not control for peer effects, and does not control for the systematic loss of students that might advantage or disadvantage a particular school.

Charter schools have proliferated in Texas and across the nation. The expansion of charter schools is now a popular reform effort for many policymakers on both the right and left of the political spectrum. To examine the efficacy of such policies, a number of researchers have focused on the effects  charter schools have had on student achievement, most of which have found little difference in achievement between the two types of schools (CREDO, 2009; Zimmer, R., Gill, B., Booker, K., Lavertu, S., Sass, T., and Witte, 2008). Yet, there is still a relative dearth of information about the characteristics of students entering and leaving charter schools and how these characteristics might be related to school-level achievement. This is particularly true with respect to charter schools in Texas. Most of the work in this area has focused on student racial and ethnic characteristics, while a fair number of studies have examined special education status and English-Language Learner status of entrants. Very little research has focused on the academic ability of students entering charter schools, the student attrition rate of charter schools, and the characteristics of the students staying and leaving charter schools. This study seeks to ameliorate this paucity of information, particularly as it pertains to high-profile and high-enrollment charter schools in Texas.

Findings 

The findings reviewed in this section refer to the results for the most appropriate comparison—the sending schools comparison—unless otherwise noted. Full results are in the body of the report or in the appendices. The CMOs included in this particular study included: KIPP, YES Preparatory, Harmony (Cosmos), IDEA, UPLIFT, School of Science and Technology, Brooks Academy, School of Excellence, and Inspired Vision. 

Characteristics of Students Entering Charter Schools

Differences in Average TAKS Z-Scores

The differences are reported in z-scores in order to make the results across school years comparable.

At both the 5^th- and 6^th-grade levels, students entering most of the CMOs in this study had TAKS mathematics and reading scores that were statistically significantly greater than comparison schools–in particular, schools that sent at least one child to the charter in question or schools located in the same zip code as the charter school.

Figure 1 shows the differences for incoming 5th grade students. Note the extremely large differences for Harmony charter schools. While the differences for KIPP were small, they were still statistically significant.

Figure 1: Difference in TAKS Math and Reading Z-Scores of Students from Sending Schools Entering and Not Entering Selected CMOs

All differences denoted with a number were statistically significant at the p < .05 level

Figure 2 shows the differences for incoming 6th grade students. Most of the differences were relatively large and positive for eight of the nine CMOs, thus showing that students entering these schools have substantially greater TAKS scores than students from sending schools. Again, the differences are particularly large for Harmony schools. The results for KIPP reflect only KIPP schools with 6th grade as the entry year.

Figure 2: Difference in TAKS Math and Reading Z-Scores of Students from Sending Schools Entering and Not Entering Selected CMOs

All differences denoted with a number were statistically significant at the p < .05 level

In only a few cases was there no statistically significant difference in TAKS scores between those students entering a CMO and not entering a CMO. Only one CMO—School of Excellence—had TAKS scores that were consistently lower than comparison schools.

For CMOs with greater levels of achievement for incoming students than comparison schools, the differences tended to be both statistically significant and practically significant. In other words, the differences appeared to be large enough to potentially explain differences in the levels of achievement between the CMOs and comparison schools. This does not mean the CMOs did not have greater student growth than other schools (growth was not examined in this study), but that the differences in TAKS passing rates often cited by supporters of charter schools and politicians could potentially be explained by the initial differences in achievement levels between students entering the CMOs and comparison schools.

Distribution of TAKS Mathematics and Reading Scores

The following two figures document the distribution of TAKS mathematics and reading scores for students in CMOs and in the various comparison groups of schools. As shown in Figure 3, eight of the nine CMOs had greater percentages of students scoring in the top 40% of test-takers and lower percentages of students scoring in the bottom 40% of test-takers. The differences were greater than ten percentage points in each of the two groups for KIPP, YES Prep, Harmony, and UPLIFT. The differences for Harmony and UPLIFT approached 15 percentage points—strikingly large disparities in the performance of incoming students. The compression of reading scores against the test score ceiling could explain the smaller differences in reading than in mathematics, but further analyses are needed to examine this possibility.

Figure 3: Difference in the Percentages of 5^th Grade Students Entering the 6^th Grade with TAKS Mathematics Scores in the Bottom 40% of Scores and Top 40% of Scores for CMOs and Comparison Schools*

* Comparison school percentage is based on the average of the results for comparison schools in the same zip code, schools in the same zip code and contiguous zip codes, and sender schools

Figure 4: Difference in the Percentages of 5^th Grade Students Entering the 6^th Grade with TAKS Reading Scores in the Bottom 40% of Scores and Top 40% of Scores for CMOs and Comparison Schools*

* Comparison school percentage is based on the average of the results for comparison schools in the same zip code, schools in the same zip code and contiguous zip codes, and sender schools

Differences in Scores for All Students and Economically Disadvantaged Students (Students Entering the 5^th Grade)

This section examines the TAKS math and reading scores of 4^th grade students identified as economically disadvantaged entering CMOs in the 5^th grade. Students entering a CMO were defined as not having been enrolled in the same CMO in the previous year. Further, a student must have been identified as economically disadvantaged to be included in the analysis. Finally, the analysis focuses only on those students enrolled in the 4^th grade in a “sending” school—a school that sent at least one student to that particular CMO in at least one of the cohorts of students included in the analysis. Statistically significant differences are noted by inclusion of the difference in z-scores in the graph. If the difference between students entering the CMO and not entering the CMO was not statistically significantly different, then no number was included in the graph.

TAKS Mathematics

As shown in Figure 5, both all students and economically disadvantaged students entering KIPP and Harmony had greater TAKS math z-scores than all students and economically disadvantaged students from sending schools that did not enter the CMOs. Moreover, the differences for economically disadvantaged students were greater than for all students. For economically disadvantaged students entering KIPP, the difference was 0.260 standard deviations while the difference for students entering Harmony was 0.530. Both differences were quite substantial. Thus, economically disadvantaged students entering KIPP and Harmony has substantially greater prior mathematics scores than students from sending schools that did not enter KIPP or Harmony.

Both all students and economically disadvantaged students entering the School of Excellence, on the other hand, had prior math scores that were lower than the prior math scores for students not entering the School of Excellence. The difference was smaller for economically disadvantaged students than for all students.

Figure 5: Differences in TAKS Mathematics Z-Scores Between Economically Disadvantaged Students Entering and Not Entering Selected CMOs for All Texas Students and Sending School Comparison Groups (4th Grade Scores of Incoming 5th Grade Students)

All differences denoted with a number were statistically significant at the p < .05 level

TAKS Reading

As shown in Figure 6, the results for reading were quite similar to the results for mathematics. For example, all students and economically disadvantaged students entering KIPP and Harmony had greater TAKS reading scores than students from sending schools not entering those CMOs. Further, the differences were much greater for economically disadvantaged students than for all students. For economically disadvantaged students entering KIPP, the difference was 0.229 while the difference for economically disadvantaged students entering Harmony was 0.495. Again, as with the differences in mathematics, the differences in reading were quite substantial for these two CMOs. Thus, economically disadvantaged students entering KIPP and Harmony were far higher performing than economically disadvantaged students from the very same schools that did not enter these CMOs.

Figure 6: Differences in TAKS Reading Z-Scores Between Economically Disadvantaged Students Entering and Not Entering Selected CMOs for All Texas Students and Sending School Comparison Groups (5th Grade Scores of Incoming 6th Grade Students)

All differences denoted with a number were statistically significant at the p < .05 level

Differences in Scores for All Students and Economically Disadvantaged Students (Students Entering the 5^th Grade)

TAKS Mathematics

As shown in Figure 7, the differences in TAKS mathematics scores for economically disadvantaged students were statistically significant and positive  for eight of the nine CMOs (School of Excellence was excluded from the graph, but had a statistically significant difference that indicated economically disadvantaged students entering the CMO had lower TAKS mathematics scores than students from sending schools not entering the  CMO). All but one of the eight differences was at least 0.200 standard deviations and three of the differences were greater than 0.300 standard deviations. Further, and perhaps more importantly, the differences were substantially larger than the differences for all students for the eight selected CMOs except Inspired Vision. Thus, undoubtedly, the economically disadvantaged students entering the CMOs were substantially different than the students from the very same schools that did not enter the CMOs. Indeed, the economically disadvantaged students entering the CMOs had far greater levels of achievement than the economically disadvantaged students that did not enter the CMOs.

Figure 7: Differences in TAKS Mathematics Z-Scores Between Economically Disadvantaged Students Entering and Not Entering Selected CMOs for All Texas Students and Sending School Comparison Groups

All differences denoted with a number were statistically significant at the p < .05 level

TAKS Reading

            As shown in Figure 8, eight of the nine CMOs had statistically significant differences in TAKS reading scores for economically disadvantaged students that revealed students entering the CMOs had greater TAKS reading scores than students from the sending schools not entering the CMOs (again, School of Excellence was excluded from the graph, but had a statistically significant, negative, and larger difference for economically disadvantaged students). All but one of the differences was greater than 0.150 standard deviations and four of the differences were greater than 0.350 standard deviations. As with the mathematics finding, this clearly demonstrates that economically disadvantaged students entering the CMOs had far greater levels of reading achievement than students from sending schools that did not enter the CMOs.

Figure 8: Differences in TAKS Reading Z-Scores Between Economically Disadvantaged Students Entering and Not Entering Selected CMOs for All Texas Students and Sending School Comparison Groups

All differences denoted with a number were statistically significant at the p < .05 level

Difference in Economically Disadvantaged Students

Figure 9 shows the results for the statewide comparison and the comparison between a CMO and other schools in the same zip code. KIPP, IDEA, Harmony, and UPLIFT all had a greater percentage of students entering the schools identified as economically disadvantaged as compared to all students in the state. Alternatively, when compared to schools located in the same zip codes as the CMOs, a lower percentage of students identified as economically disadvantaged entered the CMOs. Again, this shows that the comparison set of schools or students employed in an analysis can substantially alter the results. Indeed, in this case, comparing the percentage of economically disadvantaged students in the CMO to all students in the state suggests that the CMOs enroll a greater percentage of economically disadvantaged students. When the comparison group employed is students enrolled in schools within the same zip code as the CMO, however, an entirely different picture emerges. Indeed, now a lower percentage of students entering the CMOs were designated as economically disadvantaged.

Figure 9: Differences in the Percentage of Economically Disadvantaged Students Entering the 6^th Grade between Students Entering Selected CMOs and not Entering CMOs From All Schools and Schools in the Same Zip Code

All differences denoted with a number were statistically significant at the p < .05 level

STUDENT RETENTION RATES

            This section examines student retention and attrition from a few different perspectives. All of the analyses focus on students enrolled in schools in the 6^th grade in either the 2007-08 or 2008-09 school years. Further, the two cohorts were combined into one group of students included in the analyses. These two years were selected because they were the two most recent years for which data was available to track retention from 6^th grade through two years later—presumably 8^th grade for most students.  Prior years would have been included, but too few charter schools enrolled students in grades six through eight in prior years to yield sample sizes ;large enough for reliable estimates. This underscores the fact that even though charter schools have existed since 1997, very few have graduated complete cohorts of students over more than a few years.

School and District Retention Rates

            In this study, student retention refers to students remaining enrolled in the same school rather than students being retained in the same grade. In the analyses below, student retention rates indicate the percentage of students remaining enrolled in the same school from 6^th grade through 6^th, 7^th, or 8^thgrade two years later. All students enrolled in the 6^th grade—regardless of previous enrollment in the particular CMO, were included in the analyses.

Because this analysis examined two-year retention rates, only students enrolled in schools that had grades six, seven, and eight for at least two consecutive cohorts of students were included in the analysis So, for example, if a student was enrolled in the 6^th grade in 2008, then the school had to enroll at least 10 students in the 6^th grade in 2008, 10 students enrolled in the 7^th grade in 2009, and 10 students enrolled in the 8^th grade in 2010 to be included in the analysis.

Because so few charter schools met this criteria prior to 2008, only the last two cohorts of 6th-grade to 8^th-grade of students were included in the analysis. Even focusing on just the last two cohorts removed a large number of charter schools because most charter schools simply have not been in existence for enough years to meet the criteria set forth.

In addition, because some districts opened new schools and, consequently, large numbers of students moved from one middle school to another, schools with low school retention rates and high intra-district mobility rates were not included in the analysis. In general, any school with a district mobility rate of greater than 20% was excluded from the analysis. Ultimately, the inclusion or exclusion of such schools resulted in only a marginal difference in retention rates for comparison schools.

The results for KIPP were complicated by the 6^th grade not being the lowest grade level in most KIPP middle schools. For three of the seven KIPP middle schools, the initial grade level was the 5^th grade, not the 6^th grade. These schools had already experienced initial attrition.

Finally, the comparison set of schools employed in this particular analysis was all schools located in the same zip code as the charter school or in a zip code contiguous to the zip code in which the charter school was located.

As shown in Table 1, three of the eight CMOs had a statistically significant lower two-year school retention rates than their comparison schools while one had statistically significantly greater two-year retention rate. The three with statistically significant differences (Harmony, Brooks, and School of Science and Technology), all had retention rates lower than the retention rates for traditional public schools. Harmony had a retention rate almost 20 percentage points lower than comparison schools while both Brooks and School of Science and Technology had retention rates 24and 27 percentage points lower than comparison schools respectively. Strikingly, Harmony lost more than 40% of 6^th grade students over a two-year time span while Brooks and School of Science and Technology lost about one-half of all 6^th grade students in a two-year time span. YES Prep had a slightly greater retention rate than comparison schools. The difference was slight at 2.6 percentage points. This difference disappeared when student transfers within the YES Prep CMO were not considered as students staying enrolled at the same school.

Finally, when only traditionally configured middle schools were included in the KIPP analysis, the four schools serving grades 6 through 8 had a retention rate of 74.7%. This was only slightly lower than the overall retention rate and was still not statistically significantly different than the comparison set of schools for the four KIPP middle schools.

Table 1: School Retention Rates (6^th Grade to Two Years After 6^th Grade)

^ p < .10; * Pp < .05; ** p < .01; *** p < .001

Table 2 includes the district-level two-year retention rates. In this analysis, a student staying within the same school district or same CMO was defined as being retained in the district. Note that these rates were identical to the school retention rates for CMOs because of the manner in which I coded the data. The traditional public comparison schools all had statistically significantly greater district retention rates than the CMOs. All of the differences were at least five percentage points, with the greatest differences reserved for Brooks Academy (35.8), School of Science and Technology (28.4), and Harmony (18.1).

Table2: Student District Retention Rates (6^th Grade to Two Years After 6^th Grade)

^ p < .10; * Pp < .05; ** p < .01; *** p < .001

Stayers and Leavers by Test Scores

            While overall retention rates provide important outcome information as well as information that might affect school-level achievement, the characteristics of the students that stay or leave a school also provides critical information that may influence judgments about the academic efficacy of a particular school. For example, in examining the achievement of two schools that have the same retention rate, a school that loses lower performing students to a greater degree than higher performing students may artificially inflate overall performance levels as well as create a peer group effect that improves changes in improvement.

The following four tables include the school retention rates for students in CMOs and in traditional public schools in the same geographic location. Some schools were excluded such as schools that had high attrition rates due to intra-district student migration patterns caused by school feeder pattern or boundary changes. This was based on a comparison of school retention rates and within-district migration over a three-year period. In general, schools that had at least 20% student migration to other schools within the district were excluded from the analysis. Further, schools without a regular accountability rating were excluded from the analysis since such schools typically serve students with disciplinary issues or that have special needs. Finally, charter schools were excluded from the comparison group, thus leaving only traditional public schools in the comparison group. The analysis was first conducted without excluding charter schools and retention rates were somewhat lower. Careful examination of the data indicated that Harmony and a few special setting charter schools had very low retention rates that were lowering the average of the comparison groups in the setting, albeit by only a few percentage points.

Note that not all students had TAKS scores. Thus, the rates and data in these tables are not directly comparable to tables with the overall attrition rate.

TAKS Mathematics Scores

Table 3 compares the retention rates between CMOs and traditional public comparison schools for lower performing students on the 6^th grade mathematics examination. Students with z-scores less than -0.25 were designated as lower performing.

Four CMOs had statistically significantly lower retention rates for lower performing students: YES Prep, Harmony, Brooks Academy, and School of Science and Technology. The differences for YES Prep and Harmony were moderately large at 5 and almost 11 percentage points, respectively. The differences for School of Science and Technology and Brooks Academy were very large at almost 23 and 31 percentage points respectively. Note that there was no statistically significant difference in retention rates for all students for YES Prep, but a statistically significant difference for lower performing students.

Table 3: Student Retention Rates for Lower Performing Students on the 6^th Grade TAKS Mathematics Test for CMOs and Comparison Traditional Public Schools

^ p < .10; * Pp < .05; ** p < .01; *** p < .001

Table 4 compares the retention rates between CMOs and traditional public comparison schools for higher performing students on the 6^th grade mathematics examination. Students with z-scores greater than 0.25 were designated as lower performing. Four CMOs had statistically significantly lower retention rates for lower performing students: YES Prep, Harmony, Brooks Academy, and School of Science and Technology. The differences for YES Prep and Harmony were moderately large at 6 and almost 11 percentage points, respectively. The differences for School of Science and Technology and Brooks Academy were very large at almost 23 and 31 percentage points respectively. Note that there was a statistically significant difference in retention rates for lower performing students for YES Prep, but not a statistically significant difference for higher performing students. Closer inspection also reveals that YES Prep and Brooks Academy were the only CMOs to have lower retention rates for lower performing students and higher retention rates for higher performing students.

Table 4: Student Retention Rates for Higher Performing Students on the 6^th Grade TAKS Mathematics Test for CMOs and Comparison Traditional Public Schools

^ p < .10; * Pp < .05; ** p < .01; *** p < .001

TAKS Reading Scores

Table 6 compares the retention rates between CMOs and traditional public comparison schools for lower performing students on the 6^th grade reading examination. Students with z-scores less than -0.25 were designated as lower performing. The same three CMOs with lower retention rates for lower performing students in mathematics also had statistically significantly lower retention rates for lower performing students in reading: Harmony, Brooks Academy, and School of Science and Technology.

Table 6: Student Retention Rates for Lower Performing Students on the 6^th Grade TAKS Reading Test for CMOs and Comparison Traditional Public Schools

^ p < .10; * Pp < .05; ** p < .01; *** p < .001

Table 7 compares the retention rates between CMOs and traditional public comparison schools for lower performing students on the 6^th grade reading examination. Students with z-scores less than -0.25 were designated as lower performing.

The same three CMOs with lower retention rates for lower performing students also had lower retention rates for higher performing students. Note, however, that the difference between Brooks Academy and comparison schools was greater for lower performing students than for higher performing students, thus suggesting some selective attrition that may impact the distribution of test scores.

There was also a statistically significant difference for YES Prep. Higher performing students on the reading test were more likely to remain at YES Prep than for comparison schools. This, coupled with the lower retention rates for lower performing students also suggests some selective attrition that may have impacted the distribution of student test scores for YES Prep.

Table 7: Student Retention Rates for Higher Performing Students on the 6^th Grade TAKS Reading Test for CMOs and Comparison Traditional Public Schools

^ p < .10; * Pp < .05; ** p < .01; *** p < .001

Effects of Attrition on the Distribution of TAKS Scores

If student attrition differs across students with various levels of student achievement, then student attrition may positively or negatively impact overall school student test scores. For example, if student attrition tends to be greater for lower performing than for higher performing students, the overall test score profile of a school could improve regardless of whether the remaining students had improved test scores. Alternatively, if attrition affects higher performing students to a greater degree than lower-performing students, then a school’s test score profile could appear lower than it might have been otherwise if the higher performing students had not left the school.

Further, research suggests that peer effects have relatively powerful effects on student achievement. For example, all other factors being equal, an average performing student placed with a group of higher performing students will typically have greater gains in achievement than if placed with a group of lower performing students (need references).

Thus, the intent of this section was to examine the effect attrition may have on the composition of students with respect to test scores. This analysis focused only on those students enrolled in a school in the 6^th grade and then enrolled in the 8^th grade two years later in the same school. The TAKS score ranges were based on the 6^th grade score of the student and the percentages in the table represent the distribution of students by the 6^th grade TAKS scores for all students enrolled in the 6^th grade and the students remaining in the school in the 8^th grade.

Tables 8a, 8b, 8c, and 8d detail the distribution of TAKS mathematics and reading z-scores in the 6^th grade for all students enrolled in the 6^th grade in the 2007-08 and 2008-09 and the distribution of the z-scores for students remaining in the same school from 2007-08 through 2009-10 and from 2008-09 through 2010-11. The row labeled “Original Dist” includes the distribution of scores for all students enrolled in the 6^th grade and the row labeled “Stayer Dist” displays the z-score distribution of 6^th grade scores for only those students that remained at the school for two years after the 6^th grade. The third row, labeled “Stayer – Original” is the difference in the percentage of students in a particular category of z-scores between students staying at the school and the for all students originally enrolled. In this way, the data in the table document the changes in the distribution of 6^th grade scores due to the removal of students leaving the school.

Before examining the impact of attrition on the distribution of scores, Tables 20a and 20b also detail the vast differences in the initial distribution of scores between CMOs and comparison schools. For all but UPLIFT and Inspired Vision, the CMOs had students that had distribution of scores that were much more favorable than for the comparison schools. For example, nearly 60% of the YES Prep 6^th grade students were higher performing in mathematics as compared to only about 31% for comparison schools. In some cases, these rather substantial differences simply expanded due to attrition as will be described below.

As shown in Table 8a and 8b, KIPP, YES Prep, and Harmony and their comparison schools all evidenced a similar pattern in which the percentage of lower performing students (those with z-scores lower than -0.25) decreased and the percentage of higher performing students (those with z-scores greater than 0.25) increased. While the overall trend was the same for all three CMOs, there were, however, some apparent differences in the re-distribution of scores between the CMOs and comparison schools.

In mathematics, KIPP schools had a greater decrease in the percentage of lower- and average-performing students than comparison schools and a greater increase in the percentage of higher-performing students. The difference in the percentage of higher performing students, however,, was less than one percentage point. In reading, both KIPP and comparison schools had a decrease in the percentage of lower performing students, but the decrease was more than one percentage point greater for KIPP than comparison schools. KIPP also evidenced an increase of almost one percentage point in the percentage of average performing students while comparison schools had no increase. Finally, the increase in the percentage of higher performing students was 0.5 percentage points greater for KIPP than for comparison schools. Thus, the findings suggest a slightly greater re-distribution of scores upward for KIPP than for comparison schools.

Alternatively, the differences between YES Prep and Harmony and the comparison schools was greater than the differences for KIPP and their comparison schools and the differences suggest an important re-distributional effect from attrition of students from YES Prep and Harmony.

For YES Prep in mathematics, there was almost a five percentage point decrease in the lower- and average- performing students and almost a five percentage point increase in the percentage of higher performing students. Comparison schools had a 3.5 percentage point decrease in the percentage of lower performing students and a 0.5 percentage point increase in the percentage of average performing students and only a three percentage point increase in the percentage of higher performing students. Thus, the initial differences in performance at the 6^th grade starting point between YES Prep and comparison schools simply grew larger due to the differences in attrition between YES Prep and comparison schools. Thus, even if students made no progress on the TAKS tests, YES Prep would have appeared to have made greater progress due to differences in attrition across the distribution of scores.

With respect to Harmony schools, there was a decrease in both the percentage of lower- and average-performing students and a commensurate increase in the percentage of higher-performing students. This was ore pronounced for mathematics than for reading. For comparison schools, there was a decrease in lower performing students in both subject areas, but no real change in the percentage of average performing students. Ultimately, the increase in the percentage of higher performing students in comparison schools was lower than the increase for Harmony schools. The differences, however, were only one percentage point. Thus, Harmony have benefited more from the re-distribution of scores due to attrition more than comparison schools, but the advantage would have been less than the advantage for YES Prep.

Interestingly, there was no significant re-distribution of scores due to attrition for IDEA schools in either subject. For IDEA comparison schools, there was a slight upward re-distribution of scores. Thus, in this particular case, the comparison schools garnered a greater positive re-distributional effect from attrition than IDEA schools.

Table 8a: Distribution of Students TAKS Mathematics and Reading Z-Score Ranges for Students Enrolled in the 6^th Grade and Students Remaining in the Same School (KIPP and YES Prep)

Table 8b: Distribution of Students TAKS Mathematics and Reading Z-Score Ranges for Students Enrolled in the 6^th Grade and Students Remaining in the Same School (IDEA and Harmony)

            The remaining four CMOs and comparison schools are included in Table 8c and 8d. The re-distributional effects were greater for these four CMOs than for the first four CMOs. The smaller sample sizes and greater attrition rates are two reasons for these greater shifts in scores after attrition of students.

In both subject areas, UPLIFT experienced a greater upward re-distribution of scores than for comparison schools, although the effect in reading was relatively small. The impact on mathematics scores, however, was substantial. Indeed, There was a 6.7 percentage point decrease in the percentage of lower performing students in UPLIFT and a 4.0 percentage point increase for average performing students and a 2.7 percentage point increase for higher performing students. This re-distributional shift was far more positive than for comparison schools.

For Brooks—which had an extremely high overall attrition rate—there was a massive re-distribution of scores after the attrition of students. Specifically, in both subjects areas, there was greater than a 10 percentage point decrease in the percentage of lower performing students and an almost 8 percentage point increase in the percentage of higher performing students. For comparison schools, there was a small decrease in the percentage of lower performing students and a small increase in the percentage of higher performing students, but the changes were quite small in relation to the large changes for Brooks. Thus, Brooks Academy test scores improved rather dramatically for the two cohorts of students simply by losing large percentages of lower performing students.

In mathematics, both Inspired Vision and comparison schools lost a substantial proportion of lower performing students (5 percentage points). The scores for Inspired Vision shifted into the average- and higher-performing categories while most of the increase was for the higher performing categories for the comparison schools. For reading, there was a similar result. For comparison schools, there was a decrease of around two percentage points for both lower- and average performing students and increase of four percentage points for higher performing students. This was much greater than the one percentage point increase for Inspired Vision. Thus, Inspired Vision comparisons schools benefited more from attrition than did Inspired Vision.

Finally, the School of Science and Technology had a greater upward re-distribution of scores due to attrition than for comparison schools. For both mathematics and reading, the increase in the percentage of higher-performing students was about 1.5 percentage points. In mathematics, the School of Science and Technology also had a far greater decrease in the percentage of lower-performing students: 5.5 percentage points to 2.9 percentage points for comparison schools.

Table 8c: Distribution of Students TAKS Mathematics and Reading Z-Score Ranges for Students Enrolled in the 6^th Grade and Students Remaining in the Same School (UPLIFT and Brooks Academy)

Table 8d: Distribution of Students TAKS Mathematics and Reading Z-Score Ranges for Students Enrolled in the 6^th Grade and Students Remaining in the Same School (Inspired Vision and Scool of Science and Technology)

While these shifts may appear relatively small, in many cases the re-distribution of scores simply exacerbated existing differences in the distribution of scores between CMOs and comparison schools as shown in previous sections. Further, the differences are compounded because they occur for every cohort of students in the school. The most pronounced effects of attrition may be to reinforce certain peer group effects. If students see that peers that “cannot cut it” systematically leave a school, then the remaining students may be more motivated to work even harder to ensure continued enrollment in their school of choice.

How does this affect the performance levels reported by the state such as the percentage of students passing and percentage of students achieving commended status? The effects turn out to be quite similar. Figure 10 below shows that student attrition increased the percentage of students that had passed or met commended status on the 6^th grade mathematics test for YES Prep. In fact, the increase in the percentage of students that had met commended status increased from 57.1% to 62.0%. Again, this suggests a re-distribution of students after attrition such that lower-performing students were more likely to leave the school and higher-performing students were more likely to stay at the school.

Figure 10: Change in the Percentage of Students Passing and Meeting Commended Status on the TAKS Mathematics Test after Student Attrition for YES Prep Middle Schools

While these analyses do not reveal how student attrition impacts the actual scores, passing rates, and commended rates on the 8^th grade test, the fact that students passing or meeting the commended standard typically continue to meet those standards on future tests, the results do strongly suggest that attrition artificially increases passing and commended rates for some CMOs such as YES Prep.

FINAL CONCLUSIONS AND DISCUSSION

            This study is a preliminary examination of high-profile/high-performing charter management organizations in Texas. Specifically, the study examined the characteristics of students entering the schools, retention/attrition rates; and,the impact of attrition/retention rates on the distribution of students.

Contrary to the profile often portrayed in the media, by some policymakers, and by some charter school proponents (including some charter CEOs), the high-profile/high-enrollment CMOs in Texas enrolled groups of students that would arguably be easier to teach and would be more likely to exhibit high levels of achievement and greater growth on state achievement tests. Indeed, the above analyses showed that, relative to comparison schools, CMOs had:

• Entering students with greater prior TAKS scores in both mathematics and reading;
• Entering economically disadvantaged students with substantially greater prior TAKS scores in both mathematics and reading;
• Lower percentages of incoming students designated as ELL;
• Lower percentages of incoming students identified as special needs; and,
• Only slightly greater percentages of incoming students identified as economically disadvantaged.

In other words, rather than serving more disadvantaged students, the findings of this study suggest that the high-profile/high-enrollment CMOs actually served a more advantaged clientele relative to comparison schools—especially as compared to schools in the same zip code as the CMO schools. This is often referred to as the “skimming” of more advantaged students from other schools. While CMOs may not intentionally skim, the skimming of students may simply be an artifact of the policies and procedures surrounding entrance into these CMOs.

Thus, the comparisons that have been made between these CMOs and traditional public schools—especially traditional public schools in the same neighborhoods as the CMO schools—have been “apples-to-oranges” comparisons rather than “apples-to-apples” comparisons. The public and policymakers need to look past the percentages of economically disadvantaged students and disabuse themselves of the notion that enrolling a high percentage of economically disadvantaged students is the same as having a large percentage of lower-performing students. In fact, despite a large majority of students entering the CMOs identified as economically disadvantaged, students at the selected CMOs tended to have average or above average TAKS achievement and certainly greater achievement levels than comparison schools. This was particularly true when comparing economically disadvantaged students in CMOs and traditional public schools—the economically disadvantaged students in CMOs had substantially greater academic performance than the economically disadvantaged students in the comparison traditional public schools.

There were few differences in attrition rates between CMO and comparison schools (with Harmony, Brooks Academy, and School of Science and Technology being the exceptions) and the attrition rates did not appear to advantage or disadvantage CMOs as a group relative to comparison schools. Three CMOs did appear to have selective attrition such that scores were artificially increased by the loss of lower performing students and the retention of higher performing students. These three CMOs were Brooks Academy, UPLIFT, and YES Prep.

What is beyond the scope of this study is to determine the effect of “skimming” higher performing students from traditional public schools, the effect of selective attrition, and the effect of selective “back-filling” might have on student peer effects. If, in fact, academic achievement gains are driven by the impacts of these phenomena on peer effects, then policymakers would need to ask whether CMOs are assisting truly disadvantaged students or simply serving as voluntary magnet schools that have selective entrance and attrition.

Ultimately, while far more detailed and sophisticated research needs to occur in this area, these preliminary results should raise serious questions about how the characteristics of incoming students and the effect of attrition might impact the achievement profiles of CMOs and other schools. These questions beg to be answered before state policymakers endeavor to further expand and provide greater support to such CMOs and local policymakers move to replicate charters or adopt charters to replace local schools.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

This study was commissioned by the Texas Business and Education Coalition. TBEC was formed by Texas business leaders to engage with educators in a long-term effort to improve public education in Texas. Since its formation in 1989, TBEC has become one of the state’s most consistent and important forces for improving education in the state. Conclusions are those of the author and do not necessarily reflect the views of TBEC, its members or sponsors.

DATA AND METHODOLOGY

            This section provides a description of the data and methodology employed in this study.

Data

            This study relied on three sources of data. The first source was student-level testing data from the Texas Education Agency (TEA). The data was purchased from TEA by TBEC for the purpose of examining important education topics in Texas. The second set of data was school-level information from the Academic Excellence Indicator System (AEIS).The third set of data was the Financial Allocation Study for Texas from the Texas State Comptroller’s Office.

Student-Level Testing Data

The testing data included information on students taking the Texas Assessment of Knowledge and Skills (TAKS) from spring 2003 through summer 2011 in grades three through 12. Student information includes the school and district in which the student was enrolled when s/he took the TAKS test, grade level, economically disadvantaged status, test score, score indicator, exemption status (e.g., special education exemption, Limited English Proficiency exemption, absent, etc.) and test type. Importantly, even if a student did not actually take the TAKS, such a student would be included in the data because an answer document was submitted for the student. Because of FERPA, some information was masked by TEA. However, the student was not removed from the data and the student was still associated with a particular school and district.

School-Level AEIS data

The school-level AEIS data included a wealth of information on schools in Texas, including charter status, district in which the school was located, the region of the state in which the school was located, the overall number of students, and the number and percentage of students with various characteristics (i.e., percentage of economically disadvantaged students, percentage of White students, percentage of Latino students, percentage of African American students, etc.) and participating in specific education programs (special education, bilingual education, English as a Second Language).

Student Characteristics Included the Analyses

The characteristics examined in this section include:

1. TAKS mathematics scores;
2. TAKS reading scores;
3. Economically disadvantaged status;
4. Spanish-Language TAKS test;
5. Exemption from TAKS for Limited English Proficiency (LEP) reasons; and
6. Special Needs students as identified by the type of TAKS tests taken by the student.

TAKS Scores

Because metrics such as passing TAKS, achieving commended status on TAKS, and scale scores on TAKS vary over time, with students in later cohorts being more likely to have passed, attained commended status, or achieved a higher scale score than students in earlier cohorts, such metrics would not provide an appropriate measure to use across multiple cohorts of students. In order to compare achievement levels in a defensible manner, the TAKS scale scores were standardized across years and test administrations so schools with more students in later cohorts and fewer in earlier cohorts would not have artificially greater scores and schools with more students in earlier cohorts would not have artificially lower scores. To standardize the TAKS scores over time, the scale scores were converted to z-scores for each grade level and year. Further, z scores were calculated separately for students taking different versions of the test. Thus, a separate z-score was calculated for all students taking the standard TAKS, TAKS-modified, and TAKS-alternative versions of the test for each grade level and each subject area.

Economically Disadvantaged Status

In Texas, economically disadvantaged is determined by participation in the federal free-/reduced-price lunch program. In addition, a student can be identified as economically disadvantaged if she or he is eligible for other public assistance programs intended for families in poverty. In the data provided by TEA, the district in which the student enrolled identified whether or not a student was classified as economically disadvantaged.

English Language Learner Status

English Language Learner (ELL) students were identified in two ways. First, students were identified by having taking the Spanish-language version of the TAKS. The Spanish version was available in grades three through six. Second, the test score code provided by the state also identified those students exempted from testing for Limited English Proficiency (LEP) reasons. Thus, measures four and five were collapsed into one measure identifying students as English-Language Learner students. While there was some overlap between the two groups, only 7% of the students taking the Spanish TAKS were also identified as being LEP exempt. Ultimately, a student was identified as ELL if the student (a) took the Spanish-language TAKS in the previous year or (b) was exempted from TAKS testing because of LEP reasons.

Special Needs Students

Special needs students were identified by the type of TAKS test taken by the student. Unfortunately, this data was only available in academic years 2008 through 2010. In previous years, a substantial proportion of students identified as special needs were placed into a separate file by the state. To comply with FERPA, the student identifier process was different than the one employed for non-special needs students. Thus, the two files could not be merged together which prohibited the use of the data in years prior to 2008.

With respect to the different types of TAKS tests, the TAKS-modified (TAKS-M) and TAKS-alternate (TAKS-A) tests were developed for students that require some alternate test form based on either modifications or an alternative assessment strategy to meet the needs of the student under either Section 504 of the Rehabilitation Act of 1973 or under an Individual Education Plan. Unfortunately, relying on test type does not directly assess the number of students in special education. Some special education students do not require any special test modifications while other students (such as those with a 504 plan) may require modifications but not be designated as special education. Thus, the students taking either a TAKS-A or TAKS-M test were not designated as special education, but rather as having special needs with respect to state standardized assessments. This is an important distinction because many 504 and special education students need only minimal changes in instruction and additional assistance while those requiring special testing are far more likely to require extra attention and assistance by educators.

TEA described the TAKS-modified test in the following manner:

The Texas Assessment of Knowledge and Skills–Modified (TAKS–M) is an alternate assessment based on modified academic achievement standards designed for students who meet participation requirements and who are receiving special education services. TAKS–M has been designed to meet federal requirements mandated under the No Child Left Behind (NCLB) Act. According to federal regulations, all students, including those receiving special education services, will be assessed on grade-level curriculum. TAKS–M covers the same grade-level content as TAKS, but TAKS–M tests have been changed in format (e.g., larger font, fewer items per page) and test design (e.g., fewer answer choices, simpler vocabulary and sentence structure). 

(Retrieved from http://www.tea.state.tx.us/student.assessment/special-ed/taksm/

TEA described the TAKS-alternate test in the following manner:

TAKS–Alternate (TAKS–Alt)is an alternate assessment based on alternate academic achievement standards and is designed for students with significant cognitive disabilities receiving special education services who meet the participation requirements for TAKS–Alt.* This assessment is not a traditional paper or multiple-choice test. Instead, it involves teachers observing students as they complete state-developed assessment tasks that link to the grade-level TEKS. Teachers then evaluate student performance based on the dimensions of the TAKS–Alt rubric and submit results through an online instrument. This assessment can be administered using any language or other communication method routinely used with the student.

(Retrieved from http://www.tea.state.tx.us/student.assessment/taks/accommodations/)

While other measures of special needs or special education designation would certainly be important as well, such measures were not available in the data procured from the Texas Education Agency. The measures included in the analyses were selected for two reasons: First, because the data can be used to directly address claims of charter school proponents; and, second, research suggests each measure is associated in some manner with school-level test score levels as well as school-level growth.

Understanding Z-Scores

Transforming the TAKS scale scores into z-scores not only controls for differences in scores across time (scale scores typically increase for the same grade level in each successive year), z scores have some important properties that allow for arguably better and easier to understand comparisons of average scores between schools.

First, transforming scale scores into z scores creates a set of scores that are normally distributed as in the well-known bell-shaped curve. Second, placing the scale scores into z-scores creates some useful properties that make comparisons across schools easier. For example, a z-score distribution creates a mean of zero. Thus, the average student in a cohort of students has a z-score of zero. Thus, once the scale scores are converted into z scores, the z scores indicate how far a student’s score is from the statewide average. If a student had a positive z-score, then the student’s score was above average. If a student had a z-score that was negative, then the student’s score was below average. Not only does the z-score indicate direction, but also magnitude. So, for example, if a student had a z-score of 1.0, then that student had a score that was 1.0 standard deviations greater than the score for the average student. If a student had a z-score of -0.45, then that student had a score that was 0.45 standard deviations lower than the score for the average student.

Third, we know that a certain percentage of students fall within each standard deviation. For example, we know that 34.1% of students will have a TAKS z-score between 0.0 and +1.0 and 34.1% of students will have a TAKS z-score between 0.0 and -1.0. Because of this characteristic of normal curves, we can translate the z-scores into percentile rankings.  For example, if a student has a TAKS z-score of +2.0, then we know that only about 2% of students have a greater TAKS z-score and 98% of students have a lower TAKS z-score. If a student has a TAKS z-score of -1.0, then we know that about 85% of students have a greater TAKS z-score and about 15% of students have a lower TAKS z-score.

One difficulty in examining charter school students is determining the appropriate comparison group of students.  Often, state education agencies, charter school representatives, and media personnel compare charter schools and students to all other schools and students. While such a comparison provides some useful information, such comparisons are flawed because charter schools are located in distinct locations. As such, enrollment in a charter school is typically limited to those students that live relatively close to the charter school. So, for example, a student living in Texarkana cannot enroll in a charter school in Houston. Thus, most researchers employ a different comparison group of schools and students rather than simply all schools or students.

Thus, this study examines four different comparisons: all Texas students, same geographic location, same zip code, and sending schools. Each of the four comparisons is described in more detail below.

1) All Texas Students

One comparison made in this report was charter schools and charter students compared to all schools and all students in Texas. Thus, for example, the incoming characteristics of students entering a charter school in the 6^th grade would be compared to the characteristics of all 5^th grade students in all Texas schools.

2) Same Geographic Location

The second comparison employed in this analysis was between a selected charter school and all schools in the same geographic location. The same geographic location was defined as all schools located in the zip code in which the charter school was located or in a zip code contiguous to the zip code in which the charter school was located.

3) Same Zip Code

The third comparison employed in this analysis was between a selected charter school and all schools in the same zip code. Thus, a charter school and the students within the charter school were compared to only schools and students within schools located in the same zip code as the charter school.

4) Sending School

The final comparison employed in the analysis was between a selected charter school and the students in that charter school to schools and the students within the schools that sentat least one student to the selected charter school over the given time period. This comparison was used by both Mathematica (2010)in their analyses of charter schools. Such a comparison seems most appropriate when comparing characteristics of students, characteristics of schools, or student performance.

Selected Charter Schools

            This study focused on nine charter schools that served students in grades four through eight. Most were charter management organizations (CMOs) that included multiple schools and had schools across the different grade levels such that the CMOs served students from kindergarten through the 12^th grade. These charter schools are referred to as charter management organizations (CMOs) in this paper even though, in some cases, only one school from a CMO was included in an analysis.

Table 2 lists, in descending order, the CMOs with the greatest number of incoming 6^th grade students from Texas public schools and other schools not in the Texas public school system for the years 2005 through 2011. The number of incoming students excluded students already enrolled in that particular CMO in the previous grade. Thus, for example, a student enrolled in KIPP in the 5^th grade and then the 6^th grade was not identified as an incoming student for KIPP in the 6^th grade.

Ultimately, nine of the 15 CMOs with the largest number of incoming 6^th grade students were included in this study. These CMOs appear in bold in the table below. CMOs that utilized on-line or distance education were excluded from the study as were CMOs that focused on students at-risk of dropping out of school. Overall, these nine CMOs enrolled almost 60% of all of the incoming 6^th grade students into Texas charter schools.

The two largest CMOs—Harmony and Yes Prep—both accounted for 15% of all students entering charter schools in the 6^th grade. Two schools—Responsive Education and Southwest Virtual School—were not included in subsequent analyses because creating a set of comparison schools based on zip codes simply did not apply to a virtual school. Houston Gateway could conceivably have been selected for inclusion, but Inspired Vision had a greater percentage of incoming students with data from the 5^th grade than the Gateway charter. The Radiance Academy could have been chosen, but it was unclear as to whether the Radiance charter was associated with other charters with the same or similar names. Thus, rather than risk making a mistake in correctly identifying the complete set of schools for the CMO, I selected the next school on the list which was Inspired Vision.

I wrote this in response to Representative Aycock’s request for ideas on improving the accountability system. Unfortunately, he chose not to publish this suggestion. Perhaps he thought I was not serious (I am) or that it would never pass the Legislature (he is correct).

I think reasonable people believe individuals should not be held accountable in the workplace unless the person was provided adequate resources and support to meet her/his expectations. I also think most reasonable people would agree that schools and districts should not be held accountable unless they are provided the necessary resources and support to meet the expectations that under gird the accountability system.

Expectation #1: Allocate enough money per pupil to ALL districts in an amount sufficient to meet the expectations set forth by the legislature.

This would be determined through adequacy studies conducted by a panel of five nationally recognized experts. For any district taxing at the maximum allowable tax rate and not generating enough funds through all revenue streams to meet expectations as determined by the panel of experts, then the district could choose not participate in any state mandated testing and accountability system.

In the last school finance court case from 2005, experts found that the state was underfunding an adequate education. Since then, per pupil funding increased a miniscule 7% over the cost of living increases (This will surely decrease to near 0% with recent budget costs coupled with increased enrollment). Further, the meager growth in per pupil spending came during a time of explosive growth in the percentage of economically disadvantaged students that are far more costly to educate than non-economically disadvantaged students. Moreover, expectations have risen dramatically since 2005. Districts and schools are now expected to ensure all students are college and career ready, graduate from high school, and now pass STAAR and End-of-Course exams that are touted by state leaders as being far more rigorous. If the legislature is not going to provide the necessary resources for districts to meet the expectations of the legislature, then why should school districts be held accountable by that very same legislature?

Expectation #2: Allocate money in a fair and equitable manner, including money provided through a bi-annually updated cost-of-education index (CEI) to ensure all districts competed on an equal playing field.

This would be determined by a panel of five nationally recognized experts. If the system was not considered equitable across districts with different levels of property wealth and percentage of students living in poverty, then any district in the bottom 25% of schools in WADA would be exempted from the state’s district and school accountability system.

The weights for the weighted average daily attendance (WADA) should be based on the findings of the adequacy studies. The studies would estimate the amount of money necessary to educate a student to achieve particular outcomes as measured by a low-stakes assessment or an assessment for which no teaching to the test could corrupt the scores.

The CEI should be updated every two years and re-weighted. Districts should not operate under the “hold harmless” provision that is the largest obstacle to updating the CEI currently.

Expectation #3:  Require all individuals entering a classroom as a teacher of record to have a minimum of eight weeks of training, at least 3 weeks of some form of clinical training, and at least a minor in the subject area to which they are assigned.

Currently, low-performing schools are disproportionately staffed by teachers that obtained certification through alternative means. Worse yet, the schools with the greatest needs typically must resort to hiring teachers from private alternative certification programs, many of which provide less than a month of preparation before entering the classroom, allow individuals with college GPAs lower than 2.25 to enter the program (sometimes GPAs are as low as 2.0), and require only 12 college hours in the subject area to be taught. In comparison, high quality preparation programs (both traditional and alternative) provide hundreds of pre-service hours, have high entrance standards with respect to GPA, and require all teachers to have at least a minor and typically a major

Research has clearly shown that teachers that have received more preparation are typically more effective and stay longer in the field. That low-performing schools must resort to hiring teachers from private alternative certification programs simply decreases achievement while concomitantly increases costs through higher teacher attrition.

Unlike high-performing countries across the world, the United States has chosen to reduce barriers to entry into the profession rather than make entrance into teaching more rigorous. Texas has been at the forefront of this effort to deregulate the teaching profession by creating private alternative certification programs whose primary motive is to generate profit rather than prepare quality teachers. In fact, the worse the preparation provided by these programs, the higher the attrition rate which simply drives the demand for new teachers higher. And who benefits? Private alternative certification programs!

To change the current status of teacher preparation in Texas will likely require additional state funding for stipends for teachers in high-need areas (STEM, bilingual education, special education) that agree to teach in high-need schools as well as an appropriate school finance system that includes an appropriate CEI.

FASTLE Report Card

A final panel of experts and educators would consider all available evidence (which would be made available to the public on a website) and assign a grade for each of the three expectations and a final, overall grade. Categories could include less than satisfactory, satisfactory, or exemplary. If the overall grade was “less than satisfactory,” then the state’s district and state accountability systems would cease to operate until the rating improved to satisfactory.

My Estimation of the Current Report Card Grades

Based on my reading of the literature, analysis of the data, and discussions with experts knowledgeable of the Texas school finance system, I provide ratings for each of the three areas of responsibility.

Expectation #1: Allocate enough money per pupil to ALL districts in an amount sufficient to meet the expectations set forth by the legislature.

Rating: less than satisfactory

Expectation #2: Allocate money in a fair and equitable manner, including money provided through a bi-annually updated cost-of-education index (CEI) to ensure all districts competed on an equal playing field.

Rating: less than satisfactory

Expectation #3:  Require all individuals entering a classroom as a teacher of record to have a minimum of eight weeks of training, at least 3 weeks of some form of clinical training, and at least a minor in the subject area to which they are assigned.

Rating: less than satisfactory

OVERALL FASTLE RATING: 2011-12:

Less Than Satisfactory

I am open to suggestions and comments. I am sure some people will respond that school districts are inefficient and waste money. Some fit this bill. Others do not. And I can guarantee one thing—I can walk into any business in which a Legislator works or a business person works and find plenty of areas of inefficiencies and waste. The free market does not clean up inefficiency and waste in businesses and corporations, so any argument that a free-market approach would clean things up is simply nonsensical (that, and the fact that research shows a free-market approach to education simply does not work very well).

With respect to outright cheating and fraud, there is a clear ethical line–it is wrong. There is never any excuse.

In this case, the principal instigated a test prep plan in which teachers were told to instruct students on reading and math during science, social studies, and even music classes. The students were denied the opportunity to learn the material in these other classes. Unbelievably, the principal forced the teachers to submit grades for students in science and social studies even though the students never took science and social studies.  In Dallas, a principal engaged in massive teaching to the test, lied to parents about it, and stole the opportunity to learn science and social studies from a group of students at her school. It is truly a bizarre and unbelievable story.

http://www.dallasnews.com/opinion/editorials/20111121-editorial-a-campus-far-from-exemplary–field-elementary-in-dallas.ece

This was a clear ethical violation and Dallas ISD acted appropriately to remove her from her position.

See http://dallasisdblog.dallasnews.com/roslyncarterreport.pdf

Yet, even though Washington DC administrators were well aware of the facts of the case, they have decided that such a principal possesses the characteristics and qualities they are looking for in a school leader.

http://www.washingtonpost.com/blogs/answer-sheet/post/new-dc-school-principal-left-texas-under-cloud/2012/01/31/gIQASvsSgQ_blog.html

http://dallasisdblog.dallasnews.com/archives/2012/02/principal-cited-for-skipping-s.html

Maybe the district administrators view the principal’s efforts as a “No Excuses” effort to ensure students learned how to read and pass the standardized tests.

“Don’t let state and district requirements about what is taught be an excuse for not focusing narrowly on reading and math–it is for the kids!”

“Don’t let parental expectations about what their kids are learning in school stand in the way of raising test scores. Lie to the parents. Don;t let their expectations become an excuse for not raising test scores!”

I guess we have seen the true values and beliefs of those running the DC public school system.

I put both emails into this post. The first one sounds like any parent with a child struggling in any school. I have been in a similar situation myself. The second one is more alarming. I’ll share more thoughts after the emails.

My name is [redacted], I am an educator in the Rio Grande Valley. [Sentence redacted]. My [child] attends IDEA Public Schools in [redacted], Texas. My [child] is a second grader at IDEA and [s/he] has been attending for three years. One of the first disappointments that I have had is the bullying problem and the lack of student control that there is in school. My [child] was bullied for a while and after [s/he] reacted negatively against those who were bullying [my child], [s/he] got in trouble. That was my first issue with them. I kept [my child] there because of the IDEA reputation and also because it advertised college readiness. I have visited the classrooms, and it is somewhat chaotic. The IDEA website mentions that the teacher student ratio is always 15:1. This is not true. My [child] is in a class with 29 other students, the teacher at times has some students doing independent work while she reads to another group. That leaves the other students unsupervised. There was little supervision during recess (which they claim they are trying to solve this issue). This is where my [child] would get in trouble in school. The lessons are boring, consisting of worksheets after worksheets. My [child] looks forward to two classes, [her/his] computer math class, and the reading renaissance class. However, these classes are packed with 40 students. [My child] especially dreads math class. I am in constant contact with the principal and all of my [child]‘s teachers. The principal and assistant principals have assisted me with my [child], and I think they are trying their best with the situation that they have. For the past three years that my [child] has been there, there have been different principals for each year. The teacher turnover rate is high. The teachers often call me to let me know about my [child]‘s negative behavior, every once in a while they will let me know if [s/he] has improved, but usually it is at my inquiring. The teachers seem overwhelmed, they want to keep control of the class, but the kids do not listen. I have shadowed my [child] as well as my mother, and she agrees with all the observations I have made. My [child], I have to admit, is not perfect, but [s/he] is not a behavior problem at home. [S/he] complies with everything I ask [her/him] to do, including homework. It seems to me that [my child’s] misbehavior only happens at school. However, they have turned it around on me asking me what is happening at home that might be altering [my child’s] behavior.

[My child] feels so stressed out that two days prior to returning for the second semester, [s/he] started crying because [s/he] was dreading going back saying that [s/he] was afraid [s/he] might not behave. For the week that I have been [away from home for training for my job], and my mother has been taking [my child] to school, [my child] complains of stomach aches. I felt I was doing something good for my [child by placing her/him at IDEA], but I feel like I am making [my child] miserable. The teachers and the principals say that [my child’s] temperament is what gets [her/him] in trouble. I told [IDEA personnel] I was going to pull [my child] out and put [her/him] in public schools, they suggested I stay and develop a SAC. One of the teachers said, that if [s/he] was going to misbehave in IDEA, [s/he] would misbehave in public school. She was implying that [my child] would be a problem child everywhere. At this point I don’t know if it’s the IDEA public schools environment, or there is something wrong with my [child]. [My child] is not failing, [s/he] knows [her/his] math, [s/he] knows how to read at [her/his] level, and [s/he] is extremely knowledgeable in science. It is [my child’s] favorite subject. When [s/he] is at school [s/he] says school is boring, especially math. Because of the complaints that teachers make about [her/his] behavior, I put more pressure on [her/him]. I am making [my child] more anxious and frustrated. I am also anxious and frustrated. At this point I do not know what to do. I want the best for my [child], but I don’t know if I should pull [my child] out in the middle of this semester, what if [s/he] continues to misbehave in public schools. I would love for you to visit this school so that you could see first-hand. A [school] counselor [who is a friend of mine] pulled her kindergarten child because she was having similar issues as my [child]. I feel I am ranting, but I really would like to hear from you and ask you for your input. I am at a loss, I want the best for my [child], I feel like I am failing [my child] and I can’t give him the best education despite my best intentions.

Email Two

My mother spent another day with my [my child], the teachers complained again about [my child] not “finishing work”. [My child’s} work consisted of worksheets again. My mother counted the number of students in math class, and she counted 35. The groups are homogeneous by ability. There are kids from different grades together. Their Better Idea curriculum is based on direct instruction. (the [teachers] often don’t give students the opportunity to provide feedback to the teacher verbally, at least that’s what my mother observed in math class). I understand that public schools at the elementary level cannot have more than 22 students. Why is it that IDEA can get away from having these restrictions? The week he returned to school for the second semester, we had a SAC (that’s what IDEA calls it) –other schools call it RTI. I have not seen any improvement. There is only one special ed teacher/diagnostician for the several IDEA Campuses (I’m not sure how many campuses); that’s my understanding at least. The principal emailed her the info, but I have never met her. When we had the SAC, she was not present. It was only the principal, teachers, the school counselors, and myself. Was this done right? When my son started at IDEA three years ago, they asked if my son had a learning disability or was diagnosed as special ed., and if so, they would not be able to give him accommodations because they did not have that set in place. I guess over these past three years, they have developed this.

So, what do you think? If true, very disconcerting for a number of reasons. Most importantly, the denial of special education services of the development of a response to intervention plan without a special education educator does not seem ethical and may, in fact, be illegal.

And the use of direct instruction, worksheets, and teaching to the test (communicated by some who visited IDEA as well as by a TFA teacher in an IDEA school who emailed me) is disconcerting as well and may explain why IDEA students tend to do well on TAKS, but not on other tests such as the SAT or in college classrooms (see my previous posts).

http://www.statesman.com/news/local/austin-school-district-says-critical-idea-study-doesnt-2085799.html

Note that Dr. Julian Vasquez-Heilig, a researcher at UT Austin and recent recipient of a University Council for Educational Administration award for research by junior professors, had this to say about the two reports:

“Dr. Fuller’s work is very comprehensive,” Vasquez Heilig said Friday after reading both reports. “The AISD rebuttal is very weak in its methodology.”

Let me be clear: The students who enter and remain in IDEA Charter Schools perform quite well at the secondary level. At the elementary level, performance is above average in reading and slightly below average in math. They send most or all of the students graduating from the high schools to some form of post-secondary education. They should be commended for that. The positive aspects of IDEA were communicated to the AISD school board and administration. However, what was not communicated was the potential negative aspects of IDEA charter schools and the reasons why their performance may appear greater than other schools. IDEA presented this information to the AISD board.

Thus, my intention was to point out WHY the IDEA school performance appears the way that it does so that the Austin community could decide if the strategies employed by IDEA were strategies acceptable to the Austin community. Unfortunately, the school board never had an honest, open, and public discussion about the strengths and weaknesses of IDEA charter schools.

My rebuttal starts below.

Issues with the Caritj and Marder Response to the

Ed Fuller Study of IDEA Charter Schools

There are many, many issues with the paper authored by Caritj and Marder. Within the next few weeks, I will provide a full response. Below are some of the major issues with their paper.

I. The Response by the Authors Ignores My Major Conclusions

The authors completely ignore two of my major conclusions: (1) students entering the 6^th grade in IDEA charter schools are more advantaged than students entering the 6^th grade in public schools in the same area served by IDEA charter schools; and, (2) lower-performing students are more likely to leave IDEA charter schools than other schools in the same area.

1) The authors only address the issue of the characteristics of students already enrolled in IDEA charter schools. The entire point of the first section of my paper was to show that students entering IDEA charter schools are statistically different from students entering schools in the same area in very important ways. In my paper, I make the following conclusions based on a logistic regression analysis.[1]

• Economically disadvantaged students were 30% less likely to enroll in an IDEA charter school;
• Students taking the English-language TAKS test in reading were 2.65 times more likely than students taking the Spanish-language TAKS to enroll in an IDEA charter school;
• Students passing the TAKS mathematics test were almost two times more likely to enroll in an IDEA Charter school than students who did not pass;
• Students passing the TAKS reading test were almost two times more likely to enroll in an IDEA Charter school than students who did not pass;
• Students taking the regular TAKS test (indicating no special needs) in mathematics were 3.3 times more likely to enroll in an IDEA charter school than students not taking the regular TAKS test (indicating special needs);
• Students taking the regular reading TAKS were four times more likely to enroll in an IDEA Charter school than students not taking a regular TAKS test.

SOURCE: TEA student level data; Analysis: Ed Fuller

How about the percentage of bilingual students? As shown in Figure 5, IDEA enrolls a lower percentage of bilingual students than the majority of East Austin elementary schools.

]]> http://fullerlook.wordpress.com/2012/01/07/rebuttal-to-the-austin-isd-critique-of-my-study-on-idea-charter-schools/feed/ 2 edfuller IDEA math and reading comparisons IDEA non-test characteristics of students entering IDEA CR to SAT 2008 CR to SAT 2009 CR to SAT 2010 College Ready Results IDEA and selected RGV IDEA and Martin gr606 2011 % sped AISD ES % bilingual AISD ES % eco dis AISD ES Z score results for IDEA v East Austin http://fullerlook.wordpress.com/2011/12/18/idea-and-martin-ms-disappearance-rate/ http://fullerlook.wordpress.com/2011/12/18/idea-and-martin-ms-disappearance-rate/#comments Sun, 18 Dec 2011 13:11:06 +0000 Dr. Ed Fuller http://fullerlook.wordpress.com/?p=610 ]]> This posts looks at the most recent cohort of 6th graders enrolling in IDEA Charter Schools and Martin MS in Austin ISD. Why start with 6th graders? Because IDEA Charter Schools are separated into elementary schools (K-5) and secondary schools (6-12). So, theoretically, students enter 6th grade and remain in IDEA Charter Schools until graduation and then 100% of them enter college.

This evidence will be criticized because it is only one cohort, but it is the latest cohort available. Thus, it reflects the most current evidence we have on the success of IDEA Charter Schools.

This analysis extends my previous analyses by including middle school grade levels in addition to grades 9 through 11. This extension is important because any school level growth measures include all grades in the school. In the case of IDEA secondary schools, the grades served include grades 6 through 12.

Tables 1 and 2 show the number and percentage of 6th grade students in the 2005-06 school year taking the TAKS mathematics and reading tests that were still enrolled in the same district in the 2010-11 school year in any grade. The analysis compares three entities: IDEA Academy, Hidalgo ISD, and Martin MS in Austin ISD.

I chose Hidalgo ISD as a relevant comparison to IDEA in the Rio Grande Valley (Hidalgo is the headquarters for IDEA) and Martin MS in Austin ISD since it is the feeder pattern that IDEA would replace in Austin ISD.

Table 1: Number and Percentage of 6^th Grade Students (2005-06) Remaining in the Same District through the 2010-11 School Year by 6^th Grade TAKS Mathematics Score

Table 2: Number and Percentage of 6^th Grade Students (2005-06) Remaining in the Same District through the 2010-11 School Year by 6^th Grade TAKS Reading Score

CONCLUSIONS

Two conclusions can readily be made based on this data. The first conclusion is that IDEA Charter Schools retained a much smaller percentage of students in the same district as compared to both Hidalgo ISD and Martin Middle School. Indeed, only 47% of the IDEA Charter School 6^th grade students remained enrolled in any IDEA Charter School in 2010-11. In comparison, about 77% of the Hidalgo ISD students remained in Hidalgo ISD while 64% of the Martin MS students remained in AISD.

The second conclusion is that remaining in an IDEA Charter School is correlated with TAKS scores. Specifically, the greater the TAKS score, the more likely the student remained in IDEA. The same trend appeared with Hidalgo ISD students, but the difference in the percentage of students remaining in the district between low- and high-performers was not nearly as great as for IDEA Charter Schools. For Martin MS, a greater percentage of the average and slightly above average performing students remained in Austin ISD. The lowest-performing and high-performing were more likely to leave AISD than the students between those two groups.

This second conclusion is critically important because IDEA growth scores as calculated by FAST are very likely to be positively affected by the disappearance of the lowest performing students through peer effects. In fact, research is fairly consistent in this area, with almost all researchers finding positive effects of students being in the same classroom and school with high-performing peers.

DISCUSSION

This data is just one more piece of evidence that strongly suggests IDEA Charter Schools are not nearly as successful as IDEA Charter leaders and Austin ISD leaders claim them to be. Are they successful? In some areas, IDEA Charter Schools are successful. But, for every successful outcome, there appears to be data that suggests a troubling reason for that success.

One rationale for expanding charter schools is that such schools can help close the achievement gap. For example, the IDEA website claims, ” Closing the achievement gap is no longer a phrase, but a reality at IDEA Public Schools.” Yet, no data is provided that substantiates this claim. In fact, a search of the Texas Charter School Association, IDEA Charter school website, and a general google search for “achievement gap Texas charter schools” and “Texas charter schools closing achievement gap” did not yield a single result of the closing of the achievement gap. Hopefully, I simply missed the documentation and someone can point me to the evidence.

Because I could find no evidence, I decided to use student-level data from the Texas Education Agency to examine the achievement gap between students not participating in the federal free-/reduced-price lunch program and students participating in the federal free-/reduced-price lunch program in mathematics and reading pm the Texas Assessment of Knowledge and Skills (TAKS) in the 2010-11 school year. Further, I examined the data for all grade levels and for all schools enrolling at least 60% economically disadvantaged students. I chose the 60% cut point because all IDEA Charter Schools enroll at least 60% economically disadvantaged students.

Most organizations–including state education agencies and school districts–calculate the achievement gap incorrectly. Take for example, Austin ISD. In the AISD “State of the District” report, AISD claims the achievement gap has decreased by pointing out that the percentage of students passing the TAKS tests between White and non-White students has decreased over time.

Yet, as Daniel Koretz clearly explains in his book entitled “Measuring Up: What Educational Testing Really Tells Us,” using the percentage of students passing/proficient always provides an inaccurate assessment of the achievement gap. Other experts in this area concur. I strongly suggest reading his very easy to digest book top understand why we must use scale scores and, better yet, z-scores to assess the achievement gap.

Below I present the achievement gap in z-scores which are very similar to standard deviations.* A positive number indicates that not economically disadvantaged students outperform economically disadvantaged students and a negative number would indicate that not economically disadvantaged students were outperformed by economically disadvantaged students. The magnitude of the number indicates the size of the achievement gap. Thus, a result of .25 indicates a smaller achievement gap than a result of .75. The closer to 0 the result, the smaller the achievement gap.

As shown below, AISD has not closed the achievement gap. but neither has IDEA Charter Schools, nor all Texas schools. In most, but not all cases, the gap for IDEA schools is smaller than for AISD schools. Yet, the gap remains relatively large for IDEA Charter schools at most grade levels.

Table 1: TAKS Achievement Gaps in Mathematics and Reading (2011) for Austin ISD, IDEA Charter Schools, and all Texas Schools with at Least 60% Economically Disadvantaged Students

DATA SOURCE: Student-level TAKS data from TEA; Analysis: Ed Fuller

So, what can we conclude from this table?

First, IDEA Charter Schools have NOT closed the achievement gap between low-income and not low-income students and are not particularly close to doing so.

Second, IDEA Charter Schools generally have a smaller achievement gap than AISD schools. However, one would need to investigate the effect of students leaving each set of schools to further substantiate this claim.

Third, despite the claims of IDEA leaders that poverty does not matter and does not influence achievement, poverty clearly has a negative effect on achievement–even in IDEA Charter Schools. If poverty did not matter, there would be no achievement gap in IDEA Charter Schools. That poverty affects achievement is one of the most consistent findings in education research. It is next to impossible to find any quality study that finds otherwise.

* I calculated the gap by first calculating the z score for all students for each grade level with TAKS results. The z-score was calculated by calculating the mean, then dividing by the standard deviation. The achievement gap was then calculated for IDEA Charter Schools, AISD schools, and all Texas schools by subtracting the aggregate z-score for economically disadvantaged students from the aggregate z-score for not economically disadvantaged students, then dividing by the standard deviation for all students.