This post includes the information in my rebuttal to the Austin ISD critique of my study of the IDEA charter school system. The Statesman story can be found here:
Note that Dr. Julian Vasquez-Heilig, a researcher at UT Austin and recent recipient of a University Council for Educational Administration award for research by junior professors, had this to say about the two reports:
“Dr. Fuller’s work is very comprehensive,” Vasquez Heilig said Friday after reading both reports. “The AISD rebuttal is very weak in its methodology.”
Let me be clear: The students who enter and remain in IDEA Charter Schools perform quite well at the secondary level. At the elementary level, performance is above average in reading and slightly below average in math. They send most or all of the students graduating from the high schools to some form of post-secondary education. They should be commended for that. The positive aspects of IDEA were communicated to the AISD school board and administration. However, what was not communicated was the potential negative aspects of IDEA charter schools and the reasons why their performance may appear greater than other schools. IDEA presented this information to the AISD board.
Thus, my intention was to point out WHY the IDEA school performance appears the way that it does so that the Austin community could decide if the strategies employed by IDEA were strategies acceptable to the Austin community. Unfortunately, the school board never had an honest, open, and public discussion about the strengths and weaknesses of IDEA charter schools.
My rebuttal starts below.
Issues with the Caritj and Marder Response to the
Ed Fuller Study of IDEA Charter Schools
There are many, many issues with the paper authored by Caritj and Marder. Within the next few weeks, I will provide a full response. Below are some of the major issues with their paper.
I. The Response by the Authors Ignores My Major Conclusions
The authors completely ignore two of my major conclusions: (1) students entering the 6th grade in IDEA charter schools are more advantaged than students entering the 6th grade in public schools in the same area served by IDEA charter schools; and, (2) lower-performing students are more likely to leave IDEA charter schools than other schools in the same area.
1) The authors only address the issue of the characteristics of students already enrolled in IDEA charter schools. The entire point of the first section of my paper was to show that students entering IDEA charter schools are statistically different from students entering schools in the same area in very important ways. In my paper, I make the following conclusions based on a logistic regression analysis.
- Economically disadvantaged students were 30% less likely to enroll in an IDEA charter school;
- Students taking the English-language TAKS test in reading were 2.65 times more likely than students taking the Spanish-language TAKS to enroll in an IDEA charter school;
- Students passing the TAKS mathematics test were almost two times more likely to enroll in an IDEA Charter school than students who did not pass;
- Students passing the TAKS reading test were almost two times more likely to enroll in an IDEA Charter school than students who did not pass;
- Students taking the regular TAKS test (indicating no special needs) in mathematics were 3.3 times more likely to enroll in an IDEA charter school than students not taking the regular TAKS test (indicating special needs);
- Students taking the regular reading TAKS were four times more likely to enroll in an IDEA Charter school than students not taking a regular TAKS test.
Based on my recent work, I have analyzed six cohorts of 5th grade students entering IDEA charter schools as compared to students enrolled in schools served by IDEA charter schools. School designated as being in the same local area as IDEA charter schools were located in the same zip code as an IDEA charter school or in a zip code area contiguous with the zip code area in which an IDEA school was located. The school also had to be in a zip code that sent at least one student to an IDEA charter school. Finally, students already enrolled in an IDEA charter school in the 5th grade were excluded from the analysis.
Table 1 shows that students entering into IDEA charter schools had greater math and reading performance on the TAKS than students that remained in public schools and that lived in the very same zip codes served by IDEA.
TABLE 1: Differences in TAKS Mathematics and Reading Outcomes for Students Entering IDEA Charter Schools in the 6th Grade and Students Remaining in Public Schools from the 5th to 6th Grade for the 2005 through 2010 Cohorts of 5th Grade Students in Zip Codes Served by IDEA Charter Schools
The first portion of the table shows that a greater percentage of 5th grade students entering the 6th grade into an IDEA charter school had passed the TAKS math and reading tests. This was true for each of the six cohorts.
The second section of Table 1 shows the same result for commended status—a greater percentage of students entering an IDEA charter school in the 6th grade had achieved commended status in the 5th grade. This was true for all but one cohort in reading.
The third section of Table 1–and the most important result–shows that students entering IDEA charter schools had TAKS math and reading scores that were, on average, at least 0.2 standard deviations greater than students entering local public schools. Thus, students entering IDEA charter schools already performed substantially better than students entering public schools. This is a truly substantial difference as most educational interventions improve performance along the lines of .1 standard deviations or less.
So, when the AISD authors show that IDEA students outperform other students, they don’t examine why this might be the case. Table 1 above suggests that the reason may simply be that IDEA has higher performing students to begin with which would require other schools to have greater growth than IDEA just to appear equal in a scatterplot analysis.
As shown in Table 2, a lower percentage of economically disadvantaged students and special needs students entered IDEA charter schools as compared to local public schools. In addition, a lower percentage of 5th grade students entering IDEA charter schools in the 6th grade took the Spanish language TAKS test or were exempted from TAKS because of limited English proficiency (LEP).
Table 2: Percentage of 5th Grade Students Identified as Economically Disadvantaged, Having Special Needs, Taking the Spanish-Language TAKS, or Being Exempted from TAKS for Limited English Proficiency by Enrollment in the 6th Grade in an IDEA Charter School or a Local Public School
Thus, when compared to local public schools, a substantially different set of students enters IDEA charter schools than enters local public schools in the same area. This is important because having students with a diverse set of educational needs makes school improvement more difficult and expensive. Further, the authors simply did NOT address this finding in their paper. In fact, the authors have not bothered to purchase the data to examine these findings on their own.
What should be of concern to Trustees is whether IDEA will operate in the same manner in Austin as it has in the Rio Grande Valley. Will a substantially different set of more advantaged students enroll in the IDEA Austin charter or will IDEA Austin actually serve the same students as currently served by Allen and Eastside Memorial.
2) The authors also ignored my finding that students scoring at or below average on TAKS were more likely to leave an IDEA charter school than a local public school. Specifically, I found that students scoring at or below the state average on TAKS in IDEA schools were 85% less likely to remain at the same school than their peers in public schools in the IDEA charter market. To reiterate—9th grade students in IDEA Charter schools scoring average or below-average were 85% less likely to remain at the same school than their peers in public schools in the same neighborhoods.
Again, the AISD authors simply completely ignore this finding. This finding is critically important because it shows that lower performing students leave IDEA charter schools at higher rates than students in public schools.
II. The Response Has Incorrect Use of Data and Analytic Techniques
The authors rely on apples-to-oranges comparisons in their paper. For example, in trying to refute my analysis of the disappearance of students from 9th grade to 11th grade in IDEA charter schools, the authors rely on student mobility data from TEA. Yet, a student is designated as mobile if the student has not attended the school for 83% of the school year. So, students who are chronically absent, move to the school after the first six weeks of schools, or who leave the school before the end of the last six weeks of the year are included in the designation. This is completely different data than what I used and is simply an apples-to-oranges comparison. Any comparison is frivolous because the data simply are not measuring the same construct.
The authors also rely on scatterplots. There are multiple issues with these scatterplots.
First, the plots rely on one cohort of students. This cohort could be an outlier. More cohorts should be included.
Second, scatterplots do not “level the playing field” when examining student outcomes. The authors contend that by comparing outcomes across schools with varying levels of economically disadvantaged students, the comparisons then become an “apples-to-apples” comparison. But any introductory statistics book will tell you that scatterplots are a tool to examine simple relationships and that conclusions or comparisons about performance can only be used by employing far more sophisticated statistical techniques. Scatterplots are used in the beginning steps of analyses, but there are many, many subsequent steps that must be taken before reaching a conclusion. What the authors needed to do, but failed to do, was to perform many other scatterplots to examine the other factors influencing achievement, then use all of the factors in a regression analysis to determine which associations were statistical significant. The bottom line is that scatterplots suggest interesting relationships that need further investigation, but they don;t allow for valid comparisons of effectiveness or outcomes. So, one cannot conclude from the scatterplot that IDEA charter schools or any other school in the analysis is any more or less effective than any other school. Such conclusions need to be based on regression analysis results that control for a large number of the variables affecting achievement. This is basic statistical analysis 101.
Third, along these lines, the authors do not consider prior scores in their scatterplots nor do the scatterplots provide information on the value-added of attending an IDEA charter school. Prior scores are the strongest predictor of achievement levels, yet the authors completely ignore prior achievement. Note that my analyses show that students entering IDEA charter schools already outperform their peers entering regular public schools on TAKS. That the students outperform their peers in 11th grade should not be surprising given that they outperform their peers before entering IDEA charter schools. What the scatterplots do NOT reveal is the value-added by attending an IDEA charter school. The authors’ scatterplots simply reinforce the finding that IDEA charters skim students at the entry point for entering IDEA secondary schools.
Fourth, the scatterplots use combined free-/reduced-price lunch as the measure for economically disadvantaged status. The percentage of students on free lunch has a stronger negative effect on achievement. The authors did not examine or use such data which is available for free from the TEA website.
Fifth, the authors do not look at growth. Status measures such as percent passing or percent commended do not provide any information about the growth of students while in an IDEA charter school.
Sixth, the authors do not control for students leaving IDEA charter schools. As noted above, lower performing students are more likely to leave IDEA charter schools. Thus, the 1th grade passing and commended rates may be artificially inflated as compared to other schools since the disappearance rate for lower performing students is greater in IDEA charter schools than for surrounding local schools.
Seventh, it is curious that no plots of SAT are included. Dr. Marder has created a number of scatterplots that generally show charter schools underperform other schools. Included in this set of scatterplots is one that examines the percentage of economically disadvantaged students and SAT scores. Why was this scatterplot not included? I include my own set of scatterplots below that examine the relationship between the percentage of students achieving college-readiness status and average school-level SAT scores for the graduating class of 2008, 2009, and 2010 (2011 is not yet available from TEA).
As shown in each figure, the IDEA charter school performed below where the school was expected to perform (the line in the figure indicates where the school is predicted to perform on the SAT based on the percentage of students achieving college-readiness status). If the authors chose to rely on scatterplots, why were these scatterplots not made available? My scatterplots below were investigated using regression analysis and additional variables related to SAT scores such as the percentage of students taking the SAT. In each instance, IDEA schools had lower SAT scores than predicted by the regression analysis after controlling for a host of other variables.
Figure 1: Relationship between Percentage of Students Achieving College-Readiness Status and School Average SAT Scores (2008)
Figure 2: Relationship between Percentage of Students Achieving College-Readiness Status and School Average SAT Scores (2009)
Figure 3: Relationship between Percentage of Students Achieving College-Readiness Status and School Average SAT Scores (2010)
Again, these scatterplots were investigated through regression analysis and found to be representative of the findings from the regression analyses.
What this suggests to me is that the students who remain in IDEA charter schools performed well on TAKS, but underperformed on the SAT. This may be a result of teaching to the test as some teachers in IDEA have stated in emails to me. In fact, the poor performance of IDEA students in college supports this possibility.
As shown in Table 3, IDEA students perform at the level of the magnet schools in the Rio Grande Valley on TAKS related outcomes, but perform worse than any other school in the Rio Grande Valley in terms of performance in college. Indeed, almost 40% of IDEA students earned less than a 2.0 GPA in college–in essence, these students are “flunking out” of college. Now, one reason for this may be that IDEA sends all graduates to college. But clearly not all graduates were well-prepared for college. Perhaps other schools are more realistic about the academic ability of graduates and send some to college, some to technical schools, and some directly into the workforce.
Another interesting issue is that IDEA sends a lower percentage of students into 4 year colleges than schools with similar TAKS outcomes. Indeed, while IDEA and the two South Texas magnet schools have very similar rates of college-readiness, IDEA sends a significantly lower percentage of students into 4 year colleges than these schools. This may be due to the economic status of students or IDEA students entering private colleges or going out-of-state to attend college, but IDEA never presented any of this information to the AISD Board of Trustees to explain these rates.
Table 3: TAKS Performance, College Readiness, Attendance in 4 Year Colleges, and College Performance of Students in 4 Year Colleges for IDEA and Selected School Districts in the Rio Grande Valley (Key for titles is below the table)
% > 2200 in 9th grade = the percentage of students in 9th grade on track to achieve college-readiness in 11th grade
% CR= percentage of students achieving college-readiness as defined by TEA.
% 4 Year College: Percentage of graduates entering a 4 year public college in Texas in the fall semester after spring graduation from the Texas Higher Education Coordinating Board
% < 2.0 GPA = the percentage of students entering a 4 year college that had a GPA of less than 2.0
The authors also argue that IDEA should be compared to AISD schools. Yet, AISD schools and IDEA charter schools operate in a completely different environment and context. Only through regression analysis that controls for such factors could any comparison be made. Even then, controlling for the all the differing factors between Austin and the Rio Grande Valley would be extremely difficult. The best comparison to use when examining the impact of IDEA charter schools is the set of schools in the same area as the IDEA schools themselves.
But let’s look at some comparisons. Below, in Table 4, I compare the student retention rate by TAKS scores for IDEA charter students, students in Hidalgo ISD, and students in Martin Middle School in Austin ISD. Specifically, I examine the retention rates of students in the 6th grade through 11th grade. A student is considered retained if he/she remains in the same school district from 6th grade through 11th grade (or any grade level in the year the student should have been in the 11th grade). Less than 50% of the original cohort of 6th grade students in 2006 remained in an IDEA charter school. Only 30% of the lowest performers remained. For 6th grade students in Martin Middle School, 64% remained in the district and 64% of the lower performing students remained in the district. Further, the difference in retention rates by TAKS scores suggest that the Dr. Marder’s scatterplots that show IDEA outperforms other schools with similar poverty rates may simply be because the lower performing students in IDEA charter schools simply leave IDEA charters altogether. Is that the strategy Austin ISD wants to employ to improve student outcomes?
Table 4: Percentage of Students Remaining in the Same District from 6th Grade (2006) to 11th Grade (2011) by TAKS Mathematics Scale Score for IDEA Charter Schools, Hidalgo ISD, and Martin Middle School
Next, let us compare the percentage of special education students in IDEA elementary schools and East Austin elementary schools. Clearly, IDEA enrolls a lower percentage of special education students than schools in East Austin.
Figure 4: Percentage of Special Education Students in IDEA Elementary Schools and East Austin Elementary Schools in 2011
Figure 5: Percentage of Bilingual Education Students in IDEA Elementary Schools and East Austin Elementary Schools in 2011
The same holds true for economically disadvantaged students as shown in Figure 6.
In fact, a number of East Austin schools substantially outperform IDEA Academy in both subject areas. In fact, Graham Elementary is one of the best elementary schools in the entire state in terms of student growth. Blanton, Metz, Overton, and Norman also perform in the top 15% of elementary schools across the state.
Table 5: Average Three-Year (2008, 2009, 2010) TAKS Growth in Reading and Mathematics for East Austin Elementary Schools and IDEA Academy in Z-Scores
Why didn’t AISD simply get the principals and teachers from the five top-performing elementary schools in East Austin together with the lower performing elementary schools in East Austin such as Allen and have them collaborate to improve all schools? Collaboration and teamwork have been shown to be effective in turning around schools. Using charter schools to turn around schools has proved less effective, particularly because the charter schools typically end up enrolling a different set of students than the original public school. This would have been amenable to the East Austin community and has a much stronger research base than charter schools taking over public schools.
III. Selective Use of Quotes and Quotes out of Context
The authors take my comments out of context or use selective quotes that do not provide a full representation of my points.
For example, the authors take my “underserved student” quote out of context. In context, my conclusion is that IDEA schools do not serve a greater proportion of underserved students than schools in the same local area served by IDEA charter schools. IDEA students are only underserved in relation to students across the entire state. In fact, every single school in Region I serves a greater percentage of underserved students as compared with the entire state. Yet, students in Dallas and Austin cannot attend an IDEA charter school. Thus, the relevant comparison to IDEA charter schools is schools in the same local area as IDEA charter schools. When compared to local schools, IDEA enrolls a lower percentage of underserved students. If this trend continues in Austin, then IDEA will enroll a population that is significantly different—and more advantaged–than other schools in East Austin.
In addition, the authors do not fully complete my quote on peer effects. Researchers have consistently found that peer effects have an important influence on student achievement. Students enrolling in schools with higher performing students will be positively affected and students enrolling in schools with lower performing students will be negatively affected. Schools can create positive peer effects through positive behaviors such as high expectations and low tolerance for misbehavior or through negative behaviors such as skimming students and losing lower performing students. My analysis shows that IDEA likely uses both the positive and negative behaviors to create positive peer effects. If that is acceptable to the AISD Board of Trustees and the Superintendent, then statements should be made that enrolling more advantaged students and losing lower performing students are acceptable strategies for school improvement.
The authors’ paper is fraught with serious problems which render the paper less than helpful. The authors do not have access to the student level data needed to fully explore the effectiveness of IDEA charter schools and they rely on statistical techniques that are simply inappropriate to use to reach the conclusions that they reach.
If Austin ISD wants to know the complete picture of the effectiveness of IDEA charter schools (and the district should have fully investigated the efficacy of IDEA before awarding a contract), the district should hire an experienced education researcher and provide the researcher with access to the student level data necessary to completely evaluate IDEA charters.
 Logistic regression is the accepted statistical tool used when outcomes are binary such as a student entered or did not enter an IDEA charter school. The authors performed no such statistical analyses nor did they have access to the data hat would even allow them to do such an analysis.