A "Fuller" Look at Education Issues

The Priorities of DC Public Schools

Dr. Ed Fuller — Thu, 16 Feb 2012 11:32:58 +0000

Much has been written about the investigations into cheating in DC public schools. While the investigations are ongoing, the evidence suggests either cheating or some massive teaching to the test efforts. Neither cheating nor teaching to the test benefits children in any way. In his 2008 book entitled “Measuring Up: What Do Test Scores Really Tell Us?,” Koretz–a leading expert in testing–makes very clear that teaching to the test causes score inflation and limits rather than enhances students’ learning. Other testing experts concur with Koretz on this conclusion.

With respect to outright cheating and fraud, there is a clear ethical line–it is wrong. There is never any excuse.

In this case, the principal instigated a test prep plan in which teachers were told to instruct students on reading and math during science, social studies, and even music classes. The students were denied the opportunity to learn the material in these other classes. Unbelievably, the principal forced the teachers to submit grades for students in science and social studies even though the students never took science and social studies. In Dallas, a principal engaged in massive teaching to the test, lied to parents about it, and stole the opportunity to learn science and social studies from a group of students at her school. It is truly a bizarre and unbelievable story.

http://www.dallasnews.com/opinion/editorials/20111121-editorial-a-campus-far-from-exemplary–field-elementary-in-dallas.ece

This was a clear ethical violation and Dallas ISD acted appropriately to remove her from her position.

See http://dallasisdblog.dallasnews.com/roslyncarterreport.pdf

Yet, even though Washington DC administrators were well aware of the facts of the case, they have decided that such a principal possesses the characteristics and qualities they are looking for in a school leader.

http://www.washingtonpost.com/blogs/answer-sheet/post/new-dc-school-principal-left-texas-under-cloud/2012/01/31/gIQASvsSgQ_blog.html

http://dallasisdblog.dallasnews.com/archives/2012/02/principal-cited-for-skipping-s.html

Maybe the district administrators view the principal’s efforts as a “No Excuses” effort to ensure students learned how to read and pass the standardized tests.

“Don’t let state and district requirements about what is taught be an excuse for not focusing narrowly on reading and math–it is for the kids!”

“Don’t let parental expectations about what their kids are learning in school stand in the way of raising test scores. Lie to the parents. Don;t let their expectations become an excuse for not raising test scores!”

I guess we have seen the true values and beliefs of those running the DC public school system.

Letter from an IDEA Mother

Dr. Ed Fuller — Sun, 22 Jan 2012 18:55:21 +0000

Below are two e-mails I received from a mother whose child attends an IDEA elementary school. I am using her letter with permission. I have changed the gender of the child to neutral language and redacted certain information to protect her identity. Also, I typically do not use unsolicited, unverifiable emails. However, this person provided her name and place of employment which I was able to verify independently. Now, this person may be using someone’s name other than her own, but given the length and the repeated e-mails, that is a whole lot of trouble. So, I believe the letters to be true. I could be wrong.

I put both emails into this post. The first one sounds like any parent with a child struggling in any school. I have been in a similar situation myself. The second one is more alarming. I’ll share more thoughts after the emails.

My name is [redacted], I am an educator in the Rio Grande Valley. [Sentence redacted]. My [child] attends IDEA Public Schools in [redacted], Texas. My [child] is a second grader at IDEA and [s/he] has been attending for three years. One of the first disappointments that I have had is the bullying problem and the lack of student control that there is in school. My [child] was bullied for a while and after [s/he] reacted negatively against those who were bullying [my child], [s/he] got in trouble. That was my first issue with them. I kept [my child] there because of the IDEA reputation and also because it advertised college readiness. I have visited the classrooms, and it is somewhat chaotic. The IDEA website mentions that the teacher student ratio is always 15:1. This is not true. My [child] is in a class with 29 other students, the teacher at times has some students doing independent work while she reads to another group. That leaves the other students unsupervised. There was little supervision during recess (which they claim they are trying to solve this issue). This is where my [child] would get in trouble in school. The lessons are boring, consisting of worksheets after worksheets. My [child] looks forward to two classes, [her/his] computer math class, and the reading renaissance class. However, these classes are packed with 40 students. [My child] especially dreads math class. I am in constant contact with the principal and all of my [child]‘s teachers. The principal and assistant principals have assisted me with my [child], and I think they are trying their best with the situation that they have. For the past three years that my [child] has been there, there have been different principals for each year. The teacher turnover rate is high. The teachers often call me to let me know about my [child]‘s negative behavior, every once in a while they will let me know if [s/he] has improved, but usually it is at my inquiring. The teachers seem overwhelmed, they want to keep control of the class, but the kids do not listen. I have shadowed my [child] as well as my mother, and she agrees with all the observations I have made. My [child], I have to admit, is not perfect, but [s/he] is not a behavior problem at home. [S/he] complies with everything I ask [her/him] to do, including homework. It seems to me that [my child’s] misbehavior only happens at school. However, they have turned it around on me asking me what is happening at home that might be altering [my child’s] behavior.

[My child] feels so stressed out that two days prior to returning for the second semester, [s/he] started crying because [s/he] was dreading going back saying that [s/he] was afraid [s/he] might not behave. For the week that I have been [away from home for training for my job], and my mother has been taking [my child] to school, [my child] complains of stomach aches. I felt I was doing something good for my [child by placing her/him at IDEA], but I feel like I am making [my child] miserable. The teachers and the principals say that [my child’s] temperament is what gets [her/him] in trouble. I told [IDEA personnel] I was going to pull [my child] out and put [her/him] in public schools, they suggested I stay and develop a SAC. One of the teachers said, that if [s/he] was going to misbehave in IDEA, [s/he] would misbehave in public school. She was implying that [my child] would be a problem child everywhere. At this point I don’t know if it’s the IDEA public schools environment, or there is something wrong with my [child]. [My child] is not failing, [s/he] knows [her/his] math, [s/he] knows how to read at [her/his] level, and [s/he] is extremely knowledgeable in science. It is [my child’s] favorite subject. When [s/he] is at school [s/he] says school is boring, especially math. Because of the complaints that teachers make about [her/his] behavior, I put more pressure on [her/him]. I am making [my child] more anxious and frustrated. I am also anxious and frustrated. At this point I do not know what to do. I want the best for my [child], but I don’t know if I should pull [my child] out in the middle of this semester, what if [s/he] continues to misbehave in public schools. I would love for you to visit this school so that you could see first-hand. A [school] counselor [who is a friend of mine] pulled her kindergarten child because she was having similar issues as my [child]. I feel I am ranting, but I really would like to hear from you and ask you for your input. I am at a loss, I want the best for my [child], I feel like I am failing [my child] and I can’t give him the best education despite my best intentions.

Email Two

My mother spent another day with my [my child], the teachers complained again about [my child] not “finishing work”. [My child’s} work consisted of worksheets again. My mother counted the number of students in math class, and she counted 35. The groups are homogeneous by ability. There are kids from different grades together. Their Better Idea curriculum is based on direct instruction. (the [teachers] often don’t give students the opportunity to provide feedback to the teacher verbally, at least that’s what my mother observed in math class). I understand that public schools at the elementary level cannot have more than 22 students. Why is it that IDEA can get away from having these restrictions? The week he returned to school for the second semester, we had a SAC (that’s what IDEA calls it) –other schools call it RTI. I have not seen any improvement. There is only one special ed teacher/diagnostician for the several IDEA Campuses (I’m not sure how many campuses); that’s my understanding at least. The principal emailed her the info, but I have never met her. When we had the SAC, she was not present. It was only the principal, teachers, the school counselors, and myself. Was this done right? When my son started at IDEA three years ago, they asked if my son had a learning disability or was diagnosed as special ed., and if so, they would not be able to give him accommodations because they did not have that set in place. I guess over these past three years, they have developed this.

So, what do you think? If true, very disconcerting for a number of reasons. Most importantly, the denial of special education services of the development of a response to intervention plan without a special education educator does not seem ethical and may, in fact, be illegal.

And the use of direct instruction, worksheets, and teaching to the test (communicated by some who visited IDEA as well as by a TFA teacher in an IDEA school who emailed me) is disconcerting as well and may explain why IDEA students tend to do well on TAKS, but not on other tests such as the SAT or in college classrooms (see my previous posts).

Rebuttal to the Austin ISD critique of my study on IDEA Charter Schools

Dr. Ed Fuller — Sat, 07 Jan 2012 13:59:17 +0000

This post includes the information in my rebuttal to the Austin ISD critique of my study of the IDEA charter school system. The Statesman story can be found here:

http://www.statesman.com/news/local/austin-school-district-says-critical-idea-study-doesnt-2085799.html

Note that Dr. Julian Vasquez-Heilig, a researcher at UT Austin and recent recipient of a University Council for Educational Administration award for research by junior professors, had this to say about the two reports:

“Dr. Fuller’s work is very comprehensive,” Vasquez Heilig said Friday after reading both reports. “The AISD rebuttal is very weak in its methodology.”

Let me be clear: The students who enter and remain in IDEA Charter Schools perform quite well at the secondary level. At the elementary level, performance is above average in reading and slightly below average in math. They send most or all of the students graduating from the high schools to some form of post-secondary education. They should be commended for that. The positive aspects of IDEA were communicated to the AISD school board and administration. However, what was not communicated was the potential negative aspects of IDEA charter schools and the reasons why their performance may appear greater than other schools. IDEA presented this information to the AISD board.

Thus, my intention was to point out WHY the IDEA school performance appears the way that it does so that the Austin community could decide if the strategies employed by IDEA were strategies acceptable to the Austin community. Unfortunately, the school board never had an honest, open, and public discussion about the strengths and weaknesses of IDEA charter schools.

My rebuttal starts below.

Issues with the Caritj and Marder Response to the

Ed Fuller Study of IDEA Charter Schools

There are many, many issues with the paper authored by Caritj and Marder. Within the next few weeks, I will provide a full response. Below are some of the major issues with their paper.

I. The Response by the Authors Ignores My Major Conclusions

The authors completely ignore two of my major conclusions: (1) students entering the 6^th grade in IDEA charter schools are more advantaged than students entering the 6^th grade in public schools in the same area served by IDEA charter schools; and, (2) lower-performing students are more likely to leave IDEA charter schools than other schools in the same area.

1) The authors only address the issue of the characteristics of students already enrolled in IDEA charter schools. The entire point of the first section of my paper was to show that students entering IDEA charter schools are statistically different from students entering schools in the same area in very important ways. In my paper, I make the following conclusions based on a logistic regression analysis.[1]

Economically disadvantaged students were 30% less likely to enroll in an IDEA charter school;
Students taking the English-language TAKS test in reading were 2.65 times more likely than students taking the Spanish-language TAKS to enroll in an IDEA charter school;
Students passing the TAKS mathematics test were almost two times more likely to enroll in an IDEA Charter school than students who did not pass;
Students passing the TAKS reading test were almost two times more likely to enroll in an IDEA Charter school than students who did not pass;
Students taking the regular TAKS test (indicating no special needs) in mathematics were 3.3 times more likely to enroll in an IDEA charter school than students not taking the regular TAKS test (indicating special needs);
Students taking the regular reading TAKS were four times more likely to enroll in an IDEA Charter school than students not taking a regular TAKS test.

Based on my recent work, I have analyzed six cohorts of 5^th grade students entering IDEA charter schools as compared to students enrolled in schools served by IDEA charter schools. School designated as being in the same local area as IDEA charter schools were located in the same zip code as an IDEA charter school or in a zip code area contiguous with the zip code area in which an IDEA school was located. The school also had to be in a zip code that sent at least one student to an IDEA charter school. Finally, students already enrolled in an IDEA charter school in the 5^th grade were excluded from the analysis.

Table 1 shows that students entering into IDEA charter schools had greater math and reading performance on the TAKS than students that remained in public schools and that lived in the very same zip codes served by IDEA.

TABLE 1: Differences in TAKS Mathematics and Reading Outcomes for Students Entering IDEA Charter Schools in the 6th Grade and Students Remaining in Public Schools from the 5th to 6th Grade for the 2005 through 2010 Cohorts of 5th Grade Students in Zip Codes Served by IDEA Charter Schools

SOURCE: TEA student level data; Analysis: Ed Fuller

The first portion of the table shows that a greater percentage of 5^th grade students entering the 6^th grade into an IDEA charter school had passed the TAKS math and reading tests. This was true for each of the six cohorts.

The second section of Table 1 shows the same result for commended status—a greater percentage of students entering an IDEA charter school in the 6^th grade had achieved commended status in the 5^th grade. This was true for all but one cohort in reading.

The third section of Table 1–and the most important result–shows that students entering IDEA charter schools had TAKS math and reading scores that were, on average, at least 0.2 standard deviations greater than students entering local public schools. Thus, students entering IDEA charter schools already performed substantially better than students entering public schools. This is a truly substantial difference as most educational interventions improve performance along the lines of .1 standard deviations or less.

So, when the AISD authors show that IDEA students outperform other students, they don’t examine why this might be the case. Table 1 above suggests that the reason may simply be that IDEA has higher performing students to begin with which would require other schools to have greater growth than IDEA just to appear equal in a scatterplot analysis.

As shown in Table 2, a lower percentage of economically disadvantaged students and special needs students entered IDEA charter schools as compared to local public schools. In addition, a lower percentage of 5^th grade students entering IDEA charter schools in the 6^th grade took the Spanish language TAKS test or were exempted from TAKS because of limited English proficiency (LEP).

Table 2: Percentage of 5^th Grade Students Identified as Economically Disadvantaged, Having Special Needs, Taking the Spanish-Language TAKS, or Being Exempted from TAKS for Limited English Proficiency by Enrollment in the 6^th Grade in an IDEA Charter School or a Local Public School

SOURCE: TEA student level data; Analysis: Ed Fuller

Thus, when compared to local public schools, a substantially different set of students enters IDEA charter schools than enters local public schools in the same area. This is important because having students with a diverse set of educational needs makes school improvement more difficult and expensive. Further, the authors simply did NOT address this finding in their paper. In fact, the authors have not bothered to purchase the data to examine these findings on their own.

What should be of concern to Trustees is whether IDEA will operate in the same manner in Austin as it has in the Rio Grande Valley. Will a substantially different set of more advantaged students enroll in the IDEA Austin charter or will IDEA Austin actually serve the same students as currently served by Allen and Eastside Memorial.

2) The authors also ignored my finding that students scoring at or below average on TAKS were more likely to leave an IDEA charter school than a local public school. Specifically, I found that students scoring at or below the state average on TAKS in IDEA schools were 85% less likely to remain at the same school than their peers in public schools in the IDEA charter market. To reiterate—9^th grade students in IDEA Charter schools scoring average or below-average were 85% less likely to remain at the same school than their peers in public schools in the same neighborhoods.

Again, the AISD authors simply completely ignore this finding. This finding is critically important because it shows that lower performing students leave IDEA charter schools at higher rates than students in public schools.

II. The Response Has Incorrect Use of Data and Analytic Techniques

The authors rely on apples-to-oranges comparisons in their paper. For example, in trying to refute my analysis of the disappearance of students from 9^th grade to 11^th grade in IDEA charter schools, the authors rely on student mobility data from TEA. Yet, a student is designated as mobile if the student has not attended the school for 83% of the school year. So, students who are chronically absent, move to the school after the first six weeks of schools, or who leave the school before the end of the last six weeks of the year are included in the designation. This is completely different data than what I used and is simply an apples-to-oranges comparison. Any comparison is frivolous because the data simply are not measuring the same construct.

The authors also rely on scatterplots. There are multiple issues with these scatterplots.

First, the plots rely on one cohort of students. This cohort could be an outlier. More cohorts should be included.

Second, scatterplots do not “level the playing field” when examining student outcomes. The authors contend that by comparing outcomes across schools with varying levels of economically disadvantaged students, the comparisons then become an “apples-to-apples” comparison. But any introductory statistics book will tell you that scatterplots are a tool to examine simple relationships and that conclusions or comparisons about performance can only be used by employing far more sophisticated statistical techniques. Scatterplots are used in the beginning steps of analyses, but there are many, many subsequent steps that must be taken before reaching a conclusion. What the authors needed to do, but failed to do, was to perform many other scatterplots to examine the other factors influencing achievement, then use all of the factors in a regression analysis to determine which associations were statistical significant. The bottom line is that scatterplots suggest interesting relationships that need further investigation, but they don;t allow for valid comparisons of effectiveness or outcomes. So, one cannot conclude from the scatterplot that IDEA charter schools or any other school in the analysis is any more or less effective than any other school. Such conclusions need to be based on regression analysis results that control for a large number of the variables affecting achievement. This is basic statistical analysis 101.

Third, along these lines, the authors do not consider prior scores in their scatterplots nor do the scatterplots provide information on the value-added of attending an IDEA charter school. Prior scores are the strongest predictor of achievement levels, yet the authors completely ignore prior achievement. Note that my analyses show that students entering IDEA charter schools already outperform their peers entering regular public schools on TAKS. That the students outperform their peers in 11^th grade should not be surprising given that they outperform their peers before entering IDEA charter schools. What the scatterplots do NOT reveal is the value-added by attending an IDEA charter school. The authors’ scatterplots simply reinforce the finding that IDEA charters skim students at the entry point for entering IDEA secondary schools.

Fourth, the scatterplots use combined free-/reduced-price lunch as the measure for economically disadvantaged status. The percentage of students on free lunch has a stronger negative effect on achievement. The authors did not examine or use such data which is available for free from the TEA website.

Fifth, the authors do not look at growth. Status measures such as percent passing or percent commended do not provide any information about the growth of students while in an IDEA charter school.

Sixth, the authors do not control for students leaving IDEA charter schools. As noted above, lower performing students are more likely to leave IDEA charter schools. Thus, the 1th grade passing and commended rates may be artificially inflated as compared to other schools since the disappearance rate for lower performing students is greater in IDEA charter schools than for surrounding local schools.

Seventh, it is curious that no plots of SAT are included. Dr. Marder has created a number of scatterplots that generally show charter schools underperform other schools. Included in this set of scatterplots is one that examines the percentage of economically disadvantaged students and SAT scores. Why was this scatterplot not included? I include my own set of scatterplots below that examine the relationship between the percentage of students achieving college-readiness status and average school-level SAT scores for the graduating class of 2008, 2009, and 2010 (2011 is not yet available from TEA).

As shown in each figure, the IDEA charter school performed below where the school was expected to perform (the line in the figure indicates where the school is predicted to perform on the SAT based on the percentage of students achieving college-readiness status). If the authors chose to rely on scatterplots, why were these scatterplots not made available? My scatterplots below were investigated using regression analysis and additional variables related to SAT scores such as the percentage of students taking the SAT. In each instance, IDEA schools had lower SAT scores than predicted by the regression analysis after controlling for a host of other variables.

Figure 1: Relationship between Percentage of Students Achieving College-Readiness Status and School Average SAT Scores (2008)

SOURCE: AEIS data from the TEA website; Analysis: Ed Fuller

Figure 2: Relationship between Percentage of Students Achieving College-Readiness Status and School Average SAT Scores (2009)

SOURCE: AEIS data from the TEA website; Analysis: Ed Fuller

Figure 3: Relationship between Percentage of Students Achieving College-Readiness Status and School Average SAT Scores (2010)

SOURCE: AEIS data from the TEA website; Analysis: Ed Fuller

Again, these scatterplots were investigated through regression analysis and found to be representative of the findings from the regression analyses.

What this suggests to me is that the students who remain in IDEA charter schools performed well on TAKS, but underperformed on the SAT. This may be a result of teaching to the test as some teachers in IDEA have stated in emails to me. In fact, the poor performance of IDEA students in college supports this possibility.

As shown in Table 3, IDEA students perform at the level of the magnet schools in the Rio Grande Valley on TAKS related outcomes, but perform worse than any other school in the Rio Grande Valley in terms of performance in college. Indeed, almost 40% of IDEA students earned less than a 2.0 GPA in college–in essence, these students are “flunking out” of college. Now, one reason for this may be that IDEA sends all graduates to college. But clearly not all graduates were well-prepared for college. Perhaps other schools are more realistic about the academic ability of graduates and send some to college, some to technical schools, and some directly into the workforce.

Another interesting issue is that IDEA sends a lower percentage of students into 4 year colleges than schools with similar TAKS outcomes. Indeed, while IDEA and the two South Texas magnet schools have very similar rates of college-readiness, IDEA sends a significantly lower percentage of students into 4 year colleges than these schools. This may be due to the economic status of students or IDEA students entering private colleges or going out-of-state to attend college, but IDEA never presented any of this information to the AISD Board of Trustees to explain these rates.

Table 3: TAKS Performance, College Readiness, Attendance in 4 Year Colleges, and College Performance of Students in 4 Year Colleges for IDEA and Selected School Districts in the Rio Grande Valley (Key for titles is below the table)

KEY:

% > 2200 in 9th grade = the percentage of students in 9th grade on track to achieve college-readiness in 11th grade

% CR= percentage of students achieving college-readiness as defined by TEA.

% 4 Year College: Percentage of graduates entering a 4 year public college in Texas in the fall semester after spring graduation from the Texas Higher Education Coordinating Board

% < 2.0 GPA = the percentage of students entering a 4 year college that had a GPA of less than 2.0

The authors also argue that IDEA should be compared to AISD schools. Yet, AISD schools and IDEA charter schools operate in a completely different environment and context. Only through regression analysis that controls for such factors could any comparison be made. Even then, controlling for the all the differing factors between Austin and the Rio Grande Valley would be extremely difficult. The best comparison to use when examining the impact of IDEA charter schools is the set of schools in the same area as the IDEA schools themselves.

But let’s look at some comparisons. Below, in Table 4, I compare the student retention rate by TAKS scores for IDEA charter students, students in Hidalgo ISD, and students in Martin Middle School in Austin ISD. Specifically, I examine the retention rates of students in the 6^th grade through 11^th grade. A student is considered retained if he/she remains in the same school district from 6^th grade through 11^th grade (or any grade level in the year the student should have been in the 11^th grade). Less than 50% of the original cohort of 6^th grade students in 2006 remained in an IDEA charter school. Only 30% of the lowest performers remained. For 6^th grade students in Martin Middle School, 64% remained in the district and 64% of the lower performing students remained in the district. Further, the difference in retention rates by TAKS scores suggest that the Dr. Marder’s scatterplots that show IDEA outperforms other schools with similar poverty rates may simply be because the lower performing students in IDEA charter schools simply leave IDEA charters altogether. Is that the strategy Austin ISD wants to employ to improve student outcomes?

Table 4: Percentage of Students Remaining in the Same District from 6^th Grade (2006) to 11^th Grade (2011) by TAKS Mathematics Scale Score for IDEA Charter Schools, Hidalgo ISD, and Martin Middle School

Next, let us compare the percentage of special education students in IDEA elementary schools and East Austin elementary schools. Clearly, IDEA enrolls a lower percentage of special education students than schools in East Austin.

Figure 4: Percentage of Special Education Students in IDEA Elementary Schools and East Austin Elementary Schools in 2011

How about the percentage of bilingual students? As shown in Figure 5, IDEA enrolls a lower percentage of bilingual students than the majority of East Austin elementary schools.

Figure 5: Percentage of Bilingual Education Students in IDEA Elementary Schools and East Austin Elementary Schools in 2011

The same holds true for economically disadvantaged students as shown in Figure 6.

And, finally, let’s compare student GROWTH for IDEA Academy elementary school and East Austin elementary schools using z-scores of growth provided by the FAST system on the state comptroller’s website. A z-score of zero indicates the state average. A positive z-score indicates performance greater than the state average. A negative z-score indicates performance worse than the state average. Z-scores generally range from -3 to +3. In math and reading combined, as shown in Table 5, IDEA Academy only slightly outperforms the East Austin average and greatly under performs many East Austin elementary schools. IDEA Academy performs substantially below the East Austin average in mathematics while substantially outperforming the East Austin average in reading.

In fact, a number of East Austin schools substantially outperform IDEA Academy in both subject areas. In fact, Graham Elementary is one of the best elementary schools in the entire state in terms of student growth. Blanton, Metz, Overton, and Norman also perform in the top 15% of elementary schools across the state.

Table 5: Average Three-Year (2008, 2009, 2010) TAKS Growth in Reading and Mathematics for East Austin Elementary Schools and IDEA Academy in Z-Scores

SOURCE: FAST data from Texas State Comptroller

Why didn’t AISD simply get the principals and teachers from the five top-performing elementary schools in East Austin together with the lower performing elementary schools in East Austin such as Allen and have them collaborate to improve all schools? Collaboration and teamwork have been shown to be effective in turning around schools. Using charter schools to turn around schools has proved less effective, particularly because the charter schools typically end up enrolling a different set of students than the original public school. This would have been amenable to the East Austin community and has a much stronger research base than charter schools taking over public schools.

III. Selective Use of Quotes and Quotes out of Context

The authors take my comments out of context or use selective quotes that do not provide a full representation of my points.

For example, the authors take my “underserved student” quote out of context. In context, my conclusion is that IDEA schools do not serve a greater proportion of underserved students than schools in the same local area served by IDEA charter schools. IDEA students are only underserved in relation to students across the entire state. In fact, every single school in Region I serves a greater percentage of underserved students as compared with the entire state. Yet, students in Dallas and Austin cannot attend an IDEA charter school. Thus, the relevant comparison to IDEA charter schools is schools in the same local area as IDEA charter schools. When compared to local schools, IDEA enrolls a lower percentage of underserved students. If this trend continues in Austin, then IDEA will enroll a population that is significantly different—and more advantaged–than other schools in East Austin.

In addition, the authors do not fully complete my quote on peer effects. Researchers have consistently found that peer effects have an important influence on student achievement. Students enrolling in schools with higher performing students will be positively affected and students enrolling in schools with lower performing students will be negatively affected. Schools can create positive peer effects through positive behaviors such as high expectations and low tolerance for misbehavior or through negative behaviors such as skimming students and losing lower performing students. My analysis shows that IDEA likely uses both the positive and negative behaviors to create positive peer effects. If that is acceptable to the AISD Board of Trustees and the Superintendent, then statements should be made that enrolling more advantaged students and losing lower performing students are acceptable strategies for school improvement.

Conclusion

The authors’ paper is fraught with serious problems which render the paper less than helpful. The authors do not have access to the student level data needed to fully explore the effectiveness of IDEA charter schools and they rely on statistical techniques that are simply inappropriate to use to reach the conclusions that they reach.

If Austin ISD wants to know the complete picture of the effectiveness of IDEA charter schools (and the district should have fully investigated the efficacy of IDEA before awarding a contract), the district should hire an experienced education researcher and provide the researcher with access to the student level data necessary to completely evaluate IDEA charters.

[1] Logistic regression is the accepted statistical tool used when outcomes are binary such as a student entered or did not enter an IDEA charter school. The authors performed no such statistical analyses nor did they have access to the data hat would even allow them to do such an analysis.

Where did all those 6th graders go? IDEA and Martin MS Comparison

Dr. Ed Fuller — Sun, 18 Dec 2011 13:11:06 +0000

This posts looks at the most recent cohort of 6th graders enrolling in IDEA Charter Schools and Martin MS in Austin ISD. Why start with 6th graders? Because IDEA Charter Schools are separated into elementary schools (K-5) and secondary schools (6-12). So, theoretically, students enter 6th grade and remain in IDEA Charter Schools until graduation and then 100% of them enter college.

This evidence will be criticized because it is only one cohort, but it is the latest cohort available. Thus, it reflects the most current evidence we have on the success of IDEA Charter Schools.

This analysis extends my previous analyses by including middle school grade levels in addition to grades 9 through 11. This extension is important because any school level growth measures include all grades in the school. In the case of IDEA secondary schools, the grades served include grades 6 through 12.

Tables 1 and 2 show the number and percentage of 6th grade students in the 2005-06 school year taking the TAKS mathematics and reading tests that were still enrolled in the same district in the 2010-11 school year in any grade. The analysis compares three entities: IDEA Academy, Hidalgo ISD, and Martin MS in Austin ISD.

I chose Hidalgo ISD as a relevant comparison to IDEA in the Rio Grande Valley (Hidalgo is the headquarters for IDEA) and Martin MS in Austin ISD since it is the feeder pattern that IDEA would replace in Austin ISD.

Table 1: Number and Percentage of 6^th Grade Students (2005-06) Remaining in the Same District through the 2010-11 School Year by 6^th Grade TAKS Mathematics Score

Table 2: Number and Percentage of 6^th Grade Students (2005-06) Remaining in the Same District through the 2010-11 School Year by 6^th Grade TAKS Reading Score

CONCLUSIONS

Two conclusions can readily be made based on this data. The first conclusion is that IDEA Charter Schools retained a much smaller percentage of students in the same district as compared to both Hidalgo ISD and Martin Middle School. Indeed, only 47% of the IDEA Charter School 6^th grade students remained enrolled in any IDEA Charter School in 2010-11. In comparison, about 77% of the Hidalgo ISD students remained in Hidalgo ISD while 64% of the Martin MS students remained in AISD.

The second conclusion is that remaining in an IDEA Charter School is correlated with TAKS scores. Specifically, the greater the TAKS score, the more likely the student remained in IDEA. The same trend appeared with Hidalgo ISD students, but the difference in the percentage of students remaining in the district between low- and high-performers was not nearly as great as for IDEA Charter Schools. For Martin MS, a greater percentage of the average and slightly above average performing students remained in Austin ISD. The lowest-performing and high-performing were more likely to leave AISD than the students between those two groups.

This second conclusion is critically important because IDEA growth scores as calculated by FAST are very likely to be positively affected by the disappearance of the lowest performing students through peer effects. In fact, research is fairly consistent in this area, with almost all researchers finding positive effects of students being in the same classroom and school with high-performing peers.

DISCUSSION

This data is just one more piece of evidence that strongly suggests IDEA Charter Schools are not nearly as successful as IDEA Charter leaders and Austin ISD leaders claim them to be. Are they successful? In some areas, IDEA Charter Schools are successful. But, for every successful outcome, there appears to be data that suggests a troubling reason for that success.

Achievement Gap in AISD and IDEA Charter Schools

Dr. Ed Fuller — Sun, 18 Dec 2011 02:50:47 +0000

The achievement gap is generally considered to refer to the gap in achievement between two groups of students as measured by test scores. Other measures of the achievement gap include dropout rates, high school completion rates, and college-going rates. Generally, however, the phrase refers to test scores.

One rationale for expanding charter schools is that such schools can help close the achievement gap. For example, the IDEA website claims, ” Closing the achievement gap is no longer a phrase, but a reality at IDEA Public Schools.” Yet, no data is provided that substantiates this claim. In fact, a search of the Texas Charter School Association, IDEA Charter school website, and a general google search for “achievement gap Texas charter schools” and “Texas charter schools closing achievement gap” did not yield a single result of the closing of the achievement gap. Hopefully, I simply missed the documentation and someone can point me to the evidence.

Because I could find no evidence, I decided to use student-level data from the Texas Education Agency to examine the achievement gap between students not participating in the federal free-/reduced-price lunch program and students participating in the federal free-/reduced-price lunch program in mathematics and reading pm the Texas Assessment of Knowledge and Skills (TAKS) in the 2010-11 school year. Further, I examined the data for all grade levels and for all schools enrolling at least 60% economically disadvantaged students. I chose the 60% cut point because all IDEA Charter Schools enroll at least 60% economically disadvantaged students.

Most organizations–including state education agencies and school districts–calculate the achievement gap incorrectly. Take for example, Austin ISD. In the AISD “State of the District” report, AISD claims the achievement gap has decreased by pointing out that the percentage of students passing the TAKS tests between White and non-White students has decreased over time.

Yet, as Daniel Koretz clearly explains in his book entitled “Measuring Up: What Educational Testing Really Tells Us,” using the percentage of students passing/proficient always provides an inaccurate assessment of the achievement gap. Other experts in this area concur. I strongly suggest reading his very easy to digest book top understand why we must use scale scores and, better yet, z-scores to assess the achievement gap.

Below I present the achievement gap in z-scores which are very similar to standard deviations.* A positive number indicates that not economically disadvantaged students outperform economically disadvantaged students and a negative number would indicate that not economically disadvantaged students were outperformed by economically disadvantaged students. The magnitude of the number indicates the size of the achievement gap. Thus, a result of .25 indicates a smaller achievement gap than a result of .75. The closer to 0 the result, the smaller the achievement gap.

As shown below, AISD has not closed the achievement gap. but neither has IDEA Charter Schools, nor all Texas schools. In most, but not all cases, the gap for IDEA schools is smaller than for AISD schools. Yet, the gap remains relatively large for IDEA Charter schools at most grade levels.

Table 1: TAKS Achievement Gaps in Mathematics and Reading (2011) for Austin ISD, IDEA Charter Schools, and all Texas Schools with at Least 60% Economically Disadvantaged Students

DATA SOURCE: Student-level TAKS data from TEA; Analysis: Ed Fuller

So, what can we conclude from this table?

First, IDEA Charter Schools have NOT closed the achievement gap between low-income and not low-income students and are not particularly close to doing so.

Second, IDEA Charter Schools generally have a smaller achievement gap than AISD schools. However, one would need to investigate the effect of students leaving each set of schools to further substantiate this claim.

Third, despite the claims of IDEA leaders that poverty does not matter and does not influence achievement, poverty clearly has a negative effect on achievement–even in IDEA Charter Schools. If poverty did not matter, there would be no achievement gap in IDEA Charter Schools. That poverty affects achievement is one of the most consistent findings in education research. It is next to impossible to find any quality study that finds otherwise.

* I calculated the gap by first calculating the z score for all students for each grade level with TAKS results. The z-score was calculated by calculating the mean, then dividing by the standard deviation. The achievement gap was then calculated for IDEA Charter Schools, AISD schools, and all Texas schools by subtracting the aggregate z-score for economically disadvantaged students from the aggregate z-score for not economically disadvantaged students, then dividing by the standard deviation for all students.

College-Readiness of IDEA and other High Schools in the Rio Grande Valley

Dr. Ed Fuller — Mon, 12 Dec 2011 01:48:43 +0000

While the last post looked at elementary school performance, this post will examine college-readiness performance, the transition to post-secondary institutions of higher education, and performance in four-year universities. Again, all data used in this review are publicly available. This time, data is either from the Academic Excellence Indicator System (http://ritter.tea.state.tx.us/perfreport/aeis/) section of the Texas Education Agency, the High School to Higher Education Data section on the Texas Higher Education Coordinating Board website, or from student-level data purchased from TEA.

Table 1 below has four areas:

–percentage of first-time 9th grade students that had TAKS reading and math scores of at least 2200;

–percentage of students meeting the TEA college-readiness standards (TAKS scale score of 2200 or greater OR 500 on the reading/math SAT exam and a total score of at least 1070 OR >= 19 on the reading/math ACT and >=23 composite ACT);

–percentage of students advancing from high school to college in the fall semester after spring graduation; and,

–percentage of students in four-year universities that had less than a 2.0 GPA during the first year of college.

The percentage of 9th grade students scoring at 2200 or above on BOTH the mathematics and reading TAKS in the 8th grade is included because it is the single most important input in explaining college-readiness. Indeed, the correlation between this percentage and the TEA college-readiness standard is greater than .500.

As shown in the yellow and pink rows, the IDEA Secondary School (the school changed names during the time spans under examination, but continued enrolling the same students) had incoming 9th grade students that were extremely similar to the South Texas Health and Science Academies–both magnet schools of choice in the Rio Grande Valley. Thus, these two schools serve as the best comparison schools to IDEA.

TABLE 1: College Readiness, High to College Transition, and College Performance for IDEA Secondary School and Selected Rio Grande Valley High Schools

Also note that only McAllen Memorial (MM)had incoming students even remotely as well-prepared as the incoming 9th grade students in IDEA. And, since the percentage of students scoring at or above 2200 when entering high school is highly predictive of the percentage of students achieving TEA’s college-readiness standard, IDEA had the proverbial “leg up” on almost all other public high schools in the Rio Grande Valley.But MM certainly appears to add far more value in terms of moving students into the college-readiness range and, while still lagging in the percentage of students entering a 4-year college, certainly has far better prepared students for college-level work.

The results get more interesting, however, when we get to section three: percentage of students entering four-year colleges. While IDEA sent a greater percentage of students to 4-year colleges than other public schools, IDEA sent a lower percentage than the two schools with similar student characteristics.

Finally, and most intriguing, is that 38% of the graduates of IDEA that entered four-year colleges had less than a 2.0 GPA. In short, almost 40% of the IDEA graduates were at serious risk of “flunking out” of college. This was a greater percentage than all the other schools included in the list and every school in the Rio Grande Valley as far as I could tell (it is quite laborious scrolling through the THECB pdf). In fact, I purposefully selected some of the schools with the LOWEST percentage of students meeting the TEA college-readiness standard to see if they would have a higher percentage of students failing college-level coursework. Yet, even these schools had lower percentages of students with a GPA lower than 2.0.

So, what can we learn from this data? Well, the data about IDEA is somewhat mixed.

Yes, IDEA has a high percentage of students designated as college-ready, but that is largely explained by the characteristics of students entering the school in the 9th grade. IDEA performs just slightly better than predicted from the incoming 9th grade scores.

Yes, IDEA has a high percentage of students entering 4-year colleges. But, the percentage is below schools with similar student achievement.

Finally, many IDEA students seem woefully under-prepared for college even though the majority of the students do not enter a Tier I university. In fact, for the last cohort available (the graduates of spring 2009), 44% of students earned a GPA lower than 2.0. That strongly suggests that IDEA is not preparing students very well for success in 4-year colleges.

One possible explanation for this is that IDEA focuses on preparing students for the TAKS–engages in “teaching to the test”– rather than preparing students to think, analyze, write, and question as is expected of college students. Other explanations are certainly viable, but no one has studied this in-depth. Someone certainly should investigate and determine the reason behind the poor college performance of IDEA graduates.

So, we have to ask, what is the value-added of IDEA? Are they able to add value to the students starting the 9th grade? The data here suggests they do not add value–or at least not much anyway.

Let’s look closer.

Let’s start with looking at the relationship between the percentage of incoming 9th grade students scoring at or above 2200 on both the math and the reading TAKS in the 8th grade and the percentage of students meeting the TEA college-readiness standard for both mathematics and reading. For both 2008 graduating class, IDEA slightly outperformed the level at which they were predicted to perform based on incoming student achievement.

FIGURE 1: Percentage of Incoming 9th Grade Students (2005) at “College-Ready” Standard in 8th Grade (2004) and the Percentage of Graduates (2008) meeting the TEA College-Readiness Standard

In 2009, as shown in Figure 2, IDEA performed marginally better than the level at which it was predicted to perform based on the TAKS scores of incoming 9th grade students. Essentially, it performed as well as expected.

FIGURE 2: Percentage of Incoming 9th Grade Students (2006) at “College-Ready” Standard in 8th Grade (2005) and the Percentage of Graduates (2009) meeting the TEA College-Readiness Standard

In 2010, as shown in Figure 3, IDEA performed at the level at which it was predicted to perform based on the TAKS scores of incoming 9th grade students. Interestingly, each successive cohort regressed toward where the line of prediction. In other words, the first cohort performed better than predicted, the second cohort predicted slightly better than predicted, and the third cohort performed as predicted.

FIGURE 3: Percentage of Incoming 9th Grade Students (2007) at “College-Ready” Standard in 8th Grade (2006) and the Percentage of Graduates (2010) meeting the TEA College-Readiness Standard

Let’s turn to SAT scores. It is very difficult to fairly compare SAT scores, so this analysis must be interpreted with a great deal of caution. The most important variable explaining average school SAT scores is the percentage of students taking the SAT. Generally, the greater the percentage of participating students, the lower the scores, all other factors being equal. Unfortunately, IDEA did not correctly report to TEA the percentage of students taking the SAT for any of the three cohorts. Thus, I cannot include that variable in any analysis.

However, let’s examine the relationship between the percentage of students in the graduating classes in 2008, 2009, and 2010 and the average SAT scores of those graduating classes. The following three figures display these relationships, with 2008 first and 2010 last. One would suspect a fairly strong relationship between the two measures, especially since scoring above 500 on the SAT mathematics or reading scores and above 1050 for the composite score achieves college-ready status. The following three figures confirm this supposition. Indeed, the correlations for each of the three years are .600 or greater.

Note that in each of the three years, IDEA performs worse than predicted based on its percentage of college-ready graduates.

FIGURE 4: Percentage of 2008 Graduates Designated as College-Ready and Average SAT Scores for 2008 Graduates

FIGURE 5: Percentage of 2009 Graduates Designated as College-Ready and Average SAT Scores for 2009 Graduates

FIGURE 5: Percentage of 2010 Graduates Designated as College-Ready and Average SAT Scores for 2010 Graduates

However, remember we do not know the percentage of students taking the SAT and inclusion of this variable could result in different outcomes for IDEA. Well, let’s assume that every single IDEA student took the SAT. This would be the most conservative assumption to make since it would be in favor of IDEA in any analysis.

Yet, when I employed regression analysis to explain average school SAT scores with the percentage of poor students, the school percentage of students scoring at 2200 in math and reading in 8th grade for incoming 9th graders, school size, and the percentage of graduates achieving college-ready status, IDEA always performed significantly below where the regression would predict they would perform. This was true for each of the three years.

Why IDEA under performs in this area, we do not know. However, as mentioned previously, one possible explanation is that IDEA is engaging in “teaching to the test” strategies that inflate scores and reduces the connection between the scores and the knowledge and skills possessed by students (see Koretz, 2008; Measuring Up: What Educational Testing Really Tells Us).

In closing, the data on IDEA Charter schools again raises serious questions about the claims of superiority made by IDEA leaders. Much, much more work needs to be undertaken to answer these questions. IDEA could shed some light by sharing data so researchers could examine these issues and having researchers interview and survey students in IDEA schools. This is the type of research EVERY school organization should be doing.

IDEA Academy vs East Austin Schools on Achievement and Student Characteristics

Dr. Ed Fuller — Fri, 09 Dec 2011 21:14:34 +0000

This study uses data from the State of Texas to compare three year averages in student characteristics and performance between the one IDEA Charter elementary school with data available over the three year time frame from 2007-08 through 2009-10. The data are all available from either the Texas Comptroller’s FAST website (http://fastexas.org/) or the Academic Excellence Indicator System (http://ritter.tea.state.tx.us/perfreport/aeis/) website from the Texas Education Agency.

Before getting to achievement, let’s examine the student characteristics of East Austin schools and IDEA Academy. Unfortunately, we cannot examine prior academic ability since there is no measure prior to the third grade.

First, let’s compare East Side elementary schools to IDEA Academy in terms of the percentage of economically disadvantaged students.

FIGURE 1: Average* Percentage of Economically Disadvantaged Students

for IDEA Academy and East Austin Elementary Schools