The National Research Council has just released a report finding that neither of two key data sources on English learner populations is ideal for allocating state grants under the Title III language acquisition program, but each has its strengths. In light of this result, the NRC recommends that the two be blended to create a weighted count that might produce results that are more accurate than either data source used alone.
By Ellen Forte and Molly Faulkner-Bond for Thompson’s Title I Monitor
Education Series, April 2011, Vol. 16, No. 4 (download the PDF here)
The NRC’s Charge
When No Child Left Behind (NCLB) was enacted nine years ago, state education agencies were required to use one of two sources of data to determine the size of each state’s annual Title III grant: the U.S. Census Bureau’s American Community Survey (ACS) or the state-reported counts based on the number of students tested annually for English language proficiency. The U.S. Department of Education (ED) was required to evaluate these two sources within two years of NCLB’s implementation to determine which provided the more accurate counts of English learners (ELs) by state; the more accurate source would be used to calculate subsequent awards.
The evaluation was not done in the specified time- frame, although the department began to require use of the ACS data in 2005. In 2006, however, the Government Accountability Office issued a report stating that the population counts yielded by each of the two data sources differ, often significantly. Some states have larger populations according to the ACS, while others have larger EL populations according to their self-reported assessment data. This underscored the importance of evaluating the validity of the data from each data source and selecting the more accurate one. This evaluation would help avoid the distribution of excessive funding to some states while underfunding other states relative to their actual EL population sizes.
Findings and Recommendations
ED asked the National Research Council (NRC) to convene a panel to evaluate each data source and make recommendations about which to use. The NRC — a congressionally-chartered nonprofit organization charged with advising the federal government on scientific issues — published these findings in a report released on Jan. 10, 2011. The report, entitled Allocating Federal Funds for State Programs for English Language Learners, identifies ten “dimensions” by which the panel assessed the two data sources: conceptual fit, level of geographic detail, timeliness, quality, cost, fairness, stability, insensitivity to policies and methodological differences, transparency, and comparability.
The pattern of ratings across these dimensions for each of the two data sources indicated that each had some strengths and some weaknesses: at present, neither could be selected as the more accurate and, therefore, the single data source that states should use. Thus, the panel recommend implementation of a weighting ratio for the state data and ACS (i.e., the ACS data carries a weight of 75 percent and the state data 25 percent), but also suggested that ED and states improve the quality of state-reported data, such that, eventually, each data source may carry equal weight (50 percent each) in funding determinations.
In practice, this recommendation means that a state’s total EL population count would be calculated by multiplying the ACS count by 0.75, multiplying the state’s non-proficient count by 0.25, and then adding these two products. For example, if a state’s ACS count was 1,000 and its non-proficient count was 840, its total EL population count would be:
(1,000 x 0.75) + (840 x 0.25) = 750 + 210 = 960.
In the future, with the 50-50 weighting, the two counts would simply be averaged: (1,000 + 840)/2 = 920.
Two Imperfect Data Sources
Ideally, the best data to determine Title III awards would be consistent across all states and over time, collected via a methodology that is rigorous, transparent, and resistant to manipulation or error. The data would attend only to the characteristics that are most indicative of a student’s actual linguistic capabilities under NCLB’s statutory definition of “limited English proficient.” In this way, ED would ensure that states are using criteria to identify their EL subgroup that not only are appropriate, but also are consistent and fair across states and across time.
In reality, however, the two existing data sets each satisfy some of these criteria, but fall short of others. Out of 10 dimensions, the ACS data were deemed to be superior in four: quality, fairness, transparency, and comparability (see Fig. 1, p. 11 of the PDF). State data were superior in one dimension, while three dimensions reflected no significant differences and two dimensions presented unique limitations under each of the two sources.
The panel determined that ACS information is consistent across states and across time and is collected using methods that are rigorous, transparent, and formulated to resist statistical skewing or errors. If the ACS provided the right kind of information about EL students, it would be an ideal source, as it is the most consistent and the most fair.
Unfortunately, ACS population estimates rely upon a measurement tool that may not capture valid information. The ACS form asks: “Is a language other than English spoken in the home?” and if the response is yes, the form asks “How well does household member X speak English?” Respondents may choose one of four response choices for each member of their household: “Very well,” Well,” “Not well,” and “Not at all.” The EL population size is determined based on the number of individuals reported between the ages of 5-21 with a response other than “very well.”
As such, the ACS is no more rigorous or appropriate than a home language survey (HLS), the same kind of informal instrument that most states and districts use to initially flag students who might be ELs. As we know, there are flaws in many HLS instruments and the ACS lacks the follow-up inquiry used to further investigate ELs’ language capabilities.
The ACS instruments asks only two simple questions, to which someone other than the student responds. These responses are subjective (e.g., the respondent may feel that the child speaks well, but may not have a good basis for making that judgment), the questions addresses only the speaking modality of language (not reading, listening, or writing), and the instrument cannot yield an indication of whether the student’s proficiency is of a level necessary for success in academic settings (e.g., the child may speak English “very well” for social interactions and day-to-day life, but may lack the more formal linguistic constructs and forms used in academic settings). In states and districts, students who are flagged based on parental responses to questions like these usually take an English language proficiency test subsequent to the survey, to provide more specific information about the student’s actual linguistic needs. Not every student who is flagged by the survey may need language services, and some students who are initially missed by these questions may yet turn out to be ELs. Lacking this follow-up, the ACS may be prone both to under- and over-identify EL students.
In light of these shortcomings, the panel identified state-reported data, which represent the number of children who took and did not achieve a proficient score on the English language proficiency assessment (ELPA) in the previous year, as the better conceptual fit for the purposes of identifying the Title III population. These data, unlike the ACS, are based on students’ actual linguistic needs and proficiency levels, according to pre- determined standards and criteria set by the state. State data are also guaranteed to represent the public school population only: the ACS uses other responses to triangulate which respondents attend public school (although these are also potentially vulnerable to response errors). While the state numbers represent the previous year’s population and cannot necessarily account for population change due to EL students entering and exiting the district, the panel found that the ACS data suffer from a similar lag.
A major obstacle to the sole use of state-reported data is that the criteria used to determine which students take the ELPA and which students are deemed proficient vary from state to state, and even district to district. These data may also vary across time as states change their identification policies, achievement standards or accountability targets. Some states do not document their identification processes well, if at all, and the exact definition of English proficiency may vary widely. For example, states vary in their methods for combining or weighting student scores for different linguistic domains and some states define proficiency based on multiple years of ELPA data. As a result, the state data are non- standard and non-comparable and, therefore, cannot provide for uniformity in funding distributions across states and across time.
The Policy Implications
Based on these analyses of the two data sources, the panel found that the ACS provides less-appropriate data that are collected in a more uniform fashion, while state-reported data are more valid conceptually, but far less comparable across states. The proposed weighting formula is the panel’s attempt to strike a balance between the strengths and weaknesses of each source while keeping in step with ED’s stated purpose for the Title III formula, which is to distribute funds according to the relative state populations of the target audience.
The panel’s recommendation that states improve practices so that their reported assessment data carry more weight in this formula underscores the panel’s recognition that state data are conceptually more valid as an indicator of the EL population within a state.
At the same time, the ultimate goal of achieving a 50 percent weight for each data source signals a recognition that state data alone may never be sufficiently stable and uniform to support funding determinations. Although current education policy mandates that all states follow similar processes and set definitions for the same terms and concepts (e.g., ‘limited English proficient,’ ‘English proficient’ etc.), it remains unlikely that ED would ever go so far as to require that states use the same definitions, processes and instruments to satisfy these mandates. In other words, although all states must have standards and a definition for “English language proficiency,” ED has not signaled that it intends to impose nationwide standards or a single assessment for all states.
States may nonetheless benefit from improving certain EL-related practices. On a procedural level, improved transparency and documentation in the identification and redesignation of ELs is likely to make life easier for state and district administrators, in addition to meeting this panel’s recommendations to improve the quality of EL data over time. Over and above improving procedures and documentation, there is an increasingly strong argument to be made for actually standardizing certain standards and definitions for EL services across states even in the absence of a federal mandate.
Currently, a student’s status as EL is somewhat contingent upon where he lives, and this does a disservice to all involved. For example, the student may gain or lose services by moving across county or district lines because of the differences in identification processes and proficiency expectations across localities. Schools and districts that receive new students cannot rely on the student’s previous EL status and must spend time and resources testing new ELs according to their own standards. For the same reasons that states have recently begun a move towards adopting common standards and aligned assessments for content areas through ED-incentivized consortia, states have begun and will likely continue to realize that there is no real utility to states having entirely unique definitions of and expectations for English language proficiency.
This is not to say that all states can or should use exactly the same standards or assessment. ED funded two Race to the Top Assessment consortia to encourage the type of competition that yields innovation and better quality products and services. Nonetheless, some commonality and collaboration among states on these fronts would not only benefit students and districts, but also help ensure that states are receiving appropriate award sizes to serve these. Further, more detailed guidance about what states can and should do to improve the rigor of their ELP assessment data would be a helpful next step.
For More Information
The NRC report, Allocating Federal Funds for State Programs for English Language Learners, can be downloaded at http://www.nap.edu/catalog.php?record_id=13090.