Validity Evaluations

edCount specializes in applying an argument-based approach to validity evaluation, designed to help clients articulate and evaluate the meaning of their assessment scores in the context of their entire education system. After helping clients articulate a theory of action describing their assessment system's purpose and underlying logic model, we develop a plan to assess the validity of the model. This may suggest a variety of different studies that are necessary to establish evidence of the validity of an assessment score's meaning and use. edCount's involvement may end with developing a plan or we may conduct any one of the many types of studies deemed necessary to establish validity in the context of the assessment system.

Some studies we conduct include:

Performance Level Descriptor Studies

edCount helps state-level clients compare their descriptions of different student performance levels (e.g., "advanced," "proficient," "basic") with student performance data, to ensure that their performance level descriptors (PLDs) accurately describe student ability, and to ensure the cut scores that sort students into these levels are appropriately placed.

Consequential Validity and Instructional Use Studies

edCount designs and conducts studies to help clients assess and evaluate the impact assessment outcomes have on their schools and classrooms. The information collected from these studies can inform changes to improve instruction and decision-making processes.

Cognitive Labs

edCount designs and conducts cognitive lab studies, in which students describe their thought processes and thinking as they respond to assessment items, to collect information about whether assessments are eliciting the intended cognitive processes. These studies can help state and local entities understand why students may struggle with certain items or content areas on an assessment, and, as a result, provide feedback about ways to improve instruction.

Learner Characteristics Studies

edCount helps clients identify and evaluate the characteristics of the population of students who participate in various assessments by disseminating, collecting, and interpreting survey data.

Standard Setting and Cut Score Studies 

edCount helps clients conduct standard setting studies, establish cut scores, and validate existing cut-scores. At edCount workshops, experienced teachers provide input to ensure that differences in students’ scores on the assessment correspond to meaningful differences in performance on relevant academic standards. 

edCount @work

edCount Completes Validity and Process Evaluation for Development of National Center and State Collaborative (NCSC) Alternate Assessment System

The National Center and State Collaborative (NCSC) was a project led by five organizations and 24 states (18 core partner states and 6 Tier II states) to build an alternate assessment based on alternate achievement standards (AA-AAS) for students with the most significant cognitive disabilities. edCount led NCSC's validity evaluation work group, one of four organizational pillars for the project's activities.

Read More

Alternate Assessment Validity Evaluation Planning

edCount has helped the state education agencies (SEAs) in the District of Columbia, Hawaii, Puerto Rico, and the U.S. Virgin Islands develop plans for evaluating the validity of their alternate assessments based on alternate achievement standards (AA-AAS).

Read More

English Language Proficiency Assessment Validity Evaluation Planning

Through the federally-funded Evaluating the Validity of English Language Proficiency Assessments (EVEA) project, edCount has helped the state education agencies (SEAs) for Washington, Oregon, Montana, Indiana, and Idaho develop comprehensive frameworks (also called interpretive arguments) for considering the meaning and utility of scores from English language proficiency assessments (ELPAs).

Read More