Catholic Mission

Talking Points for Interpreting Data from the HEA Title II Passing Rate Reports

This is the first report that provides each institution with performance of a known cohort of program completers.  Each institution can evaluate their data in the context of the characteristics of their student population and the strengths and weakness of each licensure program. The data can provide the stimulus for conversation among each college's various stakeholders:  chancellors, deans of colleges of education, deans of arts and sciences, and faculty who participate in the education of teachers.  A longitudinal study of several years of data would be a useful measurement indicator of the effectiveness of program improvements that teacher preparation programs enact as a result of this report. That is a valuable use of this data and should be encouraged.  However, it is also possible that data can be misused or misinterpreted. What follows is a discussion of some important reasons to be cautious when interpreting HEA Title II data.

While there will be numerous constituencies quick to draw inferences and make generalizations about the institutional and state HEA Title II reports, in fact there is no common yardstick to enable one to compare different groups of people taking different tests with different qualifying scores.  Comparisons between states, while tempting, are unfounded because there is no standard for comparing state licenses.  There are states that offer a single comprehensive license for all teaching areas and there are others that offer over 100 different licenses. As a result, the testing requirements differ from state to state.  Those states that do adopt the same tests generally require different qualifying scores.  Passing rates, which indicate the percentage of examinees that meet or exceed the qualifying score for the particular assessment set by the state, vary based on each state's qualifying scores.  For the same test, states with lower qualifying scores will most likely have higher passing rates.

Within the same state, comparisons made between institutions are equally unsubstantiated because each institution prepares students for differing licenses utilizing different testing requirements.  It is uncommon to find two institutions within the same state with identical teacher preparation programs.  The difficulty of comparing institutions within a state is also affected by the omission from the state report of data for smaller teacher preparation programs that prepare fewer than ten completers.  These institutions' data are not available for public scrutiny, while their individual student scores are included in the average state passing rates for each assessment.  The other programs meeting the"minimum ten rule" - that is programs with 10 or more competers  -  are compared to this state average performance.

While the single assessment pass rate, aggregate area pass rate and summative assessment pass rate will be reported for teacher preparation programs, it is the summative assessment pass rate that will be used to rank by quartiles the teacher preparation programs in a state.  (A quartile is a distribution of data that has been divided into four sets.)  The summative assessment pass rate potentially represents the rate of an institution's program completers who become licensed.  (However, this statistic is inflated whether they completed all testing requirements for a license.)

What are other reasons to be cautious when interpreting or comparing institutional data?

1. The size of a college's testing population impacts its summative passing rate.  Since passing rates are reported as percents, a careful comparison should include a review of the size of the populations to be compared.  Example 1 below illustrates this.



Example 1 College 1 has 100 program completers and 2 fail a licensure test, so their summative passing rate is 98%. College 2 has 10 program completers and 2 fail a licensure test, so their summative passing rate is 80%.

While the number of program completers who were unsuccessful is equal at both colleges, the marked differences in the summative pass rate will very likely result in College 2 being ranked in a lower quartile.

2. Ranking the colleges by quartiles creates a distribution in which the summative assessment passing rates are rank-ordered into groups, in the case quartile.  For esample, a hypothetical state with 100 colleges would distribute the colleges into four quartiles with 25 colleges in each quartile.  If the same state's licensure tests consist only of basic skills tests, the first quartile could have 25 colleges all reporting 100 percent passing rates.  (The second quartile could have 25 colleges could report a distribution of summative assessment passing rates between 98 and 99 percent.) The third quartile could contain a distribution of colleges with summative assessment passing rates between 96 and 97 percent. The difference between the first and fourth quartile is five percentage points. Hence being located in a lower quartile may not be meaningful from a practical perspective. Also the range of summative assessment passing rates in a quartile varies by state and lacks any longitudinal consistency; that is, the range of performances for each state's quartiles will change annually. There is no benchmark for comparing performances.
   
   It is important to identify the range of passing rates attributed to each quartile, but it is equally important to be mindful that these ranges will not be the standard of comparison in future quartile reporting.



3. The number of tests required for a license will affect the summative assessment passing rate. Licenses that require more testing requirements increase the potential for failure and a diminished passing rate.  Teacher preparation programs whose mission is to prepare teachers with an emphasis in special education will have program completers who will be evaluated on as many as five different content tests, while those preparing teachers with an elementary education emphasis may only be reporting a summative assessment passing rate based on a single test requirement for licensure. Again, there is a greater potential for failure when program completers are required to pass more assessments.

Please note that this is not a comprehensive list of potential misinterpretations of the data. It merely indicates the need for caution in using the data for comparison purposes.