Measure Testing Process
Proper testing and analysis are critical to development of a feasible, reliable, and valid measure. The next sections describe types of testing that may be conducted during measure development (alpha and beta testing), the procedure for planning and testing, and key considerations when analyzing and documenting results of testing and analysis, including incorporation of interested party inputs after testing is complete.
The measure developer should conduct initial testing during development (i.e., pilot testing) within the framework of alpha and beta tests. Although considered part of measure testing, alpha testing may occur as early as information gathering and repeated iteratively during development of measure specifications. Measure developers should test early and often.
Alpha and Beta Testing
- Alpha Testing / Formative Testing
-
Alpha testing (i.e., formative testing) is of limited scope since it usually occurs before full development of detailed specifications. Measure developers may conduct alpha testing, particularly regarding feasibility of the concept in the context of the data source, as part of information gathering empirical analysis and may occur concurrently with development of technical specifications as part of an iterative process. Check with clinicians to ensure collection of the data elements occurs as part of the usual care process, either manually or electronically, e.g., wearable devices, and the data elements collect the data needed for measure calculation.
Alpha tests include methods to determine whether individual data elements are available and whether the form in which they exist is consistent with the intent of the measure. Types of testing used in an alpha test vary widely and often depend on the measure’s data source or uniqueness of the measure specifications. Measures that use data sources similar to existing measures may require minimal alpha testing. In contrast, measures that address areas with no development of specifications may require multiple iterations of alpha testing. Measure developers may want to consult with persons and families, e.g., a focus group, to determine if the data elements are meaningful and understandable to them.
For example, an alpha test may include a query to a large, integrated, delivery system database to determine how it captures specific data, where the query originates, and how to express the query. Results can impact decisions about measure specifications.
- Beta Testing
-
Beta testing (i.e., field testing) generally occurs after development of initial technical specifications and is usually larger in scope than alpha testing. In addition to gathering further information about feasibility, beta tests serve as the main means to assess scientific acceptability and usability of a measure. Measure developers can use beta tests to evaluate the measure’s suitability for risk adjustment or stratification and help expand previous importance and feasibility evaluations. When carefully planned and executed, beta testing helps document measure properties with respect to the evaluation criteria.
Features of Alpha and Beta Testing
The measure developer should consider the features of alpha and beta testing when planning their approach.
Feature |
Alpha Testing |
Beta Testing |
---|---|---|
Timing |
|
|
Scale |
Typically, smaller scale
|
|
Sampling |
|
|
Specification Refinement |
|
|
Importance |
|
|
Scientific Acceptability |
|
|
Feasibility |
|
|
Usability and Use |
|
|
Recommended Reports During Measure Testing
A measure developer should develop specific reports when testing a measure (or set of measures). Although completion of reports usually occurs after beta testing, measure developers should consider the need to report the results of formative alpha testing, especially if the intent is for alpha testing to precede beta testing. The first few steps of measure testing address planning and execution of testing and are identical for alpha and beta testing; the last steps address reporting and follow up after the conclusion of testing.
During measure testing the measure developer
- Develops the testing work plan
- Performs sampling
- Implements the plan
- Analyzes test results
- Refines measure, including incorporation of interested party input
- Retests the refined measure
- Updates the measure documentation