Measure developers should conduct a formal review of the measure every 3 years.  In many ways, the comprehensive reevaluation process parallels the measure development process. The CMS consensus-based entity (CBE) also requires comprehensive reevaluation every 5 years for endorsed measures. See the CMS CBE Endorsement and Maintenance Guidebook for the CBE’s latest procedures.

The comprehensive reevaluation process ensures the measure continues to be of the highest caliber possible. By periodically reviewing the measure against standard measure evaluation criteria, the measure developer helps maintain the best measure over time.

A comprehensive reevaluation consists of information gathering (including a literature review of recent studies and clinical practice guidelines), analysis of measure performance rates, and synthesis of all feedback received. Measure developers usually convene and consult a Technical Expert Panel (TEP) as part of the comprehensive review.

When updating the measure documentation, the measure developer must ensure the documentation of any updates to the evidence, clinical guidelines, impact of the measure, experience of measure use in programs, gaps in care, and measure performance and disseminated to measured entities and other interested parties in a timely manner. At the time of initial submission, the measure developer must be aware of any testing requirements needed at maintenance. As the science of measurement advances, evaluation and endorsement processes also evolve. Therefore, the measure developer must also stay abreast with changes in measure testing standards for the purposes of measure maintenance. Refer to the Measure Testing section for additional details. 

The measure developer should include specific elements in the updated business case:

• Comparison of the actual performance of the measure with the projected rates
• Statement of the impact of the measure on the quality topic focus
• Updated projections for the next evaluation period

The updated documentation must adequately address any areas of potential concern the measure developer anticipates arising during public comment or CMS CBE review.

Comprehensive Reevaluation Procedures

Develop a Work Plan

The measure developer begins the comprehensive reevaluation process by developing a work plan. When developing the work plan, the measure developer should consider two other schedules:

• Rulemaking cycle for any regulatory process governing the measure set undergoing evaluation
• The CMS CBE’s measure maintenance schedule, if applicable

Gather Information

The measure developer should conduct ongoing surveillance during measure monitoring and summarize the findings of their environmental scan in a report, e.g., ESST monthly reports. The ongoing environmental scan should focus on information published or otherwise available since the last measure evaluation.

At a minimum, this synthesis should include

• Changes to clinical guidelines on which the measure is based, e.g., monitor ECRI Guidelines Trust®
• Relevant studies that might change clinical practice, which in turn, might affect the underlying assumptions of the measure
• Relevant studies documenting unintended consequences of the measure
• Relevant studies documenting continued variation or gaps in the measured care 
• Technological changes that might affect the collection, calculation, or dissemination of data 
• Similar measures based on their structure, clinical practices, or conditions that could offer an opportunity for harmonization or might serve as replacement measures
• Relevant information gathered from the TEP or interviews with subject matter experts or measurement experts
• Patients’ perspective on the measure
• Reevaluation of the business case supporting the measure
• Feedback received since the last measure evaluation (i.e., the initial evaluation or the last comprehensive reevaluation, whichever is most recent)

The measure developer should obtain measure performance information  including, but not limited to

• Current aggregate national and regional measurement results
• Measurement results trended across the years since the measure’s initial implementation
• Comparison to the trajectory predicted in the business case
• Current distribution of measurement results by measured entity types (e.g., rural vs. urban, for-profit vs. nonprofit, facility bed size)
• Analysis of the measure’s reliability, stability, and validity since implementation
• Results of audit and data validation activities
• Analysis of any disparities in quality of care based on race, ethnicity, age, health-related social needs, region, gender, or other classifications, including a determination the reduction of elimination of any previously identified disparities 
• Analysis of unintended consequences arising from the use of the measure
• Validation and analysis of the numerator and denominator exclusions, including, but not limited to
  • Analysis of variability of use
  • Implications of rates

The measure developer compares the information gathered with projections made in the original business case and reports the measure performance and the impact of the measure. The measure developer should update the business case as appropriate and make projections for the next evaluation period.

Convene a TEP

Typically, the measure developer convenes a TEP during comprehensive reevaluation to assess the measure. A best practice is for the measure developer to continue with the TEP that was involved with measure development. However, the measure developer should review the membership to ensure continued representation of an appropriate breadth of expertise and diversity. The content on  Interested Party Engagement, and the Technical Expert Panels supplemental material provide details of the standardized process for issuing a call for nominations and convening a TEP.

During the TEP meeting, the measure developer presents the results of the environmental scan, literature review, and empirical data analysis of the measure performance data, patients’ perspective, and analysis of ongoing feedback received. If information about the patient perspective is not available, the measure developer will want to ensure the TEP includes patient representative(s). Using input from the TEP, the measure developer drafts recommendations on the disposition of the measure using the measure evaluation and selection criteria.

Identify and Document Recommended Changes 

For each measure, the measure developer compiles the information gathered in the steps previously outlined using the measure evaluation criteria.

The measure developer should identify any material or substantive changes and explain the purpose of the changes. A material or substantive change is one altering the specifications of a measure to affect the original measure’s concept or logic, the intended meaning of the measure, or the strength of the measure relative to the measure evaluation criteria.

Determine the Preliminary Recommended Disposition of the Measure

A discussion of the criteria forming the basis for the disposition decision for each measure and description of the possible outcomes is in the content on Measure Maintenance Reviews Overview.

Test Measures as Necessary

For the first comprehensive reevaluation, the measure will require evaluation of reliability and validity beyond what occurred during measure testing at the time of development. If the measure is not in use, it will require expanded testing. The Table outlines the extent of measure testing or reevaluation of validity and reliability for measures in use and not in use.

Table. Extent of Measure Evaluation as a Function of Prior Comprehensive Evaluation and Measure Use

If the measure needs testing, the measure developer should develop a plan. A description of the components of a testing plan are in the content on Develop the Testing Work Plan.

Obtain Public Comment on the Measure

If there have been substantive changes to a measure as the result of comprehensive reevaluation, the measure developer should seek public comment on those changes. If the comprehensive reevaluation results in a recommendation to retain the measure with only minor changes, it likely is not necessary to seek public comment. However, for CMS measures, proposed rules usually outline these changes and are subject to public comment. Find the process for obtaining public comment in the content on Interested Party Engagement. 

The measure developer next analyzes the comments received and refines the measure as indicated. Depending on the extent of measure revisions, the measure developer may deem it necessary to retest the measure.

Implement the Disposition Action

After review, the measure developer may be responsible for implementing the chosen measure disposition. When proposing measures for revision, suspension, removal, or retirement, the measure developer should evaluate the impact of the decision on the program using the measure when developing the implementation plan. If there are relevant regulatory or rulemaking schedules, the measure developer should include them in the implementation plan. 

If CBE-Endorsed, Maintain CMS CBE Endorsement

See the CMS CBE Endorsement and Maintenance Guidebook for the CMS CBE’s latest procedures.

Harmonization During Comprehensive Reevaluation

Whenever a measure developer reevaluates a measure, the measure developer must compare the measure to related and competing measures, assessing for the possibility of harmonization. Search CMS Measures Inventory Tool (CMIT), CBE Submission Tool and Registry (STAR), and other sites for related or similar measures. If the measure developer identifies related measures, they should consider ways they can align the measure being reevaluated with the related measures. If the measure developer identifies competing measures, they should either justify why the reevaluated measure is best in class or give a rationale for continuing with possibly duplicative measures.

If measure specifications need alterations so they can harmonize with other measures, the changes could be substantive. The comprehensive reevaluation period may be the best time to make these changes. For more information about harmonization, see the Quality Measure Harmonization, Respecification, and Adoption content. 

More Information for CMS Measure Developers

Please note, the MMS Hub and Blueprint content are not CMS-specific, and the guidance is not prescriptive. CMS measure developers can find more information about measure maintenance reviews in the Blueprint Contractual Guidance and Considerations found on the Measure & Instrument Development and Support (MIDS) Library. The MIDS library is a protected website restricted to CMS and CMS MIDS contractors. The MIDS library is a protected website restricted to CMS and CMS MIDS contractors. MIDS developers should email MMSsupport@battelle.org to request access to the MIDS Library. Once eligibility is confirmed, the MIDS developer will receive an email with instructions on accessing the site.