Skip to main content

Measure Use, Continuing Evaluation & Maintenance

Measure Production & Monitoring

Measure production and monitoring includes the ongoing tasks necessary to use the measure over time. This process varies significantly from one measure set to another depending on a number of factors, which may include

  • Scope of measure implementation
  • Measured entity
  • Data collection processes
  • Intended use of the measure (e.g., quality improvement, public reporting, pay-for-reporting, value-based purchasing)
  • Program measure use (measure evaluation criterion)

The level of effort for each task may vary by different measure attributes. 

There may be a requirement for measure developers to perform various tasks associated with ongoing implementation and production. The graphic provides an overview of the steps in the production and monitoring process.

Click image to enlarge. 

Image
A flow chart of the measure production and monitoring processes.

Conduct Data Collection and Ongoing Surveillance

Once measure development is complete and there is resolution of dry run identified implementation issues (if applicable), the measure may go to full implementation where the measured entities collect and calculate data. The measure developer’s ongoing surveillance should include scanning medical and scientific publications as well as the general media for articles and commentaries about the measure.

Ongoing information surveillance is similar to the information gathering stage covered in Measure Conceptualization. The measure developer conducts analyses of the literature and submits or uses reports for annual measure updates, as required by the project. 

As use of the measure continues, the measure developer identifies publication of new studies that address the soundness of the measure. The measure developer must pay attention to any organizations that issue relevant clinical practice guidelines (e.g., monitor ECRI Guidelines Trust®), especially for process measures. If the measure is based on a specific set of guidelines, the measure developer should monitor the guidelines’ publisher closely for any indication of plans to change their guidelines. If the measure is not based on guidelines, the measure developer can monitor the scientific and clinical literature for reports that would impact the scientific basis of the measure. New information, guideline changes, or other statements may trigger an early maintenance review. 

Measure developers can use the Environmental Scan Support Tool (ESST) and De Novo Measure Scan (DNMS) (login required) to assist with the literature review. See the ESST QuickStart Guide for how to use the ESST. When you login to the DNMS, you will have access to the DNMS QuickStart and a tutorial video.

After data collection begins, the measure developer monitors for unintended consequences the measure might have on clinical practice or outcomes. If the measure developer identifies significant unintended consequences in the literature or unusual trends in the data, especially if patient safety is the concern, they must not wait for a scheduled annual or comprehensive review, and an early maintenance review may be necessary.

Respond to Questions About the Measure

The measure developer may be responsible for reviewing stakeholder feedback and responding promptly, alerting the submitter of receipt and review of the feedback. Stakeholder feedback may include direct questions, help desk questions, comments about the measure, or the use of the measure in a specific program. Questions may offer ideas for education and outreach.

Monitor and Analyze the Measure Rates and Audit Findings

The measure developer should monitor and analyze measure performance rates and audit findings periodically. At a minimum, developers should perform annual assessments. 

  •  

    Annual assessments monitor and analyze for

    • Overall performance trends
    • Variations in performance, gaps in care, and extent of improvement
    • Disparities in resulting rates by race, ethnicity, age, social risk factors, income, region, gender, primary language, disability, or other classifications
    • Frequency of use of denominator exclusions, numerator exclusions, or denominator exceptions
    • Discretionary denominator/numerator exclusion, evaluate carefully for gaming, unintended consequences, and uneven application that could influence comparability
    • Patterns of errors in data collection or rate calculation
    • Changes in practice that may adversely affect rates
    • Impact of measurement activities on measured entities
    • Correlation of the data to either confirm the measure’s efficacy or identify weaknesses in the measure
       

Ongoing monitoring should continually assess a measure’s progression; any marked departures may be cause for concern. If the business case predicted performance targets, the measure developer should investigate any measure whose performance over time falls short of its target. Measure developers report this information during reevaluation.

Perform Measure Maintenance or Early Maintenance Review, When Appropriate

The measure developer should review each measure at least annually to ensure the codes used to identify the populations (e.g., denominator, numerator, denominator and/or numerator exclusions) are current, and to address other needed minor changes. Communicate all changes to stakeholders. Measure Maintenance Reviews describe the standardized processes for the annual update, the triennial full reevaluation, and early maintenance review.

If the CMS consensus-based entity (CBE) has endorsed the measure, the measure developer also reports the results of the maintenance review to the CMS CBE to reevaluate its endorsement at the time of CMS CBE maintenance review.

Last Updated: May 2023