Skip to main content

Stakeholder Engagement

Stakeholder Engagement in the Measure Lifecycle

Measure Conceptualization

During the measure conceptualization stage, different types of stakeholders have the opportunity to suggest topics or areas of measurement important to them. It is important for stakeholders to be able to present ideas to those who develop measures very early in the process to influence the direction of measure development. The measure developer should initiate technical expert panels (TEPs) or meet with stakeholders during measure conceptualization. The measure developer should solicit public comments during measure conceptualization and information gathering.

The Measure Collaboration Workspace (MC Workspace) has an electronic clinical quality measure (eCQM) Concepts module that permits anyone with an Electronic Clinical Quality Improvement (eCQI) Resource Center account to suggest a concept for an eCQM. Stakeholders have the option to submit an eCQM concept for feedback from other stakeholders and to ultimately submit the eCQM concept to CMS for review. While the MC Workspace is likely to attract healthcare and health information technology professionals, anyone with an interest in eCQI can get an account.

Measure Specification

A variety of stakeholders need to be involved in the measure specification stage. In addition to the measure developer and TEP, persons, families, caregivers, and clinicians need to be involved to address the feasibility of data collection before a measure progresses too far in development. The measure developer can use focus groups, interviews, and other informal methods to engage other stakeholders. The measure developer often posts a request for public comment and the measure specifications. 

CMS-contracted measure developers may use the Measures Management System (MMS) website, CMS MMS Hub, to announce the request for public comment. If the proposed measure is an eCQM, the measure developer may use the Office of the National Coordinator’s ONC Project Tracking System (Jira) to post the specifications and collect public comments. The eCQI Resource Center also posts requests for comments for eCQMs and points to where the user may find the specifications and provide comments.

Measure Testing

It is often appropriate for the measure developer to obtain stakeholder input at several points during the testing process. This process may include obtaining face validity assessments during alpha and beta testing, feasibility and burden inputs during beta testing, and other inputs based on a review of overall results. These inputs can take many forms, including formal TEPs, consultation with subject matter experts, outreach to professional associations or patient advocacy groups, and public comments. Once collection of these inputs occurs, it is important to follow up on the communications by providing additional opportunities for stakeholders to comment on the results of their inputs at future stages. Follow up increases the likelihood the measure developer operationalizes the inputs consistent with the stakeholders’ needs. It also improves the likelihood that stakeholders will remain engaged for ongoing support on current or future measures. If a TEP reviews testing results and updated specifications, the measure developer may post those summaries for further public comment. CMS-contracted measure developers may post the TEP summaries.

Stakeholders can volunteer to test measures under development. Additionally, through the MC Workspace eCQM Testing Opportunities module, measure developers may announce opportunities to test eCQMs.

Measure Implementation

CMS may request public comments on measures at different times in the implementation stage, for example, measures proposed for adoption in CMS programs either through the formal federal and rulemaking process or through an ad hoc public comment process for measures that are not subject to rulemaking. Many CMS programs must use the rulemaking process to formally propose and finalize measures. The public may comment on proposed new measures and measures proposed to continue in a CMS program. 

Measures proposed for some programs, those subject to Section 3014 of the Patient Protection and Affordable Care Act (P.L. 111-148), must first go through a pre-rulemaking process in which measure developers submit measures for CMS to consider for adoption in CMS programs. CMS reviews the submissions and develops a Measures Under Consideration (MUC) List. The Measure Applications Partnership publicly discusses measures on the MUC List and posts for public comment their recommendations for which measures to implement in the different CMS programs. 

The measure developer convenes stakeholder meetings regarding implementation of considered measures. Resolution of stakeholder questions about the measures occurs iteratively as the measure remains under consideration. The measure implementation process is completely transparent and open to the public for comments and questions.

Measure Use, Continuing Evaluation, and Maintenance 

The MMS Hub content describes the annual update, comprehensive reevaluation, and early maintenance review as distinct and separate activities; however, in practice, these activities, like the different measure lifecycle stages, sometimes overlap and can occur concurrently. All steps require solicitation of inputs via public comment, usually through the rulemaking process. The measure developer should publicly report results from, and progress on, each of these review processes. Stakeholders are engaged for comment and, in some cases, participate in a formal panel review. For more information on measure maintenance reviews, see Measure Maintenance Reviews and the Measure Maintenance Reviews supplemental material.

Ideally, there is alignment between the measure maintenance schedule for measures used in federal programs and the CMS consensus-based entity (CBE) endorsement maintenance cycle, which also includes requirements for public review and comment. However, in practice, these schedules may not align completely. The Quality Positioning System lists the CMS CBE-endorsed measures and has a mechanism enabled for comment.

The ONC Project Tracking System (Jira) provides the opportunity for public comments on eCQMs in use. The measure developer may choose to include the comments as part of the Change Review Process and contribute to changes in eCQMs during the annual update. For more information on the eCQM Change Review Process and Annual Update, refer to the eCQMs Specification, Testing, Standards, Tools, and Community supplemental material.

Last Updated: Nov 2022