|
Q1: What is the ACGME Outcome Project?
A1: The Outcome Project is a long-term initiative by which the ACGME
is increasing emphasis on educational outcome assessment in the accreditation
process. Expectations for increased emphasis on outcome assessment are reflected
in changes to Program and Institutional Requirements that require programs to:
- Identify learning objectives related to the ACGME's general competencies
- Use increasingly more dependable (i.e. objective) methods of assessing
residents' attainment of these competency-based objectives; and,
- Use outcome data to facilitate continuous improvement of both resident
and residency program performance.
Outcome Project activities to bring about these changes and to support programs
in their implementation include:
- Development of a set of general competencies;
- Support for programs through identification and development of dependable
methods for assessing attainment of the competencies;
- Development of model resident evaluation systems to provide examples of
dependable evaluation; and,
- Creation of a support system of resources that includes assessment experts,
ideas for teaching the general competencies, references for articles pertinent
to the definition, teaching and evaluation of the competencies , and a "toolbox"
of assessment techniques.
Q2: Why did the ACGME choose to concentrate on outcomes now?
A2: In a sense, the ACGME is "playing catch up" to other accrediting
bodies in the health professions, education, and business that have focused
on educational outcomes since the 1980s. At that time, the U.S. Department
of Education mandated a movement aimed at making greater use of outcome assessment
in accreditation. As a result, efforts were begun by these organizations to
expand their use of outcome measures in accreditation. In addition, since
the U.S. system of medical education depends heavily on public funding, medical
educators are called upon to offer evidence of their responsible stewardship
in preparing competent physicians to meet the health care needs of the public
that supports their efforts. The ability to demonstrate educational outcomes
as the achievement of competency-based learning objectives provides just such
evidence.
Q3: What does the ACGME mean by outcomes?
A3: Not to be confused with clinical outcomes, the Outcome Project
refers to educational outcomes which are, "evidence showing the degree
to which program purposes and objectives are or are not being attained, including
achievement of appropriate skills and competencies by students." ("Accreditation
for Educational Effectiveness: Assessment Tools for Improvement," Council on
Postsecondary Education, February 1993, p. 37) Certainly, clinical outcomes
can and should be used as educational outcomes for several of the General Competencies.
However, they are not the only educational outcomes of residency education.
Achievement of learning is the ultimate purpose of any well-structured educational
activity. In keeping with the ACGME's mission to ensure and improve the quality
of graduate medical education, the Outcome Project focuses on educational outcomes,
thus demonstrating achievement of learning by graduates of accredited residency
education programs.
Q4: What overall change to the accreditation process will result from the Outcome Project?
A4: At present, GME accreditation utilizes a "minimal
threshold model" by which programs are judged according to how they comply with
minimum standards established by the Residency Review Committees (RRCs) and
the ACGME. In the competency-based model toward which the Outcome Project is
directed, programs will be asked to show how residents have achieved competency-based
educational objectives and in turn, how programs use information drawn from
evaluation of those objectives to improve the educational experience of the
residents. Stated another way, the minimal threshold model identifies whether
a program has the potential to educate residents; the competency-based model
examines whether the program is actually educating them.
Q5: What specific changes can I expect in the process by which
our program is accredited?
A5:
- In the short term, little noticeable change will occur in the accreditation process,. Programs will be expected to demonstrate that they are using increasingly more dependable methods to evaluate the general competencies. In other words, they must demonstrate progress in implementing better evaluations.
- In the long term, the ACGME has yet to determine how it will use outcome measures, in addition to board scores, to make accreditation decisions about overall program effectiveness.
Q6: How will the Outcome Project affect the recognition of Institutions by the ACGME to sponsor graduate medical education programs?
A6: No plans exist to change the manner by which institutions are recognized by the ACGME. As is the case with the Program Requirements, the Institutional Requirements are also being revised to include references to the General Competencies and their evaluation. The responsibility remains for the sponsoring institution to ascertain that the residency programs under its aegis are adequately supported and appropriately conducted. With increasing emphasis on assessment of competency-based learning objectives resulting from the Outcome Project, that responsibility should include oversight of program quality demonstrated by results of outcome assessments considered during the internal review process.
Q7: What changes can a program make now to accommodate the
change in philosophy of GME accreditation represented by the Outcome Project?
A7: Without exception, all Program Requirements now include reference
to the need for goals and objectives as well as evaluation of the residents
and the program. Likewise, many Program Requirements already include references
to the General Competencies, although the language may vary somewhat from the
minimum language approved by the ACGME in September 1999. Therefore, in some
sense, if programs have carefully planned their curricula in alignment with
existing Program Requirements, they may already be moving in the direction of
a competency-based curriculum! The additional critical step, however, is linking
assessment with specific learning objectives. For example, if one objective
of the cardiology rotation is to distinguish pathologic from functional murmurs,
one component of the evaluation of that rotation should include assessment and
documentation of this skill. Programs should ask themselves whether or not they
are completing this important assessment step right now and, in turn, if they
are using the results of their evaluations to improve their program.
Q8: Why did the ACGME choose to identify competencies?
A8: By identifying the general competencies, the ACGME was responding in part to growing criticism from a variety of sources including the medical community itself, that residents were not adequately prepared to practice in the rapidly changing healthcare environment. From an educational standpoint, competencies can be regarded as the logical building blocks upon which assessments of professional development are based. In the case of GME, identification of the General Competencies represents specification of what residents should know and be able to do. When competencies are identified, a program can effectively determine the objectives that should guide progress toward their achievement and, in turn, what outcomes should be assessed as evidence of the program's quality, i.e., its effectiveness in meeting the objectives and thus preparing competent residents for practice.
Q9: How did the ACGME identify the six General Competencies?
A9: The General Competencies were identified by means of a thorough research and collaborative review process occurring between January 1998 and February 1999. This process involved searching literature and other documentation, gathering input, drafting preliminary language, obtaining feedback, and preparing numerous revisions. The extensive input and feedback process involved published reports, curriculum documents, surveys, interviews, and focus groups. It tapped a cross section of key stakeholders with representation from the medical profession, residents, and educators, employers of physicians, patients, and society-at-large as typified by a private foundation, the U.S. government, health care quality monitors, and community health providers. Executives representing nurses, physician assistants and allied health professionals were also queried. The Outcome Project Advisory Group reviewed a compilation of the data and made the "cut" from an original list of 86 statements to the present six General Competencies. The Advisory Group made the final recommendation of the present six General Competencies to the ACGME.
Q10: What is the difference between the "Full" and "Minimum" version of the General Competencies as viewed on the Outcome Project web page?
A10: The "Minimum Language" version of the General Competencies was endorsed by the ACGME at its September 28, 1999 meeting. It is intentionally meant as general, transitional language, to provide programs with latitude in incorporating various aspects of the general competencies into curricula and for teaching and evaluation. All RRCs are required to have this minimum version or some other version of language including the General Competencies and their evaluation in their respective program requirements by June, 2001. The "Full" version of the General Competencies includes a more complete set of descriptors for each competency. The General Competencies are currently being adapted by a Joint Initiative of the ACGME and the American Board of Medical Specialties (ABMS) to reflect the uniqueness of each specialty.
Q11: How can programs afford to develop assessment tools to measure the competencies?
A11: The ACGME does not expect programs to assume complete responsibility for development of assessment tools. Efforts are already underway and will be expected to continue during the next several years, to identify and develop acceptable tools; these tools will be made available to programs. It must also be kept in mind that, because all specialties currently require some form of evaluation, programs should already have assessment tools in place. The most important factor in their continued use and/or development, however, is that all tools should be increasingly more valid and reliable measures of competency-based learning objectives. Programs are free to share resources and to engage in collaborative efforts to assist in the development of assessment tools. The Minimum Language for the General Competencies also acknowledges that many programs will need time to do an acceptable job of implementing evaluation methods that are dependable and useful for improving their educational program. Thus, allowance is made for programs to phase in, improvements in evaluation.
Q12: With the additional expectation of implementing outcomes measures, how will the faculty find time to teach?
A12: The good news is that programs won't be "going it alone." The specialty boards and program director organizations will provide support through suggested curricula and assessment tools. A Joint Initiative between the ACGME and the American Board of Medical Specialties (ABMS) is currently engaged in adapting the General Competencies to reflect the uniqueness of each specialty, proposing methods to assess each competency, and drafting an implementation plan. The Outcome Project link on the ACGME web site will eventually post much of this information. While the ACGME has taken into account the fact that time and resources must undoubtedly be expended to implement outcome measures, as mentioned in Question 10, a period of transition is expected. To facilitate the process, programs should adapt existing tools or develop new ones which maximize use of technology in data collection, compilation and reporting. They should consider engaging in collaborative efforts and distribute responsibility for evaluation among diverse resource individuals who have a stake in the process and are knowledgeable of resident performance. They should focus on a few competency-based objectives for evaluation during naturally-occurring resident-patient-supervisor encounters, keeping in mind that not all aspects of each competency are expected to be evaluated.
Q13: What is the Outcome Project Advisory Group?
A13: The Outcome Project Advisory Group is an ad hoc11-member consultative body to the ACGME Council and staff for purposes of planning and development of the Outcome Project. Its activities are funded by a grant from the Robert Wood Johnson Foundation to support the Project. The Group reviews work in progress and makes substantive recommendations related to draft evaluation system models and other core Project activities. All of the members have expertise in one or more substantive areas central to the Project and all have achieved national recognition for contributions to their respective areas: development and implementation of models of accreditation and outcomes assessment, i.e., performance assessment, clinical outcomes, and quality improvement; and, medical education, i.e., undergraduate, graduate, and education in ambulatory settings. Among the Group's responsibilities have been the crafting of the General Competencies, development of assessment system models, and recommendations for a model to increase emphasis on outcome assessment in accreditation. The Group has also reviewed and refined the Project time line.
Q14: At the present time, subspecialty programs (with the exception noted below) are not required to respond to the General Competency program requirements (effective July 2002).
A14:
At the present time, subspecialty programs (with the exception noted below) are not required to respond to the new General Competency program requirements (effective July 2002).
The reason for this position is a practical one. We expect that experience gained and resources developed during the initial years of implementation in core specialties 1 and Transitional Year programs will facilitate effective, efficient implementation in subspecialty programs in the future.
Although ACGME policy in this matter currently relates to core programs only, some subspecialties have elected to move forward with the competencies and their assessment. Programs in Child and Adolescent Psychiatry now are now responsible for General Competency programs requirements (effective July 2002). The Residency Review Committee (RRC) for Internal Medicine included related language in subspecialty requirements, effective July 1, 2005. The RRC for Pathology included competency and assessment language in its subspecialty requirements, effective July 1, 2003. In other specialty areas such as Psychiatry, targeted initiatives to include the competencies have arisen with special leadership from subspecialty organizations working directly with respective RRCs. Some institutions have likewise decided to engage the subspecialty programs under their sponsorship in more immediate plans to integrate the competencies and their assessment into revised curricula.
Additional information pertaining to subspecialty implementation of General Competency requirements will be posted as it becomes available. (posted Feb. 13, 2006)
Q15: What should residency programs be doing now (3/06) to assess their residents?:
A15:
Programs should assess resident competencies in all 6 domains with at least one approach in addition to global/end-of-rotation clinical ratings. Recommended methods are direct observation and concurrent evaluation (and other focused assessment methods) [PC, ICS, PBLI], multi-source/360-degree evaluation involving non-MD members of the care team and patients and their families [ICS, P, SBP], checklist evaluation of improvement projects [PBLI,SBP], and cognitive tests [MK, SBP, P]. Criteria that describe different levels of performance and interventions to assist evaluators in the use of criteria are expected.
Q16: What is aggregated performance data? What aggregated data are programs supposed to submit to their GMECs (end of Phase 2 of the Outcome Project timeline)?
A16:
Aggregated performance data refers to summary resident assessment results. Examples are % of residents passing the certification exam on the first attempt; the program’s percentile rank on in-training exams by PGY level; % of residents who regularly consult the literature to address clinical questions; and % of nurses who express that residents are responsive to their patient care concerns. In general, aggregate data will provide evidence how residents overall within PGY level are performing on selected aspects of each competency of most importance to the specialty and program. It might show the extent to which residents are advancing in capabilities across the years of residency. Included among the aggregate results should be specific areas of resident competence that programs are targeting for improvement through educational interventions.
Q17: What external measures does the ACGME expect programs to use? When will programs be held accountable for external measures of performance? (Phase 3 of the Outcome Project timeline)
A17:
By 2008 residency programs are expected to be collecting feedback on their residents’ performance from at least one source external to the residency program. This could be from patients and their families, other specialists who have sought consultations from residents or referred their patients, members of the care team from specialties other than the residents’ own or from other health professions, or employers. The ACGME encourages programs and institutions to move forward with collection of data indicating resident performance of condition specific evidence-based patient care processes and, where appropriate, of outcomes of providing care (e.g. surgical complication rate, scores on validated functional outcome questionnaires, % patients who stopped smoking or loss weight following counseling or other resident-initiated intervention).
1 The core specialties are defined as specialties with designated Residency Review Committees (RRCs): Allergy & Immunology, Anesthesiology, Colon and Rectal Surgery, Dermatology, Emergency Medicine, Family Medicine, Internal Medicine, Medical Genetics, Neurological Surgery, Nuclear Medicine, Obstetrics & Gynecology, Ophthalmology, Orthopaedic Surgery, Otolaryngology, Pathology, Pediatrics, Physical Medicine & Rehabilitation, Plastic Surgery, Preventive Medicine, Psychiatry & Neurology, Radiology, Surgery, Thoracic Surgery, and Urology.
|
