State of the science: Quality improvement of medical curricula—How should we approach it?

Abstract Introduction Quality improvement (QI) of the medical curriculum is generally regarded as a continuous process of evaluating whether the specific curriculum meets relevant educational and professional standards, implementing new activities or other measures to address perceived deficiencies, and subsequently re‐evaluating the quality of the curriculum. QI is of consequence to medical learners, educators, patients, carers, specific disciplines and specialties, regulators and funders. Methods To address how we should approach QI of medical curricula, a narrative review was undertaken, drawing mainly on medical/health professions education literature, identified through searches of the MEDLINE, EMBASE, PUBMED and ERIC databases, and also on exemplar curricular frameworks and evaluation reports. Assumptions and practices in QI of medical curricula were explored critically. Results The review compares alternative conceptualisations of QI; asks questions about priorities and perspectives in what we choose to evaluate; reflects on standards used to guide QI; critically discusses methods, models and theoretical approaches to the generation of evaluation data; and considers ownership of, and engagement with QI of medical curricula. Conclusions Recommendations for curriculum teams include that discourse is necessary to achieve transparency and a shared understanding of continuous QI in a particular curricular context. Continuous QI requires data collection methods aligned to specific evaluation questions/foci; multiple methods for data collection, from different stakeholders; and appropriate evaluation models and theory to provide a framework for QI. Embracing a quality culture approach may increase the sense of ownership experienced by stakeholders. Mechanisms include creating democratic‐collegiate cultures for multiple stakeholders to collaborate in QI; engaging stakeholders in QI activities and (e.g. SoTL) projects that contribute to holistic continuous QI; and proactively embedding quality in the (co‐)creation of curriculum components and resources.


| INTRODUCTION
Quality improvement (QI) of the medical curriculum is an important priority for the curriculum team and is of consequence to many stakeholders: learners, educators, patients, carers, specific disciplines and specialities, regulators and funders. Continuous QI is distinct from, but feeds into, externally driven accreditation-focused curriculum reviews. This narrative review 1  the World Federation of Medical Education (WFME). Quality assurance (QA) is a process whereby regulatory/accrediting bodies determine the extent to which quality standards are met by medical curricula accountable to them. 3 Quality improvement is generally regarded as a continuous process whereby curriculum teams use stakeholder feedback to determine whether the curriculum meets relevant standards, then implement new activities or other measures to address perceived deficiencies and subsequently re-evaluate the quality of the curriculum. 2 Frank et al. distinguish between QA as summative and QI as formative. 4 In healthcare per se, there is significant focus on QI to enhance the patient experience/improve patient safety. For example, in the United States, the Accreditation Council for Graduate Medical Education (ACGME) requires physicians to demonstrate competencies related to quality improvement. 5 Accordingly, quality improvement curricula have been developed to teach medical learners and practitioners, and other healthcare professionals, about the principles and process of quality improvement. [6][7][8][9][10][11] However, the focus of this review is not on curricula about QI, but on QI of the medical curriculum.

| OVERVIEW OF THE QUALITY IMPROVEMENT PROCESS
The process and common mechanisms for continuous QI of medical curricula are elaborated in standard medical education texts. 2 The plan-do-study-act (PDSA) cycle for continuous QI is attributed to Deming. 12 It has been adapted for the medical curriculum by Kenwright and Wilkinson to become a cycle of plan-implementevaluate-investigate. 2 The various stakeholders typically provide a starting point for the process. The main stakeholder group in QI of medical curricula has traditionally been medical learners themselves, [13][14][15] but feedback may also be sought from other stakeholders, for example, alumni, or elective hosts. 16,17 Depending on where a particular curriculum is positioned on the continuum of medical education-undergraduate (UG), postgraduate (PG) or continuing professional development (CPD)-the specific stakeholder groups may change or the views of certain groups may hold more or less sway.
Usually by responding to surveys, or participating in focus groups or curriculum committees, stakeholders evaluate the curriculum or aspects of it. Curriculum teams review the evaluation and respond to it. This may include further exploration of specific issues; for example, organising focus groups to better understand the reasons behind negative scores or comments in a survey. The response may include making changes in existing provision for learning, teaching and assessment; addressing issues in the learning environment; or taking pedagogical or pragmatic decisions about whether changes in existing provision are appropriate or feasible. Importantly, the response from curriculum teams should incorporate 'closing the loop', 18 which includes communication of the outcome(s) to stakeholders. At the next run of the curriculum, the process begins again.

| PROBLEMS WITH QI OF THE MEDICAL CURRICULUM
There are several problems relating to continuous QI of the medical curriculum. Firstly, members of the medical education community use different language to describe QI, depending on their geographical context and/or whether they are embedded in an UG or PG context, is what is prioritised in QI of the medical curriculum and what influences this. Thirdly, in aiming to improve curricular quality, are we evaluating the curriculum against appropriate standards, andfourthly-using aligned methods? These issues matter, because our priorities and our methodological approach to QI impact which stakeholders have a voice. Finally, the literature advocates shared ownership of curricular QI, but how might we foster this? The remainder of this review addresses these problems, by exploring some of the assumptions and practices that characterise QI of medical curricula and making recommendations to curriculum teams.

| DIFFERENT CONCEPTUALISATIONS OF QI
Are we, the medical education community, 'speaking the same language' when we talk about QI? Different terminology is used in different contexts, which matters inasmuch as it reflects different conceptualisations of QI. The term 'quality improvement' is commonplace in healthcare contexts and used in some educational contexts.
However, in other educational contexts the term quality enhancement (QE) is used instead. 19,20 This represents a move away from negative connotations of the word 'improvement': Williams argues that 'improvement is often used to refer to a process of bringing an activity up to standard whereas enhancement is about raising [an already good standard] to a higher degree, intensifying or magnifying it'. 21 QE terminology is prevalent in undergraduate contexts in the European arena. 22 Indeed, in a recent publication by a Dutch group, QI is described as 'continuous enhancement of educational quality'. 23 QE is focused on developing a 'quality culture' in the educational organisation. [22][23][24] The concept of quality culture incorporates shared values, collective responsibility, institutional autonomy, transparency, effectiveness and transformative ability through empowerment. 22 The focus in QE is on development and innovation as opposed to control and compliance. 24 External QA is focused on evaluation of the education provider's internal QE (QI) processes as opposed to judging whether curricular standards have been met. 22

| PRIORITIES AND PERSPECTIVES ON WHAT SHOULD BE EVALUATED
What should we evaluate? Judgements about the quality of curricula will reflect what is valued and prioritised by stakeholders. This will in turn reflect their accountability to specific groups in society (including patients, carers and taxpayers), to funders, to regulators and professional bodies, to their specialities or disciplines, or to educational principles; or concern for their own learning and progression.
Evaluators may therefore prioritise quality in specific aspects or components of the curriculum, such as the degree of public and patient involvement, 26 the programme of assessment 27 or resources for learning.
Stakeholders may also have different perspectives on what a curriculum should be. Curriculum theory informs different approaches to curriculum design, based on the underlying purpose of the curriculum: curriculum-as-content (or -syllabus), curriculum-as-product, curriculum-as-process and curriculum-as-praxis. 28 The curriculum-as-content perspective is teacher-centric and sees curriculum as the taught subject matter. The curriculum-as-product perspective, attributed to Ralph Tyler, 29 focuses on what the curriculum aims to produce. In medical education this will be effective doctors. This perspective arguably aligns most closely with the outcomes-or competency-based medical curriculum. 30 The curriculum-as-process perspective, attributed to Stenhouse, 31 is concerned with the learning process during the curriculum, including opportunities for learning and interaction. It focuses on learner development and is therefore learner-centred. 32 Modern medical curricula may aim for the 'product' of an effective doctor, but they are also concerned with the process of learning, wanting their doctors to develop into self-directed, selfregulated, reflective practitioners. Herein lie tensions that can spill over into evaluating the quality of the curriculum, if stakeholders have different perspectives on the purpose of the curriculum. Also, where a medical curriculum must meet multiple sets of standards (e.g. those of educational and professional regulators), specific priorities or curriculum philosophies may align better with one set of standards than the other. An example of differing stakeholder priorities impacting UG medicine in the United Kingdom is that an ultimate focus on patient safety has driven the implementation of a high-stakes medical licensing assessment 33 ; whereas the general move in higher education is towards sustainable assessment (that which facilitates life-longlearning, including self-assessment, peer-assessment, reflection, portfolios and embedding assessment within the learning activities). 34,35 From the curriculum-as-praxis perspective, attributed to Grundy, there is a focus on what is 'valuable', and changes needed to enhance this within the curriculum. 28 This is consistent with a patient-centred focus in medical education, with ensuring that the curriculum is inclusive, and with ensuring it is not undermined by the hidden curriculum (learning that is part of the educational experience but not part of the formal curriculum, and potentially conflicting with the latter). Depending on what is valued and prioritised by those who judge curricular quality, and their conceptualisation of curriculum, stakeholders may legitimately hold different perspectives on where the curriculum should be centred (e.g. on the learner or the patient).
Damodaran suggests the concept of learner-centredness has become overly dominant in medical education and that the 'optimal centredness' of the medical curriculum may legitimately vary at different phases of medical education. 36 Recommendations: curriculum teams should create opportunities for discussion about what is valued in the curriculum, its purpose and its centredness. In an integrated evaluation of the curriculum, 23 although specific priorities or perspectives may shape a particular episode of data collection, there should be oversight by the curriculum team, who should synthesise and reflect on data from multiple sources.

| CONSIDERATIONS ABOUT QUALITY STANDARDS
An indicator of quality includes the provision of curricular learning objectives, learning outcomes and/or competencies, 37 because these specify to all stakeholders what should be included in the curriculum and provide standards against which the curriculum (and stakeholders) may be judged. The terms 'learning objectives' and 'learning outcomes' are sometimes used interchangeably. However, learning outcomes are intentionally learner-centric, specifying how the learner will be able to demonstrate achievement of relevant knowledge, skills and attitudes, providing a guide as to breadth and depth of learning: for example, 'apply the principles and methods of quality improvement to improve practice (for example, plan, do, study, act or action research), including seeking ways to continually improve the use and prioritisation of resources'. 38 Instead, learning objectives emphasise the material to be learned, but often without indication of breadth or depth of learning; for example, 'understand the principles and methods of quality improvement'. 39 Competencies specify achievement at a particular level of performance, according to specific standards; for example, 'apply the science of quality improvement to contribute to improving systems of patient care'. 40 Modern medical curricula generally describe themselves as outcome-based or competency-based. 30 QI might focus on ensuring that the outcomes or competencies are appropriate; that learning activities and the learning environment will allow learners to meet the outcomes or competencies; and that assessment systems will allow schools, colleges or other institutions to determine the extent to which learners have achieved the specified outcomes or competencies.
One tension is that outcomes/competencies published by regulators are often necessarily broad, leaving considerable room for local interpretation. Although this affords flexibility to curriculum teams, who can take into consideration available resources and other contextual issues, it presumably is one source of varying quality in medical curricula. A further tension is that a medical curriculum may need to conform to different sets of standards. For example, in the United Kingdom, the UG medical curriculum must conform not only to standards set by the devolved national governments and/or regulators of the higher education sector (e.g. the Office for Students in England, https://www.officeforstudents.org.uk/) but also to standards set by the professional regulator (Outcomes for Graduates specified by the GMC 38 ). These respective organisations likely hold different views on the purpose of an UG (medical) curriculum, and the challenge is to provide a medical curriculum that meets different sets of standards, which may not be fully compatible.
A third tension exists between the need for quality standards that can be interpreted locally yet ensure a minimal proficiency across all contexts. Some countries have adopted or adapted standards from other contexts; for example, the Taiwan Medical Accreditation Council (TMAC) developed their 2013 standards based on those of the Liaison Committee of Medical Education (LCME), 41 which accredits medical schools in the United States and Canada. Importing standards from another medical education context may not sufficiently take account of local culture or the needs or priorities of that population. A long-term WFME project has been to define Global Standards for Quality Improvement of Medical Education for various phases of medical education. The organisation's website (https://wfme.org/ standards/) has links to the recommended standards for each phase, the most recent of which is a 2020 version of the standards for basic medical education. 42 However, in a systematic review of empirical research into the practical application of the WFME standards for QI, 43 the authors acknowledged that these standards may be oriented towards 'western' contexts and that the WFME itself recommends they be used as a guide, adapted for local contexts.
More recent additions to the field are quality standards set by professional membership associations, such as the criteria for ASPIRE awards offered by the Association for Medical Education in Europe (AMEE). These awards aim to 'recognise excellence' (https://www. aspire-to-excellence.org/), which is a synonym for recognising quality.
Recommendations: curriculum teams must evaluate curricula against appropriate standards. Where medical curricula conform to multiple sets of standards, curriculum teams should be explicit about which standards are being used in any evaluation activity and ensure that appropriate measures are used.

| METHODOLOGY IN QI: METHODS, MODELS, THEORY
Perspectives about appropriate methodology for QI will likely be influenced by whether curriculum teams see QI as a 'dry run' for external QA or as means of ensuring a quality culture. Specific issues include the dominance of learner surveys, ethical evaluation, and the extent to which the evaluation of medical curricula is directed by application of models or theories.

| Methods for evaluation
In modern learner-centred curricula, continuous QI has become to some extent synonymous with regular evaluation of the learner experience, often using institutional or national surveys. In the higher education sector, increased marketisation, government use of survey data and possibly 'consumerist attitudes' of students have led to an 'obsession' with survey results. 44  Regulators may favour such quantitative instruments, with generic Likert-type statements that allow for comparisons across subject areas, disciplines or specialities, even institutions, but such statements may be difficult for stakeholders to interpret in their context or may not address issues of greatest interest or importance to learners, curriculum teams or other stakeholders. Perhaps this is why educational leaders in a Dutch appreciative inquiry study expressed a preference for 'narrative and just-in-time feedback'. 24 The latter is feedback from learners to curriculum teams whilst the course/ programme is running; potentially, such evaluation helps to 'close the loop' if learners can see even small-scale or preliminary changes being implemented in response to their feedback.
With learner surveys, a tension exists between the methodological requirement to achieve good response rates, and ethical considerations. In the UK UG context, institutional key performance indicators include the percentage of students in a specific discipline that complete the NSS. Because UG student participation cannot be mandated, there is a veritable industry around encouraging students to respond and encouraging faculty to promote the surveys. It seems ironic that incentives are discouraged in ethical education research 47 48 Nonetheless, low response rates remain an issue, at least for trainers, because just 32% of trainers responded in 2021 and 34% in 2022. 49,50 This may not be surprising, given the ongoing pandemic at the time of 2021 data collection, and the subsequent 'catch-up' on non-Covid cases, but non-response bias must be of concern.
Although learner surveys still hold sway, they increasingly ask for free text responses, which can be more helpful to curriculum teams than per cent satisfaction scores. However, to genuinely understand stakeholder perceptions and enhance quality, curriculum teams should utilise additional forms of data collection, which may be more engaging for learners, allow for a deeper exploration of issues, facilitate just-in-time responses to learners, and/or give other stakeholders a voice. Moreover, 'triangulation of different instruments and procedures' is advocated, to give 'a holistic overview' of curriculum quality. 23

| Evaluation models
Application of a curriculum evaluation model may guide the collection of data for continuous QI, providing direction, defining parameters, facilitating a systematic approach, and specifying 'a relationship of parts'. 51 Coles and Grant proposed a model requiring triangulation of data from the intended curriculum (the curriculum 'on paper'), the delivered curriculum ('in action') and the curriculum experienced. 25 This has been used in evaluation of a community-based medical curriculum. 52 In a bid to recognise the complexity of different stakeholders in the curriculum, Anderson et al. 53 60 ; on the other hand, it has been used to analyse social accountability frameworks as a prelude to evaluating social accountability in medical education. 61

| Theory-informed evaluation
Greater understanding about, and embedding of, quality in medical curricula may be achieved through a theory-informed approach to continuous QI, utilising evaluation models aligned with specific theories. 53  Recommendations: applying methodological lessons from medical education research, curriculum teams should align data collection methods to specific evaluation questions/foci; use multiple methods for data collection, from different stakeholders; and use appropriate evaluation models and theory to guide and organise evaluation.

| OWNERSHIP OF AND ENGAGEMENT WITH CONTINUOUS QI
Institutional approaches to ensuring quality curricula may focus on responding to outcomes from previous national learner surveys, or on designing and implementing in-house surveys to anticipate issues before the next national learner survey is administered. Institutions may enact interventions and policies to address issues of broad concern across disciplines or specialties. This 'top-down' generic response to stakeholder evaluation takes ownership of QI away from key stakeholders. 23 By emphasising formative, developmental evaluation and encouraging engagement with QI, we may foster a quality culture and increase the sense of ownership in the quality of the medical curriculum. 23 But how?
There is a growing body of literature advocating and demonstrating the value of learners as active participants in continuous QI; not simply as survey respondents, but in the design, data collection and interpretation of evaluations. Klemencic argues that students possess capital relevant to QI, through their direct experience of learning, teaching, assessment and environment. 63 By providing data on their experiences, but also by participating in governance structures, and by acting as direct advocates for a quality education, they can contribute significantly to continuous QI. 63 Institutions can facilitate student participation in governance structures by establishing student evaluation committees whose members, with relevant training, can generate and analyse QI data. 64 Faculty can act as mentors and partners, sharing knowledge about adult learning theories, teaching and learning strategies, and best practice in feedback. 65 At Harvard Medical School, student participation in QI has been extended such that student working groups undertake projects to address policy issues or priorities such as 'enhancing diversity in the curriculum'. 65  also applied to faculty and administrators. 66 Although continuous QI should be holistic, with oversight and integration from the curriculum team, 23 at the level of individual stakeholders, contributory QI activities or projects could be targeted towards identifying needs of a specific stakeholder group ('context' in the CIPP model 58 ), identifying an innovative solution (input), then piloting and evaluating it (process and product). Current interest in scholarship of teaching and learning (SoTL) presents an opportunity to engage faculty in continuous QI. 67 SoTL has been described as 'systematic inquiry into student learning which advances the practice of teaching by making inquiry findings public', 68 but SoTL inquiry could also be well-suited to one or other foci of evaluation, per the CIPP model. 58 More generally, proactive embedding of quality should be a target in developing any curricular component or resource. For example, applying the concept of co-creation of the curriculum, 23,69 Huser et al. mentored medical students to produce online resources for the curriculum, 70 stressing the importance of applying a framework for quality in online materials. 71 Recommendations: curriculum teams should provide oversight and integration, but encourage individual QI activities and (e.g. SoTL) projects to contribute data to holistic QI. Quality should be embedded proactively, by applying quality frameworks when developing curricular components/resources.

| CONCLUSIONS
Discourse is necessary to achieve transparency and a shared understanding of continuous QI in a particular curricular context.
Continuous QI requires data collection methods aligned to specific evaluation questions/foci; multiple methods for data collection, from different stakeholders; and appropriate evaluation models and theory to provide a framework for QI. Embracing a quality culture approach may increase the sense of ownership experienced by stakeholders.
Mechanisms include creating democratic-collegiate cultures for multiple stakeholders to collaborate in QI; engaging stakeholders in QI activities and (e.g. SoTL) projects that contribute to holistic continuous QI; and proactively embedding quality in the (co-)creation of curriculum components and resources.