Institutional accreditation in medical education: The experience of the survey visit teams
Mohammad H Yarmohammadian1, Elahe Khorsani2, Roohangiz Norouzinia3, Samaneh Mirzaei4, Soheila Ehsanpour5, Nikoo Yamani5, Fatemeh Rezaei6
1 Professor, Health Management and Economics Research Center, Isfahan University of Medical Sciences, Isfahan, Iran
2 Department of Pharmacoeconomics and Pharmaceutical Administration, School of Pharmacy, Tehran University of Medical Sciences, Tehran, Iran
3 PhD Candidate, Health Management and Economics Research Center, Isfahan University of Medical Sciences, Isfahan; Faculty Member, School of Paramedical Science, Alborz University of Medical Science, Karaj, Iran
4 Department of Health in Emergency and Disaster, School of Public Health, Shahid Sadoughi University of Medical Sciences, Yazd, Iran
5 Medical Education Department, Medical Education Research Center, Isfahan University of Medical Sciences, Isfahan, Iran
6 Associate Professor, Medical Education Department, Medical Education Research Center, Isfahan University of Medical Sciences, Isfahan, Iran
|Date of Submission||07-Sep-2019|
|Date of Acceptance||12-Nov-2019|
|Date of Web Publication||28-Feb-2020|
Dr. Fatemeh Rezaei
Department of Health in Emergencies and Disasters, Isfahan University of Medical Sciences, Isfahan
Source of Support: None, Conflict of Interest: None
OBJECTIVES: This study discussed about challenges and opportunities of institutional accreditation in Iranian medical universities. The lesson learned of the first round of the accreditation would direct initiatives and solutions for future accreditation.
METHODOLOGY: This research is a qualitative content analysis research that studied the experience of the survey visit teams. Semistructured interviews with a purposive sampling approach were done to reach data saturation. Constant comparative method was used to analyze the data.
RESULTS: The emergent themes were incompatible scheduling with accreditation workload, accreditation sustainability, the weakness of the survey instrument, advantage of the survey instrument, the quality of evaluators' survey, the opportunity of progress for universities, accreditation perspective, defendable cost opportunity, perceived injustice, and the leading strategies. In this study, the relationship between categories and themes was shown with a systematic approach.
CONCLUSION: In medical education systems, it is necessary to continue the research and development studies using the findings in each accreditation period to achieve the prospect of international accreditation.
Keywords: Accreditation, institutional management teams, medical education
|How to cite this article:|
Yarmohammadian MH, Khorsani E, Norouzinia R, Mirzaei S, Ehsanpour S, Yamani N, Rezaei F. Institutional accreditation in medical education: The experience of the survey visit teams. J Edu Health Promot 2020;9:39
|How to cite this URL:|
Yarmohammadian MH, Khorsani E, Norouzinia R, Mirzaei S, Ehsanpour S, Yamani N, Rezaei F. Institutional accreditation in medical education: The experience of the survey visit teams. J Edu Health Promot [serial online] 2020 [cited 2020 Jul 11];9:39. Available from: http://www.jehp.net/text.asp?2020/9/1/39/279805
| Introduction|| |
Evaluating the quality of medical education through accreditation of medical institutions is one of the serious health systems challenges of most countries. The institutional accreditation is especially important for patient safety and quality of patient care in medical and paramedical careers.,,, In addition, the accreditation encourages resource utilization and institutional self-studies that both are essential for continuous quality improvement.
Studies showed that accreditation has an effect on quality assurance, reducing work hours of residents without increasing patient load, balancing the number of graduate medical students with residency positions, and improving learners, faculty, institution, and patient outcomes.,,, However, the effectiveness of accreditation mechanisms is in doubt recently. Accreditation is very time-consuming for faculties and requires some bureaucratic procedures and logistical support.,
Elledge also pointed to necessity of evidence-based research in the accreditation of medical education. In this regard, although quantitative researches are more comfortable and applicable in accreditations, qualitative researches identify the issues which are not easily quantified., Other studies also recommended exploring accreditation outcomes by focusing on the experience of stakeholders and participating professors.,
In Iran, the development of medical universities has necessitated both qualitative and physical improvement of medical education., Hence, packages for reform and innovation in medical education have been developed according to the upstream documents. An institutional accreditation as one of the main missions of the reform programs was executed for the first time in 2015. To meet the accreditation goals, a qualitative study was designed to extract the challenges and opportunities of the first round accreditation. The experiences of the survey visits teams would help evaluate the way that institutional accreditation was implemented. Hence, continuous improvement will be possible in subsequent accreditation rounds. This research explored challenges and opportunities of institutional accreditation through the experiences of survey teams.
| Methodology|| |
This study was a qualitative content analysis research conducted in Iran. The semistructured face-to-face interview was used for data collection lasting 30–60 min. The researchers tried to discover and analyze the experiences of the survey teams in the following areas: (1) requirements of accrediting medical institutions efficiently and effectively, (2) opportunities and challenges of the accreditation process, (3) strategies to improve accreditation process, (4) weaknesses and strengths of accreditation standards, and (6) strategies to improve the competences of the evaluators according to the experiences of the evaluators.
The research team used a purposive sampling approach to select the participants. Participants were members and supervisors of the visit teams having experience in evaluating medical institutes and universities. These people had expertise in educational surveys, management of academic departments, and management of the specialized boards in the ministry of health. Of 59 evaluators, interviewers reached data saturation after interviewing with 11 participants.
All the three interviewers used MAXQDA 2020 (VERBI Software 2019, Berlin, Germany) for data analysis. They typed each recorded interview in verbatim before uploading it to the MaxQDA software. Interviewers pretested questions with three evaluators that were not part of the sample. Then, interviewers held weekly meetings to revise the interview guide. In this way, codes captured by all three coders were categorized, while those captured by one, but not the others were subjected to further analysis. Constant comparative method was used to analyze the data.
Interviewers were Ph.D. students of the Health Management Department of the Isfahan University of Medical Science, Isfahan, Iran. They were specialists in both clinical and nonclinical educational fields. The initial categories and themes were returned back to the participants to give any comments or corrections.
Anonymity of participants was assured by replacing the names with the interviewee's registering codes. Interviews were conducted in a private setting identified in consultation with the participants. This research was approved by the national center for strategic research in medical education.
| Results|| |
In this study, 72% of the participants were men. Six participants were supervisors of the visit teams, and the others were team members. Participants were from nine provinces and four scientific regions. Evaluators' experience in an educational survey ranged from 5 to over 25 years. Eight of 11 evaluators had Ph.D. degrees, four had medical specialist degrees, and one had Master of Science degree.
The research teams categorized the findings on the following three stages: inputs, process, and outputs [Figure 1]. Emergent themes within each stage are described in the text and highlighted in selected quotes.
Incompatible scheduling with accreditation workload
Participants expressed scheduling defects in the following three areas: short duration of the institutional self-study, short interval between accreditation steps, and short-interval survey visits.
All these caused to failure in following the scheduling of operational programs such as ignorance of some necessary steps in accreditation, failure to review the institutional self-study documentations before the survey visit, and shortage of supervisor's time in holding focused discussion sessions before the survey visits.
“The accreditation took issue with the inappropriate time interval between evaluation steps. When the day of visit survey approached (1 week ago), some universities still had not delivered their self-study documents to evaluators. We should coordinate with the secretariat office to track the information from that university. Sometimes, the documents of universities were delivered to us even 48 h before or even 2 h before visit survey day.”
Participants persisted that supporting upstream documents is required to have a law-based accreditation. In addition, policy-makers should assure the compliance of educational accreditation with major goals of the health system.
“Accreditation should not end when one or two accreditation pioneers having authority in the ministry of health replace because heavy workload, infrastructures, and cost spent on it. If we are going to accredit universities just once and do not see the end, it is like a race that has begun but has not ended. A legal structure that is not dependent on any situation is needed to do this. Accreditation should be institutionalized in the health system in a way that does not be contingent on the current minister of health…”
The weakness of the survey instrument
Participants were concerned about the validity and reliability of the survey instrument. Failure of the instrument in the following subjects was discussed more: inconsistency of the instrument to assess universities with different facilities, hidden logic of evaluators in rating as there was no need to mention any evidence for an indicator, and appearance of different interpretations on each qualitative measure.
“I really came to the conclusion that evaluators look at accreditation measures with different glasses and relying on me as an evaluator may create multiple judgments. How can the accreditation commission know that checkmarks were not subjective and were ticked objectively? Because we just ticked measures and do not specify the details. In this case, it is unclear why I have ticked a measure sufficient or insufficient.”
Advantage of the survey instrument
Participants were satisfied with some characteristics of the standards and indicators in the survey instrument. Standards were functional and executable as the instrument designers had deep experience in educational survey, benchmarked standards against patterns, and developed the instrument incompatible with our country. Comprehensiveness of the checklists in the following eight areas was considered as the main strength of the survey instrument: student education, research, student services, mission and objectives, management, resources and facilities, and academics.
“Although I went to two universities that had international accreditation, accreditation should be adapted to the localized criteria for each country, such as its needs, expectations, specifications, facilities, and human resource. Therefore, in addition to international standards, the specific standards for the country should be considered in accordance with the health, educational, and cultural needs. The current accreditation led to achieve this goal. Those who did the accreditation had many years of research work, come together under a committee, and implement the current accreditation.”
The quality of evaluators' survey
Participants believed that different factors contributed to the effect on quality of the survey as follows: (1) motivating evaluators, (2) taking advantage of teamwork opportunity, (3) personal characteristics of the evaluators, (4) ambiguity of some documentation procedures for medical institution, (5) possibility of self-judgment of the evaluators, (6) time limitation, (7) burden of playing multiple roles on team leaders (supervisors), (8) incomprehensiveness of evaluators' training, and (9) selection criteria for evaluators.
“Some evaluators also did not have enough experience, and this affected their judgment. Sometimes, unfortunately, prejudice affected the survey. However, in most cases, cooperation between the teams was appropriate. Very little time during the survey visit and the high volume of documentation and places to be visited were the great challenges…”
“Some universities provided information that was not needed. For example, they submitted a documentary for three to four questions. It was clear that the serious attempt was not made by some parts of the universities (with emphasis), and they did not complete the forms or did not understand at all how to fill those forms.”
“To enhance the competence and ability of evaluators involved in the accreditation process, strategies should be developed, such as analyzing the surveys and judgments, giving feedback to evaluators, and selecting and appreciating superior evaluators.”
“The workload pressure on me was so intensive that I do not think I had experienced such tense in the period of my doctoral thesis. We need to obtain, study, and integrate information quickly. A disadvantage was concerned with coordinating evaluators after they dispersed. There was no opportunity to immediately discuss the survey visit outcomes in a meeting and later it became very difficult for the team leader to work with team members…”
The opportunity of progress for universities
Participants claimed that understanding the importance of accreditation caused to creating commitment to continuous quality improvement, institutionalization of organizational learning culture, and benchmarking.
“The membership of all universities in the survey teams was very impressive in empowering, delegating authority, and decentralizing of regional universities. This was contrary to previous plans that involved first-class universities most of time.”
“I did not expect that universities prepare documentation in such a discipline, order, perseverance, and interest. During the survey visits, when talking with people, many welcomed the accreditation… Accreditation led us to view ourselves and showed a solution to the upgrade…When I go to our college, I am unconsciously looking for standard evidence and trying to find someone who can fix it. It is a very positive point to see that little by little sensitization is creating in all academics in all levels. That means the desire to change…”
Participants believed that this accreditation is a gate for entering to the international accreditation process which would cause to accomplish sustainable development and evolution in medical education.
“The outcomes of the institutional accreditation should be a basis for development, fixing weaknesses, and reinforcement of the strengths in the areas of infrastructure, facilities, academics, personnel, budget allocation, and so on. Accreditation should allow the establishment of interuniversity communication within the country and universities abroad to develop educational infrastructure, exchange of professors and students, and admittance of foreign students.”
Defendable cost opportunity
Accreditation was perceived a new experience that can broaden documentary allocative efficiency and has sustainable effect.
“Although accreditation had cost a lot to people working in practice (because of the time spent that should be allocated to their original work) and for the ministry to hold meetings, in my opinion, the strengths of this work were greater overally.”
Participants criticized for the evaluation of all universities by the survey instrument that is just appropriate to the structure of universities affiliated with the Ministry of Health. On the other hand, they claimed that the level of universities affected accreditation' results. Participants complained about preawareness of high-level universities of accreditation requirements.
“…I do not say that the survey instrument should be changed or limited questions should be design for small universities, but accreditation scores should be classified for different types of university. Comparing all universities with one benchmark is unfair. Facilities, infrastructure, logistic services, and the number of faculties in some universities are not comparable. The more faculties, the more research works.”
“Major universities having a close connection with the ministry of health employed faculty members that worked simultaneously in the ministry. These academics had helped their universities update regularly and get closer to what the ministry looked for. However, small universities had neither access nor information and perhaps ever had not been seen…”
The leading strategies
Finally, participants proposed the following modifications for the next round of the accreditation:
- Employing some of the current experienced evaluators in the next accreditation rounds along with new evaluators
- Holding continuous and interperiod workshops
- Analysis of judgments of evaluators and feedback
- Decreasing concern related to the budget of fixing weaknesses
- The need to accreditation of clinical field education in hospitals
- Distant education of evaluators during accreditation
- Determining the minimum expected evaluation score for different types of universities in a justice-centered way.
- Establish permanent accreditation secretariats in universities.
The final conceptual model was presented according to the context analysis. In this model, the emergent themes in the input stage must be resolved before the survey visit begins. For instance, accreditation sustainability depends on having the legal protection that was established before the survey visit stage. Categories in the process stage show the necessity of preparing the contingency plan. For example, if there is any conflict of interest or bias of the self-judgment, then alternative strategies should be considered in this stage. The outcome stage represents mostly the impact of the first accreditation round. Policy-makers should consider how to use outcome effects to improve both input and process weaknesses.
| Discussion|| |
In this study, emergent themes were divided into three categories [Figure 1]: institutional self-study (inputs), survey visit (process), and postsurvey visit (output). Studies showed that generally the institutional accreditation has three main areas of internal survey, survey visit, and the presentation of the final report in all international institutions.,,, However, how to implement these steps will enhance the quality of an institutional accreditation.
Maladapted scheduling with workload in self-institutional phase of the accreditation exposed the survey teams with two challenges: first, delay in access to self-study documents and second, receiving large amount of documents that could not be evaluated at all. These challenges could reduce the quality of evaluators' survey as they did not study the institutional self-study documents at the time of survey visit. The survey visit should be based on confirming the institutional self-study results. Therefore, universities that delay in submission of their self-study results or do not send documents completely should not be evaluated.,, However, survey teams in our study complained about time limitations in reviewing self-study documents that were sent with delay.
In some studies, providing advanced training and counseling for evaluators have been mentioned as one of the main stages in accreditation., In this study, evaluators were not trained due to the lack of continuous periodic training. Therefore, there was no opportunity to resolve problems and ask questions in justification meetings. In addition, a long interval between the primary training sections and the accreditation executive steps imposed evaluators in a big challenge. In this regard, the interviewees purposed for regional and continuous training that is better than consecutive workshops in a limited time.
Interviewees mentioned that the ambiguity of documentation procedures caused to lack of integrity of information sent by universities. Regarding this, Thakur et al. recommended that if an institutional self-study guide had been prepared and provided before justification meetings, a significant step would have been taken in integration of the sent information. On the other hand, a database can be designed in that a survey report template automatically alerts the inaccuracies of the sent information. Therefore, developing a standard self-study reporting template for the next rounds of accreditation is one of the suggestions of this research.
In general, while accreditation has a significant effect on universities' progress, executive issues and challenges impose a lot of financial burdens simultaneously. Hence, the cost-effectiveness must be guaranteed. As a new experience, the imperfect quality is acceptable to some extent in the first round of accreditation. This new experience implied as defendable cost-opportunity in this study.
The interperiod evaluation is needed between the accreditation intervals., The evaluation preserves the achieved commitment to quality improvement in universities., The commitment is required for ensuring accreditation sustainability even if policy-makers change. Hence, university liaisons that have the authority of accreditation can be assigned to follow the improvements in their own universities between the accreditation periods.
In some accreditation systems, field visit is considered about 4 days or 3 working days. Time limitation emerged when 1-day visit was assigned to each university regardless of universities' ranking. In addition, dispersal of faculties' locations, traffic, shortage of survey teams' members, and observational or interview-based verification of some accreditation standards could lead to poor quality of the accreditation.
Interviewees pointed that the high working pressure due to the burden of multiple roles on team leaders can be solved by employing a specialist coordinator or team secretary. Therefore, the workload of team leaders reduced the incentives for taking responsibility in other visiting teams. However, these leaders might be the most deserving persons for directing visit survey. A team secretary or coordinator in the composition of teams is a person with experience in the field of accreditation. They are responsible for the integration of findings of the visiting teams and determining the compliance of the institution's educational programs with the standards. In another way, the responsibility of preparation, revision, and validation of the survey information can be delegated to an official institution.
Different interpretations on each evaluation measure and hidden logic of evaluators in scoring an indicator were questioned the validity of the survey instrument. Interviewees demanded for constant comparative analysis of different evaluators' opinions in each accreditation domain by national accreditation committee. Although it is recommended to check reliability and validity of checklists before starting field survey visit,,, participants purposed comparing self-institutional study with final reports of the visiting teams.
Rezaeian et al. indicated that the learning culture encourages universities to continuously assess and improve the quality of medical education. In this regard, results showed that accreditation led to institutionalization of organizational learning culture which itself would lead to continuous quality improvement culture.
Limitations and strengths
The interval between the accreditation and interviews is one of the challenges that could lead to information recall bias. On the other hand, distribution of interviewees in different provinces of the country made the face-to-face interview very difficult. However, the research team continued the interview to reach data saturation.
The main strength of this study was using three different coders that require discussion meetings during the analysis process. Therefore, data can be very consistent, precise, and reliable. Besides, we developed a model of accreditation for the next rounds of accreditation.
| Conclusion|| |
In all accreditations, at least three stages of institutional self-study, the survey visit, and postsurvey visit should be considered in accordance with the systematic approach to achieve continuous quality improvement. In this regard, it is important for each university to perform research and development studies between two accreditation periods. Finally, regarding to the outcomes of the postsurvey visit, a corrective proposal can be announced in the two stages of the survey visit and the institutional self-study to the national accreditation commission. In this case, we will use the findings to achieve the prospect of an international accreditation.
The research team would like to thank all the members of accreditation teams who supported the educational accreditation project and also the authorities of the Isfahan University of Medical Science.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Stensaker B, Langfeldt L, Harvey L, Huisman J, Westerheijden D. An in-depth study on the impact of external quality assurance. Assess Eval High Educ 2011;36:465-78.
French E, Summers J, Kinash S, Lawson R, Taylor T, Herbert J, et al
. The practice of quality in assuring learning in higher education. Qual High Educ 2014;20:24-43.
Huggan PJ, Samarasekara DD, Archuleta S, Khoo SM, Heng JJ, Ping CS, et al
. The successful, rapid transition to a new model of graduate medical education in Singapore. Acad Med 2012;87:1268-73.
Blouin D, Tekian A. Accreditation of medical education programs: Moving from student outcomes to continuous quality improvement measures. Acad Med 2018;93:377-83.
Archuleta S, Ibrahim H, Stadler DJ, Shah NJ, Chew NW, Cofrancesco. Perceptions of leaders and clinician educators on the impact of international accreditation. Acad Med 2015;90 Suppl 11:S83-90.
Jagsi R, Weinstein DF, Shapiro J, Kitch BT, Dorer D, Weissman JS. The accreditation council for graduate medical education's limits on residents' work hours and patient safety. A study of resident experiences and perceptions before and after hours reductions. Arch Intern Med 2008;168:493-500.
Jolly P, Lischka T, Sondheimer H. Numbers of MD and DO graduates in graduate medical education programs accredited by the accreditation council for graduate medical education and the American Osteopathic Association. Acad Med 2015;90:970-4
Jaspers MW, Mantas J, Borycki E, Hasman A. IMIA accreditation of biomedical and health informatics education: Current state and future directions. Yearb Med Inform 2017;26:252-6.
Alrebish S, Jolly B, Molloy E. Accreditation of medical schools in Saudi Arabia: A qualitative study. Med Teach 2017;39 Suppl 1:S1-7.
Elledge R. Current thinking in medical education research: An overview. Br J Oral Maxillofac Surg 2018;56:380-3.
Nove A, Pairman S, Bohle LF, Garg S, Moyo NT, Michel-Schuldt M, et al
. The development of a global Midwifery Education Accreditation Programme. Glob Health Action 2018;11:1489604.
Abdall SZ. Quality assurance in sudanese higher education: current status and challenges ahead. J Total Qual Manage 2016;17:1858-6996.
Mirzazadeh A, Hashemi H, Naseripour H, Fotouhi A, Yazdani K, Tavakoli S, et al
. Establishment of new evaluation and accreditation system for graduate medical education in Iran. J Med Educ 2004;5:69-73. [DOI: 10.22037/jme.v5i2.803].
Rezaeian M, Jalili Z, Nakhaee N, Shirazi JJ, Jafari A. Necessity of accreditation standards for quality assurance of medical basic sciences. Iran J Public Health 2013;42:147-54.
Pourabbasi A, HaghdoostA, Harandi MF, Dehghani MR, Halavati I, Akbari H, et al
. Monitoring and evaluation of packages for reform and innovation in medical education; methodology and preliminary results. Teb Tazkyeh 2018;27:1-8.
Committee on the Accreditation of Canadian Medical Schools; 2016. p. 1-46. Available from: http://cacms-cafmc.ca/
.[ December 2019]
Association of American Medical Colleges, editors. Guide to the Institutional Self-Study for Medical Education Programs with Full Accreditation Surveys in 2016-2017. Washington, DC: Liaison Committee on Medical Education; 2015.
D.o.M. Education, editor. TEPDAD, Self-Evaluation Report Guide for Accreditation of Undergraduate Medical Education Programs Outside of Turkey. Turkey: Association for Evaluation and Accreditation of Medical Education Programs (TEPDAD) Ege University, Faculty of Medicine; 2018.
Procedures of the Caribbean Accreditation Authority for Education in Medicine and other Health Professions. 2013-2017. Kingston, Jamaica: The CAAM-HP Secretariat; Revised 2018: 17-49.
Thakur JS, Sharma D, Jaswal N, Bharti B, Grover A, Thind P. Developing and implementing an accreditation system for health promoting schools in Northern India: A cross-sectional study. BMC Public Health 2014;14:1314.
Chandran L, Fleit HB, Shroyer AL. Academic medicine change management: The power of the liaison committee on medical education accreditation process. Acad Med 2013;88:1225-31.
Maniate JM. Redesigning a resident program evaluation to strengthen the Canadian residency education accreditation system. Acad Med 2010;85:1196-202.
Exhibit 10: ADEA principles for accreditation in dental education: (As approved by the 1997 ADEA House of Delegates). J Dent Educ 2017;81:866-7.
Lai CW. Experiences of accreditation of medical education in Taiwan. J Educ Eval Health Prof 2009;6:2.
Boelen C, Woollard B. Social accountability and accreditation: A new frontier for educational institutions. Med Educ 2009;43:887-94.
Mariano ER, Rosenquist RW. The road to accreditation for fellowship training in regional anesthesiology and acute pain medicine. Curr Opin Anaesthesiol 2018;31:643-648.
Gharibi F, Tabrizi JS. Development of an accreditation model for health education and promotion programs in the Iranian primary healthcare system: A Delphi study. Health Promot Perspect 2018;8:155-62.