Faculty's perspective on skill assessment in undergraduate medical education: Qualitative online forum study
Meenakshi P Khapre1, Harshal Sabane2, Sonia Singh3, Rashmi Katyal4, Anil Kapoor5, Dinesh K Badyal6
1 Department of Community and Family Medicine, AIIMS, Rishikesh, Uttarakhand, India
2 Department of Community Medicine, SSR Medical College, Belle Rive, Mauritius
3 Department of Anatomy, Dayanand Medical College and Hospital, Ludhiana, Punjab, India
4 Department of Community Medicine, Rohilkhand Medical College and Hospital, Bareilly, Uttar Pradesh, India
5 Department of Medicine, Peoples College of Medical Sciences and Research Centre, Bhopal, Madhya Pradesh, India
6 Department of Pharmacology, Christian Medical College, Ludhiana, Punjab, India
|Date of Submission||10-Jul-2019|
|Date of Acceptance||10-Oct-2019|
|Date of Web Publication||30-Jan-2020|
Dr. Meenakshi P Khapre
Department of Community and Family Medicine, AIIMS, Rishikesh, Uttarakhand
Source of Support: None, Conflict of Interest: None
BACKGROUND: India is at the nascent stage of competency-based medical education. Faculties trained in medical education are the main driving force for change. The present study explores the perception of faculties about the current practices and problems in medical/dental/nursing undergraduate assessment, barriers to adoption of best practices, and solutions for addressing them.
METHODOLOGY: A qualitative study was designed and data collected through an asynchronous online discussion forum. A group of 31 health professionals (FAIMER fellows selected on the basis of active participation in department of medical education of respective colleges) participated in the forum. An open-ended topic guide with prompts was designed. The forum was initiated by release of discussion topics (threads) at the start of the month and remained in forum throughout the month. Researchers moderated and recorded day-to-day events. All online forum data were coded line by line and analyzed using conventional content analysis.
RESULTS: Four categories generated were: (1) Low utility of current skill assessment system due to low validity and reliability; (2) Barrier in adopting newer assessment tool due to the absence of felt need of faculties and students, mistaken beliefs, and limited resources; (3) Poor implementation of newer assessment tools such as formatives and objective structured clinical examination with no blueprinting; and (4) Solutions proposed were regular formative assessment, criterion-based examination, quality-assured faculty development programs, and administrative support.
CONCLUSIONS: Barriers in adopting newer assessment tools are related to the faculty's perception and resource constraint. This can be addressed by quality-assured faculty development programs and effective implementation of competency-based education.
Keywords: Assessment, competency, faculty, medical education, qualitative study
|How to cite this article:|
Khapre MP, Sabane H, Singh S, Katyal R, Kapoor A, Badyal DK. Faculty's perspective on skill assessment in undergraduate medical education: Qualitative online forum study. J Edu Health Promot 2020;9:20
|How to cite this URL:|
Khapre MP, Sabane H, Singh S, Katyal R, Kapoor A, Badyal DK. Faculty's perspective on skill assessment in undergraduate medical education: Qualitative online forum study. J Edu Health Promot [serial online] 2020 [cited 2020 Jun 2];9:20. Available from: http://www.jehp.net/text.asp?2020/9/1/20/277345
| Introduction|| |
Two decades earlier, assessment in medical education was only in the form of written and oral examination limited to the cognitive domain. Medical schools later faced a variety of challenges from society, patients, doctors, and students. Some of them were exponential growth in knowledge, associated technologic (”disruptive”) innovations, and societal changes as given by Densen. This led to the development of new curricula, introduction of new methods of learning and assessment, and a realization of importance of faculty development. “Changing examination system without changing curriculum has a much more profound effect on the nature of learning than changing curriculum without altering examination system” as said by Miller. This depicts the importance of assessment. Assessment and evaluation are crucial steps in the educational process. Assessment has a powerful steering effect on learning and curriculum. Many universities around the world have recognized importance of assessment and changed their curriculum to competency based that assures, achievement of professional competence at the end of course., In competency-based education system, assessment drives learning by active involvement of trainee in his/her learning and assessment, creation of an authentic environment for learning and assessment, emphasis on direct observation, and use of frequent formative feedback. In India, the existing examination system is predominantly cognitive-based. Faculties being busy in administration, patient care, and research, spare very limited time for updating teaching-learning practices. As a result, students are assessed with the same traditional methods and not able to develop a compassionate doctor–patient relationship. Newer tools such as objective structured clinical examination (OSCE)/objective structured physical examination are used infrequently and ineffectively. Assessment is currently a one-time exercise conducted at the end of the year as a surrogate marker of competence. Assessment which should be consumer-driven is actually provider driven by the medical teacher. Faculties trained in medical education or member of the department of medical education are main driving force for change in the education system. The present study was undertaken to explore the view of these faculties related to the current practices and problems in undergraduate assessment, barriers to adoption of best practices in assessment, and propose solutions for addressing them.
| Methodology|| |
Setting and Participants
The study setting was faculty development 2-year fellowship program at CMCL-FAIMER Regional Institute. The program invites the application annually from health-care fraternity in the Southeast Asian region. After screening process and telephonic interview, 16 fellows from multidisciplinary health-care field were selected based on their involvement in medical education. These fellows together of year 1 and year 2 comprised 31 health professionals from different states of India and abroad having medical, dental, and nursing background. In addition to the onsite session of 1–2 weeks' fellows, actively participated in monthly discussion topic – online forum (Listserv) throughout the year. It was a private group where fellows registered with id and password. Fellows participated as a part of mandatory requirements for fellowship completion. As per FAIMER Data Use, Sharing, Authorship and Ethic policies (updated Dec 2018), electronic consent was obtained from entire Listserv after informing them about the purpose of data collection, plan of analysis, and intent of publication in future. Complete enumeration sampling technique with 31 registered fellows was employed in the current study.
This study was designed as a qualitative online forum  among registered fellows (participants). The online forum was selected because participants belong to different states/countries, and we aspired for a comprehensive understanding of existing assessments system in various health-care institutes.
Online forum topic
The study was an outcome of ML web e-learning monthly discussion forum on topic “Skills assessment” moderated by researchers. Three topics were discussed with prompts as given [Table 1].
Data collection procedure
At the start of moderation month, the asynchronous online forum was initiated by releasing discussion thread on the topic given in [Table 1] in Listserv group and remained for the whole month. Participants could post messages about the topic at their convenience in any form as experiences, conversations, response to other messages with use of pauses, emoji, etc., As all participants were comfortable with English, only English language was used. The number and length of messages were not limited, yet at least one message per topic in a period of first 7 days followed by responding to two fellows during discussion was must as a part of course completion requisite. Answering the topic question and responding to other participants took place at various times, so the exact time of data saturation was difficult to determine. Criteria for focus group data saturation were set at least two responses for each category and subcategories generated, and no new categories further added in consecutive days. SS oversaw the communication, steered discussion with prompts, and maintained netiquettes. MK did record-keeping of discussion on daily basis, memos, and notes. Any technical issue if encountered was solved by site maintenance team of FAIMER. MK posted summary of the topic at the end of 15 days and participants were encouraged to give feedback and add further to discussion.
Online forum data were analyzed simultaneously with data gathering by conventional content analysis using inductive approach. Discussion was printed out as transcripts, were thoroughly read and reread for line by line coding to generate initial codes. These were later grouped and regrouped in subcategories, reduced further to generic categories. Finally, categories were established. This was an ongoing process with data gathering done by two independent researchers (MK and HS) trained in qualitative studies. Any discrepancies, in categorization, were resolved by discussion.
Rigor of study
To ensure reliability and validity of data, Guba and Lincoln's criteria, including credibility, dependability, transferability, and confirmability, was used.
Approaches to improve credibility in this study were existing long-term engagement with participants, member check, and investigator triangulation. Long-term engagement was assured by nature of the FAIMER Fellowship Program. For member checking, discussion points were summarized by the author and posted on the Listserv for cross-checking by participants. Investigator triangulation was achieved by analyzing forum data separately by two independent investigators.
For dependability of data, all stages and processes of the study were recorded daily in detail to make the study process traceable for reviewers.
To ensure the confirmability of data, an audit trail was done by TS (acknowledged).
With regard to the transferability of findings, participant's comments were presented without alterations. Demographic characteristics of participants were reported for further examination by readers.
| Results|| |
Thirty-one fellows participated in the online discussion with most of them having > 5 years of teaching experience. Fifty-one percent belong to pre- and paraclinical, 10% to clinical department, while 4% and 1% were dental and nursing faculty, respectively. There was a wide variety of participants with respect to positions, duration of service, and department [Table 2]. This thread recorded 94 messages.
All online forum data were coded, and finally, four categories emerged as shown in [Table 3].
Low utility of the current assessment system
Participants felt that the current assessment system is not appropriate to create competent physicians, as they lack validity. Some of the problems pointed were poor reliability, unable to assess skills (higher level of Miller pyramid), while only the outcome of skill is assessed. Assessment tools used like ”Higher-order MCQ are criticized for their inability to distinguish whether a high score in exam is due to true knowledge or random guessing” (Associate Professor, Physiology). Long cases assessment of a student, based on their presentation of a single long case is likely to be subjective and dependent on personal and intellectual preference of examiner. It is mostly a rare case and patient–student interaction often goes unobserved. Students are assessed by final diagnosis of the case rather than process of reaching to diagnosis.
As quoted ”While examining the blood pressure of subject, student is judged on basis of correct reading only.” “End posting exams student is only assessed on final diagnosis as examiner does not have so much time to be their during history taking” (Assistant Professor, Community Medicine).
Some newer assessment skill is rebuked to diminish the importance of learning in a real environment like OSCE, while some are patient-driven. In some universities, no separate head is considered for passing in practical and theory.
”Students need not pass OSCEs in each discipline or even specified number of OSCEs in each discipline, as it is an integrated assessment. Students may omit certain disciplines which they find difficult to master and focus on other disciplines and still manage to pass final examination” (Associate Professor, Pediatrics).
”There are student in the university who do not read or attend anesthesia class but still manage to pass surgery exams” (Associate Professor, Anesthesia). Interns are given completion based on a number of procedures conducted and not on how it was done. Participants were of opinion that ”Our syllabus is exam-oriented. On that run, we try to finish it somehow within the time limit and students try to get through exam somehow. So students are just getting examination driven knowledge.” (Professor, Orthopedic) Assessment and teaching/learning activities are not oriented toward educational objective.
Participants have tried to bring change in the assessment system at their level in respective colleges. However, they had to face many barriers related to difficulty in motivating other faculty members. Need for change in assessment tool is not perceived as felt need of faculties and students, as they are not aware or sensitized for newer assessment tools. In addition, faculties have no knowledge of how to do benchmarking of assessment.
”Lack of awareness about need as well as required know-how about benchmarking of assessment. Without this benchmarking, it is difficult to develop strategies and design tools appropriate for its assessment.” (Associate Professor, Medicine).
Bad experience in the past while bringing change in assessment tools with limited support from seniors and colleagues, participants avoid to make further efforts. They are comfortable with current assessment tools, and some feel that formative assessment increases stress among students.
Some external factors that prevent participants from adopting newer assessment techniques are high student–teacher ratio, resistance from senior, and lack of administrative support. There are limited resources in terms of workforce, infrastructure (simulation lab), and workload. Existing policies at workplaces are not in favor of creating the best learning environment.
”Excellence in teaching is not rewarded with promotions. There should be rewards for excellence in teaching on a level similar to research (sorrow emoji).”(Professor, Pediatrics).
Poor implementation strategies
Formative assessment with feedback is not practiced regularly. Workplace-based assessment (WPBA) is not done, and interns are assessed by logbook on required amount of procedures to be done. Students perform the procedures only during the internship period. Blueprinting of paper is not given due importance. Due to the absence of standardized protocol, “Must know” area is not emphasized proportionately. In addition, summative assessment is not implemented uniformly across all medical colleges.
”Ophthalmology paper is for 40 marks in AP and Telangana states (states of India) where same ophthalmology paper is for 100 marks which makes a lot of difference in time given by candidate, preparation and student outcome”(Professor, Ophthalmology).
Solutions suggested were frequent formatives with feedback focusing on proficiency rather than knowledge. ”Need to move from comfort zone of normative procedures to criterion-referenced” (Associate Professor, Anatomy), i.e., the performance of students is assessed against a standard criterion and not just in comparison to others. This will assure competency. Sub-competencies and objectives should be defined as per local need in supervision of Medical Education Department. Accreditation needs to be stringent, “It is the responsibility of accreditation agencies to certify that educational program in medical schools can deliver what they promise. Longitudinal Faculty development program should be made mandatory.”(Professor, Pediatrics).
Participants also suggested solutions, pertaining to teaching-learning practices in resource-constrained settings as follows:
”Availability of cadavers, histology and pathology specimens for teaching and learning without resorting to expensive virtual tools.” (Professor, Anatomy).
”Sharing of resources among all institutions: e. g. online library resources shared at the national level” (Associate Professor, Nursing).
”Efficient Use of developed technological resources and talent in the IT industry for Technology-enhanced learning” (Assistant Professor, Ophthalmology).
There is a need of an optimum combination of learning with technology that is accessible and sustainable.
| Discussion|| |
The main finding in the study was adherence to traditional curriculum, poor assessment methods, lack of faculty development program, and resource constraint. Student assessment is done in traditional way and curriculum is not reformed for the last 21 years. However, step has been taken in the last 3 months to reform medical education in India. Multidisciplinary faculty of various cadre involved in department of medical education of respective colleges is the strength of this study. Due to geographically distant locations, the discussion was based on asynchronous online platform, and therefore, data saturation could not be confirmed. We tried to overcome it by setting up arbitrary criteria for data saturation by brainstorming among researchers. In the current study, it was found that there is a need to review the current assessment system as it tests only “know” and limited “know-how” area of Miller pyramid. The current method of skill assessment is in the form of long case that is criticized to have low utility value (low reliability, low validity, and low impact) as also identified by Olson et al., Norcini, and Ponnamperuma et al. Long-case examination is based on one unobserved case that affects its content validity. It has been observed in other studies too that long cases poorly assess history-taking ability, communication skills, and physical examination skills., Literature review by Memon et al. found lack of transparency and fairness in oral assessment. OSCE though is a reliable tool to assess skill is criticized for depicting an incomplete picture of clinical competence as in reality clinical environment is complex in nature. In line with views of participants related to poor evaluation, a study done by Deswal and Singhal  also found problems in medical education in India, such as outdated curriculum, nonuniform assessment system, and poor internship supervision.
Participants had tried to bring change in assessment tools through fellowship projects but had to face many barriers mostly related to lack of felt need from other faculties/students and misconceptions regarding newer assessment techniques. This may be due to the poor quality of faculty development program. Medical Council of I ndia has mandated the basic medical education course, but with workload to train 40,000 medical teachers by 4000 trained teachers is hampering the quality of workshop., There is a lack of academic leadership with no support from seniors as observed in the current study. Academic leaders could help in engaging purposes with people's moral, building capacity to generate forces for change, understanding change process, developing learning culture and culture of evaluation, and fostering professional development at all possible levels., Participants in the study felt that existing teacher–student ratio is low which is in line with Deswal and Singhal who claimed 30%–40% shortage in medical teachers. Deswal and Singhal also opined about shortage of clinical material and no incentive given to college for establishing good clinical/simulation labs.
Some colleges had initiated newer assessment skills but not effectively implemented. Formatives aid in deep learning while immediate feedback, direct student learning, and highly appreciated by student. This feedback is lacking during the internal assessment. Most of the medical colleges have a fixed standard (50% cutoff) for examination. This compromises validity of proven valid tools like OSCE. Therefore, standard setting is required to improve the quality of assessment. Blueprinting of examination is rarely practice in India., This leads to inadequate coverage of curriculum and educational domain over a stipulated period of time.
The solution suggested were regular formatives with feedback mechanism with the introduction of logbook, OSCE, and other skill assessment tools. Standard setting should be done for assessing the performance quality of students instead of standard cutoff. Multiple assessment methods are necessary to capture all or most aspects of competencies required for medical graduates. Therefore, rather than evaluating by individual methods, student must be evaluated assessment program as a whole. Interns should be assessed by WPBA and criterion-based licentiate examination at the end. As accreditation plays an important role in maintaining quality of medical education, all medical schools should be accredited by the MCI and National Accreditation and Assessment Council. In resource-limited settings, plausible solutions for using effective assessment tool can be managed by sharing of resources between medical colleges, use of technology students to create low-cost “Made in India” models.
Participants were of the view that competencies should be driving force for medical training and curriculum planning. MCI had taken promising steps toward the implementation of CBME with recent release of undergraduate curriculum to be implemented from batch 2019. The findings of this study were based on view of 31 participants, which are very less compared to large number of medical colleges in India. Therefore, to get better insight into implementation of skill assessment and improving the student outcome, we further recommend longitudinal collaborative action research projects.
| Conclusions|| |
The current skill assessment system lacks ability to develop the array of abilities of a fresh graduate, to perform expected roles in providing health care to the community. There are internal barriers (mistaken beliefs, lack of felt need, low motivation, and insufficient knowledge) and external barriers (constraint of resources, unfavorable policies) to successfully adopt newer assessment tools in India. These barriers can be addressed by a quality-assured faculty development program and strategies to combat resource constraints.
We highly acknowledge Dr. Tejinder Singh, Program Director, CMCL, FAIMER Regional Institute, professor, Department of Pediatrics, Christian Medical College, Ludhiana, for being the constant source of inspiration during our journey to FAIMER. Furthermore, we acknowledge all the faculties and fellows who participated in the study and consented for publication.
Financial support and sponsorship
This study was supported by the FAIMER, CMCL, Ludhiana, India.
Conflicts of interest
There are no conflicts of interest.
| References|| |
Densen P. Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc 2011;122:48-58.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63-7.
Assessment methods in medical education. Int J Health Sci (Qassim) 2008;2:3-7.
Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, et al.
Core principles of assessment in competency-based medical education. Med Teach 2017;39:609-16.
Goswami S. Problems and challenges in medical education in India. Eur J Contemp Educ 2015;11:31-7.
Im EO, Chee W. Practical guidelines for qualitative research using online forums. Comput Inform Nurs 2012;30:604-11.
Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9th
ed. Philadelphia: Lippincott, Williams & Wilkins; 2012.
Olson LG, Coughlan J, Rolfe I, Hensley MJ. The effect of a structured question grid on the validity and perceived fairness of a medical long case assessment. Med Educ 2000;34:46-52.
Norcini J. The validity of long cases. Med Educ 2001;35:720-1.
Ponnamperuma GG, Karunathilake IM, McAleer S, Davis MH. The long case and its modifications: A literature review. Med Educ 2009;43:936-41.
Eva KW. On the generality of specificity. Med Educ 2003;37:587-8.
Pavlakis N, Laurent R. Role of the observed long case in postgraduate medical training. Intern Med J 2001;31:523-8.
Memon MA, Joughin GR, Memon B. Oral assessment and postgraduate medical examinations: Establishing conditions for validity, reliability and fairness. Adv Health Sci Educ Theory Pract 2010;15:277-89.
Weersink K, Hall AK, Rich J, Szulewski A, Dagnone JD. Simulation versus real-world performance: A direct comparison of emergency medicine resident resuscitation entrustment scoring. Adv Simul (Lond) 2019;4:9.
Deswal BS, Singhal VK. Problems of medical education in India. Int J Community Med Public Health 2016;3:1905-9.
Zodpey S, Sharma A, Zahiruddin QS, Gaidhane A, Shrikhande S. Faculty development programs for medical teachers in India. J Adv Med Educ Prof 2016;4:97-101.
Evans DJ, Zeun P, Stanier RA. Motivating student learning using a formative assessment journey. J Anat 2014;224:296-303.
Yazbeck Karam V, Park YS, Tekian A, Youssef N. Evaluating the validity evidence of an OSCE: Results from a new medical school. BMC Med Educ 2018;18:313.
Patil SY, Gosavi M, Bannur HB, Ratnakar A. Blueprinting in assessment: A tool to increase the validity of undergraduate written examinations in pathology. Int J Appl Basic Med Res 2015;5:S76-9.
Sood R, Adkoli BV. Medical education in India – Problems and prospects. J Indian Acad Clin Med 2000;1:210-2.
Tavakol M, Dennick R. The foundations of measurement and assessment in medical education. Med Teach 2017;39:1010-5.
van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ 2005;39:309-17.
[Table 1], [Table 2], [Table 3]