Meta-evaluation of published studies on evaluation of health disaster preparedness exercises through a systematic review
Hojjat Sheikhbardsiri1, Mohammad H Yarmohammadian1, Hamid Reza Khankeh2, Mahmoud Nekoei-Moghadam3, Ahmad Reza Raeisi1
1 Health Management and Economics Research Center, Isfahan University of Medical Sciences, Isfahan, Iran
2 Research Center in Emergency and Disaster Health, University of Social Welfare and Rehabilitation Sciences, Tehran, Iran
3 Research Center for Health Services Management, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran
|Date of Submission||01-Nov-2017|
|Date of Acceptance||15-Nov-2017|
|Date of Web Publication||10-Jan-2018|
Prof. Mohammad H Yarmohammadian
Health Management and Economics Research Center, Isfahan University of Medical Sciences, Hezarjerib Street, Isfahan
Source of Support: None, Conflict of Interest: None
Objective: Exercise evaluation is one of the most important steps and sometimes neglected in designing and taking exercises, in this stage of exercise, it systematically identifying, gathering, and interpreting related information to indicate how an exercise has fulfilled its objectives. The present study aimed to assess the most important evaluation techniques applied in evaluating health exercises for emergencies and disasters.
Methods: This was meta-evaluation study through a systematic review. In this research, we searched papers based on specific and relevant keywords in research databases including ISI web of science, PubMed, Scopus, Science Direct, Ovid, ProQuest, Wiley, Google Scholar, and Persian database such as ISC and SID. The search keywords and strategies are followed; “simulation,” “practice,” “drill,” “exercise,” “instrument,” “tool,” “questionnaire,” “ measurement,” “checklist,” “scale,” “test,” “inventory,” “battery,” “evaluation,” “assessment,” “appraisal,” “emergency,” “disaster,” “cricise,” “hazard,” “catastrophe,: “hospital”, “prehospital,” “health centers,” “treatment centers,” were used in combination with Boolean operators OR and AND.
Results: The research findings indicate that there are different techniques and methods for data collection to evaluate performance exercises of health centers and affiliated organizations in disasters and emergencies including debriefing inventories, self-report, questionnaire, interview, observation, shooting video, and photographing, electronic equipment which can be individually or collectively used depending on exercise objectives or purposes.
Conclusion: Taking exercise in the health sector is one of the important steps in preparation and implementation of disaster risk management programs. This study can be thus utilized to improve preparedness of different sectors of health system according to the latest available evaluation techniques and methods for better implementation of disaster exercise evaluation stages.
Keywords: Disaster, emergency, evaluation, exercise, health, preparedness
|How to cite this article:|
Sheikhbardsiri H, Yarmohammadian MH, Khankeh HR, Nekoei-Moghadam M, Raeisi AR. Meta-evaluation of published studies on evaluation of health disaster preparedness exercises through a systematic review. J Edu Health Promot 2018;7:15
|How to cite this URL:|
Sheikhbardsiri H, Yarmohammadian MH, Khankeh HR, Nekoei-Moghadam M, Raeisi AR. Meta-evaluation of published studies on evaluation of health disaster preparedness exercises through a systematic review. J Edu Health Promot [serial online] 2018 [cited 2018 Dec 18];7:15. Available from: http://www.jehp.net/text.asp?2018/7/1/15/222736
| Introduction|| |
Among several components of disaster management, the health centers and affiliated units can reduce physical, financial, and social damage due to disasters by providing the preparedness plans and appropriate strategies.,, In developed countries, most health centers are requested to have codified programs to prepare and improve their abilities to response unexpected events, and these programs should be designed in a way that; the operators, time and methods, and activate or deactivate programs are known, and the ways of hospital transferring and discharge are identified, and information communication and management are taken into account. These programs should be also up-to-dated, and all staff should be adequately familiar with program for disaster. Revision and improvement of health centers' preparedness plans for the proper and timely reaction is major role for reducing damages caused by disasters. Otherwise, carrying out disaster exercises is the most important way to create, maintain, and improve preparedness plans.,
Running exercise courses in different sectors of health system are one of the important steps to prepare and deploy disaster risk management programs, especially response phase. Exercises simulate the realistic conditions so that people improve their mental and physical skills in situ ations similar to real conditions and provide an appropriate response based on existing programs to emergencies and disasters., Disaster exercises can be used for testing and validating policies, programs, procedures, teaching personnel, their roles and responsibilities, as well as improving the individual performance, and improving interorganizational communication and coordination. There are two types of disaster exercises: (a) Discussion-based exercise and (b) Operation-based exercise. Discussion-based exercise includes four types of exercises: Seminars, workshops, games, and tabletop; and operation-based exercise, which is, in fact, the real exercise implementation, consists of 3 types of exercises including drills, functional-scale, and full-scale exercises. Types, features, and objectives of exercises are presented in [Table 1].
|Table 1: Specifications and objective of disasters preparedness exercise.|
Click here to view
Different steps should be taken to carry out an exercise evaluation. Exercise evaluation is important step in designing and implementing exercise. It systematically investigates and indicates how much the exercise has fulfilled its objectives. It also determines strengths and weaknesses of disaster exercise program. After exercise, evaluators should investigate exercises during a session with all key participants and delegates of involved units. This session aims to collect information on individual performance and application of information to revise and improve preparedness program and response process. It should be noted that identification of unskilled and unprofessional staff and finding their fault, error, and poor performance is not among the objectives of the session; and programs will be reviewed and modified based on the available information and results of evaluation at the end of session.,
Evaluation is defined as the review and investigation of value and utilization of phenomena program, plan, policy, or procedure and finding a way for improving the quality of program through utilizing available proper, moral, and accurate methods. Common questions of all evaluations are as follows: Do the components of program have appropriate and effective performance? How is the good performance or best practice? Why do the program and its components do not well? How durable are the program and its consequences? Is this program more effective than other programs? What do we learn about this program? What do customers think about this program? Despite numerous studies on the need for evaluation, unfortunately, evaluation is not effectively an integral part of most programs, and we usually think of a program when we are faced with problems or questions about itself. Evaluation experts believe that all evaluations may be encountered with bias because evaluators' decisions on what they investigate, what methods and tools they use, to whom they talk, and even, their professional and personal experiences, affect the implementation and result of evaluation. Therefore, we should take measures to assess credibility, validity, and accuracy of evaluation program. Therefore, the evaluation of evaluation (meta-evaluation) program should be first included in evaluation program.
Given the importance of health preparedness and important strategy for doing disaster exercise to maintain and promote the preparedness for appropriate and timely response to disasters and reduction of physical, psychological, social, and economic damages of society, the present study is a meta-evaluation taking advantages of a systematic review with the aim of evaluation of related and involved units and organizations at health sector during exercises for emergencies and disasters.
| Methods|| |
The present study was a meta-evaluation through systematic review of published studies relating to evaluation of health preparedness exercises for emergencies and disasters. This study performed based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.
| Search Strategy|| |
This study was conducted during July 2017 to review all published English and Persian articles in the field of evaluation of health disaster preparedness exercises. For this purpose, it has been studied databases including ISI web of science, PubMed, Scopus, Science direct, Ovid, ProQuest, Wiley, Google Scholar, and Persian database such as from January 1, 2000 to June 24, 2017. The search keywords and strategies are followed; “simulation,” “practice,” “drill,” “exercise,” “instrument,” “tool,” “questionnaire,” “ measurement,” “checklist,” “scale,” “test,” “inventory,” “battery,” “evaluation,” “assessment,” “appraisal,” “emergency,” “disaster,” “cricise,” “hazard,” “catastrophe,” “hospital,” “prehospital,” “health centers,” “treatment centers,” were used in combination with Boolean operators OR and AND. Key words were combined and written in search box of databases included ([simulation OR practice OR drill OR exercise] AND [instrument OR tool OR questionnaire OR measurement OR checklist OR scale OR test OR inventory OR battery] AND [evaluation OR assessment OR appraisal] AND [emergency OR disaster OR crisis OR hazard OR catastrophe OR tragedy OR mass casualty incident] AND [hospital OR prehospital OR treatment center OR health center]). All synonyms of the key words were search with using MESH strategies.
Selection of articles and document
Independent reviewers (HS and MN) screened abstracts and titles for eligibility. When the reviewers felt that the abstract or title was potentially useful, full-text copies of the article were retrieved and considered for eligibility by both reviewers. If discrepancies occurred between reviewers, the reasons were identified and a final decision was made based on third reviewer (MY).
Inclusion and exclusion criteria
The inclusion criteria were included: Published papers during 2000 till 2017, Published in English and Persian language, Published in indexed and peer-reviewed research journal and only allocated to disaster exercise evaluation. The exclusion criteria were included: the study were reports or papers which aim to provide experiences in designing and developing exercise without testing performance, evaluation of equipment and emergency essentials, assessment of health preparedness in disasters, evaluation of clinical exercise response including clinical diseases such as respiratory, cardiovascular diseases.
The initial electronic database search of the literature resulted in a total of 5578 documents. At the next step, duplicated, books, dissertations, and presentations were filtered and the number of documents decreased to 2789 articles. Based on systematic screening, at the first stage, it reviewed the titles and abstracts to find those related to evaluation of health exercises for emergencies and disasters and extracted 123 eligible articles. In the next step, all 123 selected full-text papers were considered, and finally, 10 papers which reported evaluation of health preparedness exercises for emergencies and disasters. [Figure 1] shows the search strategy and the selected articles in accordance with the PRISMA guidelines.
Study quality assessment
Quality assessment of the included studies were done using the CASP tools. The score of quantitative studies ranged from 2 to 7 and majority of those did not provide any ethical statement, study design, sampling, and reflexivity related to research process. In the cross-sectional studies, only three out of ten articles,, were used appropriate methods, also majority of them did not consider important confounding factors accounted.
| Results|| |
Demographic of studied
Number of health and treatment centers of these ten studies were consist 34 hospital, 4840 personnel (disaster management expert, staff, evaluator, volunteer, firefighters, and police). Details of each study and their special features were reported regarding exercise type and level, duration, location, year, participants, instrument type, evaluation dimensions, evaluation methods and technique, scoring, validity and reliability, and rescores. The studies were mainly conducted in United States,,,,,,, the Netherlands, Australia, and Italy. According to conducted studies, eight exercises were often operation-based and full-scale,,, and drill,,, and two studies on their exercises were discussion-based and tabletop., Furthermore, the majority of disaster exercises varied from 2 h to 3 days at hospitals and during the exercises. Research results also indicated that most organizations which participated in full-scale exercises of hospitals including firefighting, police officer and infectious disease control center, and voluntary organizations. The results of studies indicated that evaluation of 3 exercises,, was carried out by external evaluator and 6 exercises by internal evaluators,,,, and one exercise by both internal and external evaluators.
Main results and meta-evaluation
Evaluation is among the most important stages of disaster exercises which should have prepared tools before exercise. According to results of study, there are different tools and techniques to collect data for evaluating performance of health and medical systems through disaster exercise and they include self-report (completion of questionnaires by participants), questionnaire, interview, observation, shooting video, photographing, electronic equipment, as well as use of qualitative research techniques such as Delphi to prepare tool items which can be utilized alone or together depending on objective of exercise. Research results indicate that different exercises mostly aimed to enhance self-confidence in employees,,, improve perception of preparedness, increase awareness of roles and responsibilities of our organizations or other organizations which participated in response operations, implement the incident command system, identify gaps and limitations of plans, programs and protocols of health disasters, share information between participating organizations in disaster response program, and provide an opportunity for exercising emergency programs of organizations,,, review and improve interorganizational communications, provide an opportunity for interorganizational collaboration, and development of knowledge, attitude, perception, skill, and behavior in participants.,
The studies indicated that the most important functions of hospitals were evaluated by exercise. Those functions included early warning system; leadership; control; coordination; inter- and intra-organizational communications; risk perception; hospital discharge process; triage; contingent planning; documentation; incident command system; and decontamination and surveillance of communicable diseases. Selection of people for the evaluation of different exercises was the most important finding which was emphasized by studies., The results of studies indicated that selection of evaluators for implementation of standard evaluation processes was one of the most important steps in evaluation process because an evaluators decisions on what they would investigate, what methods and tools they would use, to whom they talk, and even their personal and professional experiences affect implementation and outcome of evaluation; and thus the evaluators' training before evaluation could play significant roles in identification of strengths, weakness, and improvement ability of programs., According to research results, the most important features of a disaster exercise evaluation tool should include experience in design, implementation, and evaluation of disaster exercises; disaster-related academic knowledge and attitude; experience in disaster response and preparedness programs as well as participation in disaster training courses.,,,, Majority of studies utilized standard program of Homeland Security Exercise and Evaluation Program Guidelines to design tools for the disaster exercise evaluation.,, Some of them also reviewed past studies and used panel of experts as the basis for the preparation of evaluation tools. Lack of transparency in examination time of exercise results based on the evaluation method was one of the important points in the assessment of exercise evaluation techniques, and only the research by reported the organizational performance on the basis of evaluation results immediately after holding a joint session consisting of participating officials and staff in exercise. According to the assessment of evaluation techniques, except for 4 evaluation techniques.,,, Majority of tools and scales were designed without any validity and reliability.,,,,, Assessment of evaluation techniques indicated that each of existing tools only evaluated a part of disaster management activities in a health sector including hospitals and health centers; and an evaluation tool could not alone cover health system disaster response programs due to diverse activities. Moreover, th e summaries of each paper related to evaluation of health field preparedness exercises in emergencies and disasters are shown in [Table 2].
|Table 2: Papers summary of health preparedness exercises evaluation in emergencies and disasters|
Click here to view
| Discussion|| |
This systematic paper reviewed the latest evaluation methods and techniques of health exercises to prepare for responding to incidents and disasters. This research determined that among different health centers, most hospitals carried out different operation and discussion-based exercises to be prepared for response to incidents and disasters.,,,,, The results of conducted studies indicated that there were various techniques and methods for the evaluation of health exercises including observation,,,, interview, photography,, shooting video, and use of electronic equipment by Hot wash and Debriefing., Use of any of the abovementioned techniques depended on type and objectives of exercise, various applications, and strengths and weaknesses which should be taken into account by maneuver officials. The studies emphasized that an evaluation technique cannot solely evaluate an exercise in a standard manner and it is better to investigate various dimensions of an index or performance by a combination of different evaluation methods., Disaster experts also believe that the evaluation outcomes are valuable when disaster exercise evaluations are based on the quantitative as well as qualitative data. Studies also indicate that a disaster exercise evaluation method will be superior in the case of considering different items including its ease of use, function-based nature, accuracy, transparency, reliability and validity, and compliance with cultural indices.,,
There is not any study on success and superiority of an exercise evaluation method, but there is also a research on evaluation and effectiveness of one of the evaluation methods including shooting video and photography. Benefits of shooting video as one of evaluation methods include secondary evaluation of individual performance; investigation of participants' performance in exercise based on the time and place; possibility of playing the exercise video for participants to show their performance; recording the exercise document; better evaluation of teamwork; and management, improvement, and help in participants' fast and stable learning; and also behavior improvement compared to the mere verbal feedback.
Studies also indicate that various tools designed for evaluating different disaster exercise; and implementing these tools basically is responsibility at hospitals and are often provided as function-based in checklists, but there is not any comprehensive tool which is applicable to all health sectors including health, treatment and support sector, for designing and implementing a variety of exercises.,,,,,, According to the study on available tools, a majority of them did not have any validation process (validity and reliability) and only 4 studies were reported validation including the validity (face, content, and construct) and reliability.,,, The studies also reported that a function-based tool as well as a set of valid and reliable tools (toolkit) should be designed for exercise evaluation according to a variety of exercises and their objectives.,,
In addition to existence of a valid tool, evaluator's role is one of the most important results of studies on evaluation of health exercises. Some studies utilized the evaluators outside the exercise organization,,, and several studies used the internal evaluators,,,,, and other exercises used both types of evaluators. Studies reported that selection of evaluators was one of the most important stages of design and implementation of exercise evaluation and they emphasized that a person, who was selected as evaluator, should have academic knowledge about disaster management, emergency medicine, disaster preparedness, experience in exercise design, acceptable knowledge on preparedness of fatal incidents, and organizational plans and structure for a preparedness system during an exercise because results of an exercise evaluation could be strongly influenced by evaluator's beliefs, decisions, physical, and mental health; and they emphasized that an evaluator should play an impartial and inactive role and carry out exercise evaluation only on the basis of exercise tool.,,,, The studies also report that it is essential to hold sessions for justification and familiarization of evaluators by application of evaluation tools to evaluate disaster exercise., The most important limitation of this research was the lack of study on effective evaluation tools due to the lack of reported results of evaluation and change and improvement of disaster management programs and structures.
| Conclusion|| |
The results of literature review indicate that there are different techniques and tools which can be used based on the types and objectives of exercises to evaluate health sector exercises and improve preparedness for appropriate response to disasters and incidents. Furthermore, many existing tools are not validated (validity and reliability), and thus the localization and validation stages need to be performed for the scientific use of these tools according to culture of any community. According to an important point of studies, there is not still any scientific document for superiority of existing exercise evaluation techniques, and thus it is suggested that disaster management researchers should conduct interactive studies to assess effectiveness of various types of exercise evaluation techniques in the future. Since every tool and evaluation method can be used to evaluate one or more dimensions of performance in a health sector and given the variety of types and objectives of exercise, this paper recommended developing a valid and reliable tools (toolkit) for exercises evaluation of different dimensions of health feild including hygiene, treatment, education, and logestic. This study can be thus utilized to improve preparedness of different sectors of health system according to the latest available evaluation techniques and methods for better implementation of disaster exercise evaluation stages.
This paper produced from PhD thesis with titled designing and validation of a comprehensive evaluation tool for health field exercises in emergencies and disasters. We would like to thank the Isfahan University of Medical Sciences for financial support of this research.
Financial support and sponsorship
This study was financially supported by Isfahan University of Medical Sciences.
Conflicts of interest
There are no conflicts of interest.
| References|| |
Khankeh H, Khorasani-Zavareh D, Masoumi G. Why the prominent improvement in prehospital medical response in Iran couldn't decrease the number of death related road traffic injuries. J Trauma Treat 2012;1:2167.
Sheikhbardsiri H, Raeisi AR, Nekoei-Moghadam M, Rezaei F. Surge capacity of hospitals in emergencies and disasters with a preparedness approach: A Systematic review. Disaster Med Public Health Prep 2017;11:612-20.
Tavakoli N, Yarmohammadian MH, Safdari R, Keyvanara M. Designing a model of patient tracking system for natural disaster in Iran. J Educ Health Promot 2017;6:77.
Arab M, Zeraati H, Akbari Haghighi F, Ravangard R. A study on the executive managers' knowledge and performance, and their hospitals preparedness against earthquake events and their relationships at public hospitals (affiliated by Tehran University of Medical Sciences (TUMS) 2005-2006). J Health Adm 2009;11:7-14.
Ejeta LT, Ardalan A, Paton D. Application of behavioral theories to disaster and emergency health preparedness: A Systematic review. PLoS Curr 2015;7: pii: ecurrents.dis.31a8995ced321301466db400f1357829.
Khankeh H. Disaster Hospital Preparedness, National Plan. Tehran: University of Social Welfare and Rehabilitation; 2012.
Yarmohammadian MH, Rezaei F, Haghshenas A, Tavakoli N. Overcrowding in emergency departments: A review of strategies to decrease future challenges. J Res Med Sci 2017;22:23.
] [Full text]
Khademipour G, Nakhaee N, Anari SMS, Sadeghi M, Ebrahimnejad H, Sheikhbardsiri H, et al
. Crowd simulations and determining the critical density point of emergency situations. Disaster Med Public Health Prep 2017:1-7.
Parsons E, McAdams T. MDH Homeland Security Exercise and Evaluation Program (HSEEP) Training Activity: OEP-Minnesota Department Of Health. Secur TDH; 2013.
Radi D, McAdams T. MDH Homeland Security Exercise and Evaluation Program (HSEEP) Training Activity: EPR-Minnesota Department Of Health; 2007.
Henstra D. Evaluating local government emergency management programs: What framework should public managers adopt? Public Adm Rev 2010;70:236-46.
Adib HM, mousavi MS, Lotfi MS, Aminolroayaee E. Evaluating the quality of lesson plans by nursing and midwifery faculty members of selected nursing schools in Iran. Educational development of jundishapur 2013; 4:25-33.
Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med 2009;360:1418-28.
Wang L, Antoun C, Sanders R, Nichols E, Olmsted-Hawala EL, Falcone B, et al
. Experimentation for Developing Evidence-Based UI Standards of Mobile Survey Questionnaires. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM; 2017.
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med 2009;151:264-9, W64.
Singh J. Critical appraisal skills programme. J Pharmacol Pharmacother 2013;4:76. [Full text]
Djalali A, Carenzo L, Ragazzoni L, Azzaretto M, Petrino R, Della Corte F, et al
. Does hospital disaster preparedness predict response performance during a full-scale exercise? A pilot study. Prehosp Disaster Med 2014;29:441-7.
Gebbie KM, Valas J, Merrill J, Morse S. Role of exercises and drills in the evaluation of public health in emergency response. Prehosp Disaster Med 2006;21:173-82.
Savoia E, Testa MA, Biddinger PD, Cadigan RO, Koh H, Campbell P, et al
. Assessing public health capabilities during emergency preparedness tabletop exercises: Reliability and validity of a measurement tool. Public Health Rep 2009;124:138-48.
Ablah E, Scanlon E, Konda K, Tinius A, Gebbie KM. A large-scale points-of-dispensing exercise for first responders and first receivers in Nassau County, New York. Biosecur Bioterror 2010;8:25-35.
Agboola F, McCarthy T, Biddinger PD. Impact of emergency preparedness exercise on performance. J Public Health Manag Pract 2013;19 Suppl 2:S77-83.
Kaji AH, Lewis RJ. Assessment of the reliability of the Johns Hopkins/Agency for Healthcare Research and Quality hospital disaster drill evaluation tool. Ann Emerg Med 2008;52:204-10, 210.e1-8.
Savoia E, Biddinger PD, Burstein J, Stoto MA. Inter-agency communication and operations capabilities during a hospital functional exercise: Reliability and validity of a measurement tool. Prehosp Disaster Med 2010;25:52-8.
Thomas TL, Hsu EB, Kim HK, Colli S, Arana G, Green GB, et al
. The incident command system in disasters: Evaluation methods for a hospital-based exercise. Prehosp Disaster Med 2005;20:14-23.
Haverkort JJM, Leenen LPH. Developing the fourth evaluation dimension: A Protocol for evaluation of video from the patient's perspective during major incident exercises. Disaster Med Public Health Prep 2017;11:594-9.
Curnin S, Heumüller E. Evaluating emergency management capability of a water utility: A pilot study using exercise metrics. Utilities Policy 2016;39:36-40.
Greenberg B, Voevodsky P, Gralla E. A capabilities-based framework for disaster response exercise design and evaluation: Findings from oil spill response exercises. J Homel Secur Emerg Manag 2017;13:1-17.
Jenkins JL, Kelen GD, Sauer LM, Fredericksen KA, McCarthy ML. Review of hospital preparedness instruments for national incident management system compliance. Disaster Med Public Health Prep 2009;3:S83-9.
Kaji AH, Langford V, Lewis RJ. Assessing hospital disaster preparedness: A comparison of an on-site survey, directly observed drill performance, and video analysis of teamwork. Ann Emerg Med 2008;52:195-201, 201.e1-12.
McCarthy ML, Brewster P, Hsu EB, Macintyre AG, Kelen GD. Consensus and tools needed to measure health care emergency management capabilities. Disaster Med Public Health Prep 2009;3:S45-51.
[Table 1], [Table 2]