The effectiveness of in-service training “pharmacopeia home health” based on Kirkpatrick's model: A quasi-experimental study
Mahtab Firooznia1, Amir Hamta2, Sareh Shakerian3
1 Department of Community Based Education of Health Sciences, Virtual School of Medical Education and Management, Shahid Beheshti University of Medical Sciences, Tehran, Iran
2 Department of Social Medicine, Faculty of Medical Sciences, Qom University of Medical Sciences, Qom, Iran
3 Pediatrics Clinical Research Development Center, Qom University of Medical Sciences, Qom, Iran
|Date of Submission||24-Feb-2020|
|Date of Acceptance||14-Apr-2020|
|Date of Web Publication||31-Aug-2020|
Dr. Sareh Shakerian
Department of Community Based Education of Health Sciences, Virtual School of Medical Education and Management, Shahid Beheshti University of Medical Sciences, Valiasr Ave, Tehran
Source of Support: None, Conflict of Interest: None
BACKGROUND: Assessing the effectiveness of in-service training courses in order to achieve the desired goals and reduce the waste of costs and opportunities in the system is necessary. The purpose of this study was to determine the effectiveness of the “pharmacopeia home health” course considering its importance in different aspects using the Kirkpatrick model.
MATERIALS AND METHODS: The present study was a quasi-experimental conducted at community health workers (CHWs) on three levels of reaction, learning, and behavior. In each phase, a valid questionnaire was used to measure the outcome according to the Ministry of Health guidelines with pretest and posttest measurements. The data were analyzed through SPSS 23, using descriptive statistics and repeated measures test and general linear model.
RESULTS: The results of the study showed that at the level of reaction in terms of content and holding, conditions of implementation were favorable. Findings at the level of learning showed that the training course was only effective in enhancing the knowledge and awareness about drug maintenance and had no significant effect on other areas. In the third level, the results of the CHWs' performance showed that in some areas, the results were influenced by the demographic variables.
CONCLUSION: The present study showed the effectiveness of education in different areas using on the Kirkpatrick model. Given the lack of impact of education in some areas in the present study and on the other, it is necessary to consider cooperative learning methods in order to develop the effectiveness of the courses.
Keywords: Behvarzes, community health worker, effectiveness, home health, Kirkpatrick's model, pharmacopeia
|How to cite this article:|
Firooznia M, Hamta A, Shakerian S. The effectiveness of in-service training “pharmacopeia home health” based on Kirkpatrick's model: A quasi-experimental study. J Edu Health Promot 2020;9:218
|How to cite this URL:|
Firooznia M, Hamta A, Shakerian S. The effectiveness of in-service training “pharmacopeia home health” based on Kirkpatrick's model: A quasi-experimental study. J Edu Health Promot [serial online] 2020 [cited 2022 May 24];9:218. Available from: https://www.jehp.net/text.asp?2020/9/1/218/293929
| Introduction|| |
Essential medicine supply is one of the components of primary health care. The Iranian National Health-Care System predicted medications for home health (khanehay behdasht) to maximize access to basic health services. The “pharmacopeia home health” package is one of the health system development programs that are available to community health workers (CHWs) since the early years of the home health establishment. The “pharmacopeia home health” package contains information about medicines that are available to CHWs and prescribed when there is no access to physicians. In rural areas, CHWs are the first that patients refer to, and they treat patients according to guidelines of the Ministry of Health within the scope of authority and medicines of the “pharmacopeia home health.” Another important issue is paying attention to the maintenance and storage of medicines, observation of the supply chain and consumption, due to medicines are one of the most expensive products offered, neglecting to principals of the maintenance it causes spoilage and waste of resources. On the other hand, inappropriate consumption of drugs can exacerbate the condition or cause serious damage., Therefore, prescribing and how to take the drug by the patient must be fully compliant with health system guidelines. In general, in prescription, there is sensitivity to its side effects and costs. It is clear that adherence to the principles of appropriate maintenance and prescription of medicines based on the guidelines of the Iranian Ministry of Health by CHWs requires sufficient knowledge and skills. Nowadays, in-service training is considered as one of the main strategies to increase the knowledge, skills, capability, and efficacy of employees. Therefore, human resource training is not only desirable but also an activity that every organization should devote resources to do that. Given the limited resources of each system, a training is cost-effective that can achieve predetermined goals. There are many models to evaluate and assess educational effectiveness, all of which aim to examine the performance and improvement of training programs. The Kirkpatrick model is one of the most widely used models for the evaluation of training effectiveness introduced by Donald Kirkpatrick in 1959. According to this model, training can be evaluated at four levels of reaction, learning, behavior, and result. This model starts from the first level and then goes to the second and third levels if the time and budget allow; the information of each level is the basis for the evaluation of the next levels.,, In order to ensure the effectiveness of in-service training courses and considering the necessity of strengthening the health-care system and the importance of pharmacopeia in functional tasks of CHWs, the present study was designed and conducted using the Kirkpatrick model.
| Materials and Methods|| |
The current quasi-experimental study with one-group pretest–posttest design was performed in 2018; the ethical issues were considered.
The sample size included all CHWs (n = 70) employed in Qom Province, Iran. The content of the educational intervention consisted of the “pharmacopeia home health” training package approved by the Iranian Ministry of Health. The expected outcomes of the first three levels of the model including reaction, learning, and behavior were measured using the relevant valid instrument.
For the evaluation of the first level, i.e., reaction/satisfaction of the participants, the training evaluation questionnaire issued by the Department of Human Resource Training and Empowerment, Qom University of Medical Sciences, with a reliability of 0.94 was used. This questionnaire examines three areas of content, presentation, and instructors, and scores are given based on a five-point Likert scale. Its content validity was approved by the Department of Human Resource Training and Empowerment, Qom University of Medical Sciences, with the participation of education experts.
In order to evaluate the learning level, the pretest–posttest method including questions related to educational contents to evaluate learners' knowledge, attitude, and behavior (scenario design) in six domains) keeping the medicines, analgesics and fever reducers, diarrhea treatment medicines, parasite infection treatment, pediculosis treatment and antihistamine, and antibiotics). Content validity index (CVI = 0.87) for instruments has been determined and computed based on simplicity, relevancy, and clarity for each item using the opinions of 10 experts including managers, program experts, and training experts. The reliability of the instrument was confirmed by Cronbach's alpha coefficient of 0.88. Cronbach's alpha coefficient was calculated for each of the domains including keeping the medicines 0.75, analgesics and fever reducers 0.77, diarrhea treatment medicines 0.80, parasite infection treatment 0.71, pediculosis treatment and antihistamine 0.82, and antibiotics 0.91.
Pretest was conducted before training. Then, immediately after the presentation of the educational content, the same questions were given as posttest, and to evaluate the lasting knowledge of the CHWs, the questions were asked again from the participants 3 months later.
To evaluate performance, the checklist was developed and validated based on the guidelines of the Iranian Ministry of Health. Content validity was confirmed by the same opinion expert panel. An intra-rater reliability observer was calculated by Pearson correlation coefficient (r = 0.78, P < 0.05) for one observer in 10-day intervals. Performance has been evaluated by an expert observation (researcher) during the 6 months following training.
SPSS software version 23) IBM Inc., Chicago, IL, USA (was employed for data analysis using descriptive statistics, repeated measurements test, and general linear model.
| Results|| |
In the current study, of 70 CHWs, 25 (35.7%) were male and 45 (64.3%) female. The mean age of participants was 38.96 ± 7.9 years, and their mean work experience was 15.73 ± 9.07 years.
Descriptive statistics at reaction level showed that the highest mean score (3.26 ± 0.77) in the educational content domain belonged to the applicability of the content in increasing job-related knowledge [Table 1].
|Table 1: Descriptive statistics of evaluation scores of the content, instructors, and holding of the course|
Click here to view
The results of the analysis of first-level data using generalized linear model showed that demographic variables had no significant relationship with educational content (P > 0.05). However, the satisfaction level of female CHWs was significantly higher than that of their male colleagues (P = 0.02).
The results of the study on learning level based on content provided by the instructors in terms of attitude, knowledge, and behavior showed that the trend of knowledge score (range: 0–88) had the highest change in the three stages of the test [Table 2].
|Table 2: Descriptive statistics of total score in pre- and posttest and lasting knowledge by areas of attitude, knowledge, and behavior|
Click here to view
Post hoc test results showed a significant relationship between pretest and posttest and lasting knowledge in keeping the medicine domain (P < 0.05) [Table 3].
|Table 3: Results of the post hoc test (pairwise comparison) of long-term effect|
Click here to view
Evaluation of the three levels of CHWs based on the content provided by the instructors showed that there was a significant relationship between the variables of gender and age and work experience in home health with the behavior of CHWs only in the domains of prescription of antibiotics and satisfaction of clients with the behavior of CHWs and educations and advice provided by them (P < 0.05), but no significant relationship was observed in other domains (P > 0.05) [Table 4].
|Table 4: Results of the analysis of general linear model in performance domain of community health workers by demographic variables|
Click here to view
| Discussion|| |
The present study aimed at evaluating the effect of home health pharmacopeia training program at three levels of reaction, learning, and behavior.
The results showed that most of the participants were satisfied with the content, the instructors, and conditions of holding the course. Regarding the evaluation of the instructors, the results showed that the most dissatisfaction was attributed to the third instructor due to the lack of using active teaching methods and the poor performance in resolving the ambiguities and answering the questions.
The studies by Dorri et al. and Yoon et al. showed the satisfaction of the learners with professional continuing education programs based on the Kirkpatrick model.
In a study on the evaluation of research workshops based on the Kirkpatrick model, Abdulghani et al. showed that 24% of the participants were satisfied, 53.4% conditionally satisfied, and 22.4% dissatisfied with the program. The results of the study by Hojjati et al. showed the desirable status of the training course. The results of the studies by Farnia et al. and Ahanchian et al. in general education and Rezazadeh Bahadoran et al. were consistent with those of the current study at reaction level.,,
Results of the evaluation of the learning level showed that the training course was only effective in enhancing the knowledge and awareness of CHWs of drug maintenance, but no significant effect was observed in other domains. Therefore, the effectiveness of the training course at the learning level was questionable, except for drug maintenance domain. The results of the present study were in line with those of the studies by Schober and Orangi et al., showing that none of the in-service training courses held in Shiraz, Iran, were effective in teaching methods and techniques, classroom management, students' academic performance, and teachers' professional performance. The results of all the above-mentioned studies were in line with those of the present study, but in contrast, the results of the study by Hojjati (2012) showed that in-service training course for nurses could increase knowledge and skills and cause behavioral change in learners. However, since the current study was almost practical, its results could not be generalized. Yoon et al. also reported the training course effective for medical interns.
Evaluation of the behavior of CHWs in home health showed that in some domains, the effect of age, gender, and work experience was significant on behavior. In the analysis of the data obtained from the domains of prescription of antibiotics and treating clients, it was shown that the mean score of CHWs reduced by 0.03 (standard error = 0.13) with age increase. The results of the study by Farnia et al. were in line with those of the present study on behavior and performance domain. They attributed this reduction to theoretical training and lack of motivation in learners. On the other hand, the results of some studies were not consistent with those of the present study.
Many studies showed that the utilization of modern teaching methods promotes learning at different levels. The results of the study by Heydari et al. showed the effectiveness of training courses using modern teaching methods such as teamwork, brainstorming, and group discussion based on the Kirkpatrick model of promotion at three levels.
The results of the study by Farjad showed that despite increased use of the four-level Kirkpatrick model in education, no adequate attention is paid to new approaches in order to increase the effectiveness of programs. On the other hand, today, with changing cultural and socioeconomic conditions in rural areas of Iran, it is necessary to employ modern teaching methods along with monitoring and surveillance to enhance the level of knowledge and skills of CHWs in order to respond to their educational needs in the best possible manner.
The study was also faced some limitations including inability to measure the performance of CHWs in all domains covered by the checklist, use of scenario instead of observation of actual behavior of the health worker when referring to home health, not providing new pharmacopeias by the health system, inaccessibility of the population covered by the home health to required medicines, and a large gap between holding the training course and implementation of the new pharmacopeia. It is recommended to identify deficiencies and weaknesses of the training program issued by upstream officials, and similar studies should be conducted in different regions considering modern teaching methods and various aspects of program implementation for effective interventions and appropriate policies.
| Conclusion|| |
The results of the present study showed the effect of the evaluation of training courses based on the Kirkpatrick model. Furthermore, given the evidence available in the literatures, the effectiveness of teaching and evaluation from various aspects based on evaluation models, it seems to provide valuable information from different aspects of the program to managers and policymakers and also enhances the effectiveness of the programs and prevents waste of resources.
This study is taken from the master's thesis of Community-Based Education of Health Sciences with the code IR.SBMU.SME.REC.1398.012. We would like to thank the health staff for their cooperation in Qom Province.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Kalantari B, Takian AH. Health Transformation Plan in Health System 92-96. Vol. 1. Tehran; 2018. p. 50-168.
Network Management Center Ministry of Health and Medical Education. Pharmacopeia Home Health. Tehran: Arvij; 2018. p. 1-5.
Sam-Khanian MR. Planning human resource training. Tehran: Mehr Borna; 2005. p. 70-83. [In Persian]
Vajaragh F, Cyrus A. In-Service Training Planning (Planning, Staff Training). 10th
ed. Tehran, the Publisher; 2015. p. 1-75.
Philips JJ. Return on Investment in Training and Performance Improvement Programs. Houston, TX: Gulf Publishing Co.; 1997.
Kirckpatrick DL, Kirkpatrick JD. Evaluating Training Programs. The Four Levels. 3rd
ed. San Francisco, California: Berrett-Koehler Publishers; 2006.
Tang JS, Chen CL, Chuang CC, Chen CJ, Feng JY. A study of learning effectiveness in disaster nursing course based on Kirkpatrick's evaluation model (Conference Paper). AHFE International Conference on Human Factors in Training, Education, and Learning Sciences. Vol. 785. 2018. p. 375-84.
Dorri S, Akbari M, Dorri Sedeh M. Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation. Iran J Nurs Midwifery Res 2016;21:493-7.
Yoon HB, Shin JS, Bouphavanh K, Kang YM. Evaluation of a continuing professional development training program for physicians and physician assistants in hospitals in Laos based on the Kirkpatrick model. J Educ Eval Health Prof 2016;13:21.
Abdulghani M, Shaik HM, Khamis SA, Al-Drees N, Irshad AA. Research methodology workshops evaluation using the Kirkpatrick's model: Translating theory into practice. Med Teacher 2004;36:24-9.
Hojjati H, Mehralizadeh Y, Farhadirad H, Alostany S, Aghamolaei M. Assessing the effectiveness of training outcome based on Kirkpatrick model: Case study. Nursing Management 2013;2:35-42.
Farnia MA, Malekiavarsin S, Farshbaf Joo V. Examining the effectiveness of educational courses presented for the agricultural and non-agricultural members of cooperatives in the East Azerbaijan province by the use of Kirkpatrick model. Cooperative Agric J 2013;1(4):63-84.
Ahanchian M, Sharafi S, Vafaee M, Hajiabadi F. Evaluate the effectiveness of internship program in nursing student using Kirkpatrick's model. RME 2017;9:17-9.
Rezazadeh Bahadoran H, Khosravi Babadi A, Toontab Haqiqi S. Effect of in-service education courses on staff's efficiency of Islamic Azad University South Tehran branch based on Kirk Patrick Model. Economics and Business, 2012;2(1):11-22. [In Persian] 2011;2:11-22.
Schober HM. The effect of inservice training on participating teachers and student in their economics classes. Econ Educ 1984;15:282-95.
Orangi A, Gheltash A, Shahamat N, Yosleyani G. An investigation into the impact of in-service training on the professional performance of teachers of Shiraz. J New Approaches Educ Adm 2011;2:95-114.
Heydari MR, Taghva F, Amini M, Delavari S. Using Kirkpatrick's model to measure the effect of a new teaching and learning methods workshop for health care staff. BMC Res Notes 2019;12:388.
Farjad S. The evaluation effectiveness of training courses in university by kirkpatrick model (case study: Islamshahr University). Procedia Soc Behav Sci 2012;46:2837-41.
Ziabari SS, Shakerian S. Effect of supervision and education on community health workers performance in the field of primary health care. Int J Community Med Public Health 2020;7:437-42.
[Table 1], [Table 2], [Table 3], [Table 4]