Identification of the most appropriate variables for measuring the efficiency of Iranian public hospitals: Using Delphi technique
Aziz Rezapour1, Zeynab Foroughi2, Niusha Shahidi Sadeghi2, Mehran Faraji1, Alireza Mazdaki1, Ali Sarabi Asiabar1, Noureddin Niknam2, Keivan Rahmani3, Saeed Mohammad-Pour4
1 Health Management and Economics Research Center, Iran University of Medical Sciences, Tehran, Iran
2 Department of Health Services Management, School of Management and Medical Information, Iran University of Medical Sciences, Tehran, Iran
3 Health Management and Economics Research Center, Isfahan University of Medical Sciences, Isfahan, Iran
4 Department of Health Economics, School of Management and Medical Information, Iran University of Medical Sciences, Tehran, Iran
|Date of Submission||02-Dec-2018|
|Date of Acceptance||15-Feb-2019|
|Date of Web Publication||29-Jul-2019|
Dr. Saeed Mohammad-Pour
Department of Health Economics, School of Management and Medical Information, Iran University of Medical Sciences, Tehran
Source of Support: None, Conflict of Interest: None
CONTEXT: Selecting variables is a fundamental step in evaluating comparative efficiency because the results of measuring efficiency depend on the used variables.
AIMS: The aim of this study is to provide a comprehensive set of input and output variables for measuring efficiency with an emphasis on application in general hospitals in Iran.
MATERIALS AND METHODS: This study comprised a literature review followed by a Delphi survey process. After extracting the variables from the literature review in order to reach consensus on them and identify the native variables, the researchers used the Delphi technique in three rounds. Thirty Iranian hospital managers, in Alborz, Saveh, Qazvin, Qom, and Hamadan universities, participated in this study. For analysis, the interquartile range (IQR) and median were used. IQR was used to assess the agreement of Delphi panel members.
RESULTS: After literature review, nine indicators were identified as input variables and 11 indicators were identified as output variables. After the proposed changes by Delphi members, 24 input variables and 24 output variables were identified to measure hospital efficacy. Finally, ten variables were selected as inputs and ten variables were selected as outputs to measure the performance of public hospitals in Iran by using the consensus of the members in the Delphi panel.
CONCLUSIONS: This study proposes a framework for selecting the most appropriate variables for measuring the hospital efficiency with an emphasis on nonparametric methods. Choosing variables to measure hospital efficiency requires infrastructure such as an intelligent information system.
Keywords: Delphi technique, efficiency, Iran, public hospital
|How to cite this article:|
Rezapour A, Foroughi Z, Sadeghi NS, Faraji M, Mazdaki A, Asiabar AS, Niknam N, Rahmani K, Mohammad-Pour S. Identification of the most appropriate variables for measuring the efficiency of Iranian public hospitals: Using Delphi technique. J Edu Health Promot 2019;8:140
|How to cite this URL:|
Rezapour A, Foroughi Z, Sadeghi NS, Faraji M, Mazdaki A, Asiabar AS, Niknam N, Rahmani K, Mohammad-Pour S. Identification of the most appropriate variables for measuring the efficiency of Iranian public hospitals: Using Delphi technique. J Edu Health Promot [serial online] 2019 [cited 2019 Dec 5];8:140. Available from: http://www.jehp.net/text.asp?2019/8/1/140/263448
| Introduction|| |
International organizations consider economic growth through efficiency as one of the most important economic goals of the countries. Therefore, organizations' efforts have always been to achieve maximized outcomes and output with spending minimal cost and resources.,, In this regard, international health organizations, such as the World Health Organization, are particularly interested in assessing the performance of health systems., Hence, one of the main goals of the countries is to improve the quality and efficiency of the health system and the optimal use of resources. Hospitals, as the most important consumer units of the resources in the health-care sector, spend the highest costs and health budgets. This declares the importance of evaluating the effectiveness of these complex institutions.,,, Accordingly, several experimental studies have examined the strategic importance of efficiency in hospitals. Small and large private hospitals achieve higher levels of efficiency rather than public hospitals. In fact, the weakness in hospital management leads to waste of money, human resources, buildings, and equipment. By preventing or reducing the waste of resources, they can be used to provide more services, or to develop the accessibility and also improve the quality of hospital services. On the other hand, inefficiency in the utilizing hospital resources is widespread in developing countries. In Iran, despite the increase in the share of the health system in gross domestic product, the use of hospital resources was ineffective after implementation of the Health System Reform Plan in 2014.
As a matter of fact, hospital is an organization that simultaneously faces multiple inputs and outputs, and it is not easy to assess its efficiency. The methods of assessing efficiency are as follows:
Parametric methods estimate Frontier production functions with econometrics approach and calculate technical efficiencies of individual firms in an industry. However, nonparametric method uses the linear programming model to determine the best performance in a sample and then, the efficiency is measured based on the differences in the observed amounts. Data envelopment analysis (DEA) is one of the most common nonparametric methods.,,,, Regarding the natural characteristics of hospital's services (its multidimensions), DEA is a special nonparametric method and a powerful tool for measuring hospital productivity.,,,
Despite the various methods for measuring the performance of health-care organizations, such as balanced score card, legal inspections, third-party assessments, and statistical indicators, there is no agreement on using the appropriate measurement method for the performance of these organizations. The study of the literature reveals the diversity of techniques and the variables used in measuring hospital efficiency in developing countries. The selection of input and output variables is an essential step in comparative performance evaluation because the results of performance measurement methods depend on the variables used. While a wide range of variables have been used to measure the efficiency of hospitals in developed countries, few efforts have been made in developing countries to provide a comprehensive framework for selecting the most appropriate variables.
Therefore, according to the importance of selecting the most suitable variables in hospital efficiency studies and the lack of a specific framework for presenting these variables, this study is conducted with the propose of identifying and providing input and output variables for measuring the efficiency of hospitals with the emphasis on its application in general hospitals in Iran.
| Materials and Methods|| |
After initial extraction and structured the input and output variables to measure hospital efficiency with the goal of reaching consensus and agreement on and identifying indigenous variables, the researchers used a comprehensive overview and Delphi technique in three stages. In addition, in this research, Delphi technique was used for two reasons including geographic dispersion and impossibility of running face-to-face interviews.
In this research, the modified Delphi technique was used, and several modifications were made in various questions (open/closed) and their (qualitative/quantitative) analysis method. According to the scope of the research which was limited to hospitals of the Ministry of Health in Iran, 50% (thirty persons) of the managers of Iran's medical sciences hospitals (based on the classification of education hub in the Ministry of Health), including universities of Iran, Alborz, Saveh, Qazvin, Qom, and Hamadan medical sciences members of the Delphi panel, were formed.
[Table 1] shows the profile of the Delphi panel members. The entire Delphi process was conducted during the course of training of hospital managers in Iran University of Medical Sciences.
The selection of the members of the Delphi panel was based on their experience and knowledge about hospital management and improving the hospital efficiency, their willingness to participate in the study, and the availability of this group for researchers. In order to have a high accountability rate in starting the Delphi study, the research's information and consent forms to participate in the Delphi study were submitted to the members.
In order to determine the input and output variables to assess the efficiency of hospitals, the first review of the literature was done. The review of the texts was done according to the research's subject and purpose through method.
For this purpose, the review was conducted in the period of 1999–2018 in databases including the Web of Science, Google Scholar, PubMed, Scopus, and ProQuest. The used keywords included hospital, data envelopment analysis, efficiency, technical efficiency, scale efficiency, Malmquist Index, and productivity.
The used studies included experimental investigations about measuring the efficiency of hospitals and their input and output variables. The studies which entered into the present research were supposed to have the following criteria: (1) being published in English language from January 1999 to January 2018, (2) input and output variables are defined for assessing the efficiency of hospitals, (3) DEA method has been used to measure the efficiency.
Hospital efficiency assessment variables were prepared based on a review of the literature. Then, these variables were investigated by a researcher and a health economist in terms of relevance to Iran's ground conditions. Subsequently, all the variables identified by the researcher, in the form of a list, were submitted to the members of the Delphi panel, and they were requested to indicate their input and output variables for measuring the efficiency in Iranian hospitals if they were not included in the list. Data from the first stage were analyzed using quantitative content analysis. The type of content analysis was conceptual, in which the concept was selected and the number of repetition was counted. Each variable was reconsidered if it seemed to be unclear or needed to be combined to other variable.
All the variables identified in the first stage were given to the members of the Delphi panel in a questionnaire format for scoring each of the variables based on their importance. The rating range was from 1 (unimportant) to 5 (very important). In addition, along with each variable, there was a space for expressing free comments of contributors. At the end of the second round, a report was presented to the panel members, in which the frequencies of the selected responses (distribution of scores) as well as the median scores and interquartile range (IQR) of each variable were presented. According to the researchers, achieving a median of 4 or higher, in the 5-point Likert spectrum, was a criterion for agreement between panel members. This method (definition of a median as an agreement) is one of the most common criteria for reaching consensus in the Delphi approach. Finally, based on the comments and scoring of the group, modifications were made to the variables.
In this step, each member of the Delphi panel received a questionnaire including questions and ratings created by the researcher in the previous stages in order to be reviewed. In fact, the third phase provided an opportunity for the participants to review their information and judgment more clearly. Hence, in the third stage, a structured and graded questionnaire designed by Likert scale was given to the experts. The method of analysis was the calculation of IQR and median. IQR was used in order to assess the success of the experts' agreement., Furthermore, median was calculated for each item so that the agreement for the importance of each item was scored.
| Results|| |
The findings of the first round were as follows:
After researching selected information bases and finding the resulted articles by using DEA, the repeated articles were removed and 35 full-text archives in English language were selected. Twenty potential indexes were recognized to be used in the mentioned studies. There were nine and ten indexes related to, in turn, input variables and output variables. [Table 2] summarizes the results of this finding.
Three rounds of Delphi were done. In the first, second, and third rounds, 26 (86%), 21 (70%), and 15 (50%) people out of thirty possible respondents responded to the designed questionnaire, respectively.
In the first round, the managers announced their view points about maintaining, changing, combining, or removing the extracted variables derived from previous studies. After performing the Delphi members' suggested changes, 24 input and output variables for assessing hospital efficiency were recognized. As indicated in [Table 3] qualitative variables in addition to quantitative ones were suggested by hospital managers with regard to input variables. Geographical condition and the culture of services received were two examples of qualitative variables regarding output variables; hospital-related infections and patients' mortality were known as negative outputs. Few of them represented the most efficient performance in hospital.
The findings of the second round were as follows:
After finalizing variables in first round questionnaires Delphi, the questionnaire to measure the importance of each variable was shared with panel members. As shown in [Table 2], 5 variables (about 20%), and 19 variables (about 80%) of input variables obtained the median 5 and 4 respectively. Regarding output variables, 11 variables (45%) with a median of 5 as the highly important variables and 13 variables (55%) with a median of 4 as important variables were recognized. In addition to the importance assessment, deciding about the agreement on suggested indices was known as the other finding of the second Delphi findings; in other words, out of the 24 input variables, 6 variables with very high agreement (IQR = 0), 15 variables with average agreement (IQR = 1), and 3 variables with two agreement (IQR = 2) were recognized.
Meanwhile, regarding output variables, 7 variables with high agreement and 17 variables with average agreement were recognized. Those variables with median 4 and over and IQR of 1 and <1 were inserted in the third Delphi round in order to final decision-making and agreement.
The findings of the third round were as follows:
In this round, a two-part questionnaire was given to the members. These two parts included, in turn, a report of scoring system for the second round's variables and selected variables achieved from the second round. After finishing the third round and result analysis finally, 10 variables as inputs and 10 variables as outputs were selected to measure the performance of public hospitals in Iran using the consensus views of the participants in the Delphi panel. [Table 4] shows Final variables of hospital efficiency measurement.
| Discussion|| |
Developing countries such as Iran have considered the process of measuring hospitals' efficiency in recent years. Meanwhile, the dependency of comparing the results of hospitals efficiency on selected variables leads to clarify the role of variable selection in measuring efficiency;, However, few studies have provided a framework for selecting performance variables in the hospital. This has led to the use of studies from a variety of variables and, ultimately, to provide ambiguous and different solutions to the causes of various measurements. The present study was conducted to recognize domestic variables for measuring efficiency in Iranian hospitals. Hence, Delphi method was used by thirty managers of the hospitals affiliated to Iran University of Medical Sciences including Iran, Alborz, Saveh, Ghazvin, Ghom, and Hamadan University of Medical Sciences. Delphi research was done in three rounds to achieve alliance. Finally, 10 variables as inputs and 10 variables as outputs were selected for assessing efficiency in Iranian general hospitals as input and output variables.
The validity of these selected variables is important from some mythological dimensions. First, the members of Delphi members were experienced hospital managers who were oriented completely toward all the aspects of hospital assessment.
Furthermore, the selection of variables was done based on the viewpoints of people who were present completely in all the Delphi rounds. In addition, providing report about the experts' viewpoints before every round is known as the strong aspect of the present investigation. However, the managers' disorientation to the assessment models of hospital efficiency and the necessity of regarding the pros and cons of each kind of models constituted the limitations of the present study.
Based on Pabon–Lasso model, only three performance indices including bed turnover, bed occupancy rate, and average length of stay were highly regarded in assessing hospitals' efficiency. Hence, this model cannot satisfy the possibility of assessment of different performance dimensions of hospitals and compare them with each other.
In addition, in parametric methods such as Frontier production function, the assessment of hospital efficiency depends on the type of production function for evaluating the efficiency of the units. Exert of assumption is the limitations of this method for measuring performance. However, in nonparametric methods such as data envelopment analysis independent of any hypothesis, the decision unit's efficiency is measured in comparison with other similar units.
It should be noted that selecting input and output variables is one of the most important aspects of this method.
In most of the studies related to assessing hospitals' efficiency performed by DEA, a series of specific variables were used as input and output variables. A systematic review conducted in Iran demonstrated that the number of humane forces including doctors and nurses and also hospital beds was used as the input variable. Furthermore, the number of surgery cases, admitted patients, outpatients, bed occupancy rate, and average length of stay were used as output variables in the mentioned investigations. Afzali et al. provided a framework to select the most suitable variables to assess efficiency in Iranian hospitals. The findings of this study showed that there is little conceptual transparency for the selection of variables, and the variables that have been selected so far limited. In this study, it has been suggested that, in addition to traditional variables, developmental variables should also be used to obtain a full spectrum of performance and quality of care. Therefore, we should have a more reliable tool for measuring the efficiency of the hospitals. In the present study, members of the Delphi panel have proposed a combination of variables. However, the number of proposed variables is much higher. Based on a general rule in the Data Envelopment Analysis Method, the total number of units to be evaluated should be three times more than or equal to the total input and output variables. Therefore, a series of variables utilized in most of the studies must be used. They should consist of the number of doctors and nurses as input variables and also bed occupancy rate and number of discharged patients as output variables. According to the present study's results and the limitations of DEA method in using all of these variables, it is possible to assess hospitals' efficiency based on different groups of input and output variables by making scenarios. This leads to the analysis of the sensitivity of efficiency scores among various hospitals according to selected input and output variables.
In the present study, some indices such as patient readmission and mortality rates were suggested as hospitals' output variables. It should be noted that the lower index reflects the good performance and efficiency of a hospital. The DEA measures the efficiency of hospitals by using mathematical optimization approaches.
Caballer-Tarazona et al. studied efficiency analysis in three hospital units in Valencia for making suitable guidelines. They utilized DEA nonparametric approach for assessing efficiency.
In this study, the input variables included the number of physicians, the number of beds (the number of beds used for each ward) and the output variables included the number of counseling sessions, the number of successful counseling sessions, and the number of surgical interventions. In this study, it is recommended to use qualitative variables such as satisfaction of consumers as output variables In this study, it has recommended to use qualitative variables such as consumer satisfaction as output and synthetic variables that summarize various variables as input variables. The results of this study showed that not only quantitative indicators such as bed occupancy and the average length of stay should be considered for performance measurement, but also quality indicators such as patient satisfaction as one of the most important outcomes of the hospital should be considered. Nevertheless, assessing efficiency in quantitative fields is less difficult than that in qualitative ones. In addition, the absence of reliable and common criteria among different hospitals to assess qualitative indices is known as the limitation of using these sorts of variables.
In general, various factors influence the selection of variables and assessment of the efficiency of hospitals in addition to the possible results and suggestions.
In this regard, the DEA method has been used in most of the performance measurement studies in Iran's hospitals. Furthermore, the emphasis has been on inputs rather than on outcomes because managers have more control over inputs than outputs. Another limitation on the selection of input and output variables in Iran is the limitation of information systems that do not provide accurate data on the variables required.
| Conclusions|| |
It seems that hospital managers take into consideration quantitative indices in addition to qualitative ones for assessing hospital efficiency Of course, it should be considered that the use of a reliable and commonly used tool between hospitals to assess the quality variables for entering these variables is necessary for assessing the efficiency of the prerequisites. Health system managers and policymakers should note that the limitation of the use of resources and inputs in their health system should focus more on the efficiency and effectiveness of service sector departments and organizations. The use of performance measurement tools based on measurable and reliable variables can be used as a tool for evaluating hospital managers. Furthermore, the results can be utilized as the guidelines by hospital managers to assess and promote hospitals' efficiency. Up-to-date and integrative information systems are one of the main substructures of efficiency assessment. The selection of suitable variables for assessing hospital efficiency needs appropriate substructures. In addition, managers' perceptions of managerial insights and knowledge can be effective in choosing appropriate variable and measuring efficiency in hospitals.
The authors acknowledge the very helpful contributions of the reviewers to the quality of the final manuscript.
Financial support and sponsorship
The project was funded by the Health Management and Economics Research Center at Iran University of Medical Sciences. Ethics code is IR.IUMS.REC1396.32011.
Conflicts of interest
All authors have no conflicts of interest to declare.
| References|| |
Ketabi S, Ahmadi N, Moazam E, Mobasherizade S. Evaluation of the efficiency of medical diagnostic laboratories by the method of data envelopment analysis. J Lab Sci 2015;9:102-9.[In Persian]
Kounetas K, Papathanassopoulos F. How efficient are Greek hospitals? A case study using a double bootstrap DEA approach. Eur J Health Econ 2013;14:979-94.
Afzali H, Moss J, Mahmood M. A conceptual framework for selecting the most appropriate variables for measuring hospital efficiency with a focus on Iranian public hospitals. Health Serv Manage Res 2009;22:81-91.
Mahfooz S, Pouragha B, Abedi Z, Satarivand S. Efficiency evaluation of Shahid Beheshti University of Medical Sciences hospitals in Tehran using data envelopment analysis method. Health Promot Manage 2015;5:11-21.[In Persian]
Jehu-Appiah C, Sekidde S, Adjuik M, Akazili J, Almeida SD, Nyonator F, et al.
Ownership and technical efficiency of hospitals: Evidence from Ghana using data envelopment analysis. Cost Eff Resour Alloc 2014;12:9.
Caballer-Tarazona M, Moya-Clemente I, Vivas-Consuelo D, Barrachina-Martínez I. A model to measure the efficiency of hospital performance. Math Comput Model 2010;52:1095-102.
Azad E, Ketabi S, Soltani I, Bagherzade M. Efficiency analysis and allocation of resources to different parts of Shariati Hospital in Isfahan using data envelopment analysis. Health Inf Manage 2012;8:938-47.[In Persian]
Mehrtak M, Yusefzadeh H, Jaafaripooyan E. Pabon lasso and data envelopment analysis: A complementary approach to hospital performance measurement. Glob J Health Sci 2014;6:107-16.
Mitropoulos P, Mitropoulos I, Karanikas H, Polyzos N. The impact of economic crisis on the Greek hospitals' productivity. Int J Health Plann Manage 2018;33:171-84.
Nikokar S, Ketabi S, Moazam E. Provide a combined model of data envelopment analysis and a spatial analysis for evaluating the performance of hospital managers. Health Inf Manage 2011: 01-11.[In Persian]
Park J, Fowler K, Giebel S. Measuring hospital operating efficiencies for strategic decisions. Int J Bus Soc Sci 2011, 2.13: 67-88.
Salehzadeh R, Ketabi S. Measuring the efficiency of Qom hospitals with data envelopment analysis and analytic hierarchy process. Health Inf Manage 2011;8:479-89.[In Persian]
Goudarzi R, Pourreza A, Shokoohi M, Askari R, Mahdavi M, Moghri J, et al.
Technical efficiency of teaching hospitals in Iran: The use of stochastic frontier analysis, 1999-2011. Int J Health Policy Manag 2014;3:91-7.
Najarzade M, Tourabipour A, Ghasemzade R, Salehi R. Evaluation of Ahwaz hospitals efficiency by data envelopment analysis in the years 85 to 89. J Jundishapur Health Sci 2013;4:77-87.[In Persian]
Torabipour A, Najarzadeh M, Arab M, Farzianpour F, Ghasemzadeh R. Hospitals productivity measurement using data envelopment analysis technique. Iran J Public Health 2014;43:1576-81.
Cheng Z, Cai M, Tao H, He Z, Lin X, Lin H, et al.
Efficiency and productivity measurement of rural township hospitals in china: A bootstrapping data envelopment analysis. BMJ Open 2016;6:e011911.
Lobo MS, Rodrigues Hde C, André EC, de Azeredo JA, Lins MP. Dynamic network data envelopment analysis for university hospitals evaluation. Rev Saude Publica 2016;50:22.
Rahmani N, Keshavarz A, Tabatabayee S, Roohollah K. Assessing the role of ownership on changes in total factors using indicators Malmquist and data envelopment analysis method in Qazvin hospitals. J Payavard Salamat 2013;6:300-10.[In Persian]
Rayens MK, Hahn EJ. Building consensus using the policy Delphi method. Policy Polit Nurs Pract 2000;1:308-15.
Hsu CC, Sandford BA. The Delphi technique: Making sense of consensus. Pract Assess Res Eval 2007;12:1-8.
Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-80.
Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK, et al.
Internet-delivered interventions aimed at adolescents: A Delphi study on dissemination and exposure. Health Educ Res 2008;23:427-39.
Ramanathan R. An Introduction to Data Envelopment Analysis: A Tool for Performance Measurement. SAGE; 2003.
Bowlin WF. Measuring performance: An introduction to data envelopment analysis (DEA). J Cost Anal 1998;15:3-27.
Moradi G, Piroozi B, Safari H, Esmail Nasab N, Mohamadi Bolbanabad A, Yari A, et al.
Assessment of the efficiency of hospitals before and after the implementation of health sector evolution plan in Iran based on Pabon–Lasso model. Iran J Public Health 2017;46:389-95.
Xu GC, Zheng J, Zhou ZJ, Zhou CK, Zhao Y. Comparative study of three commonly used methods for hospital efficiency analysis in Beijing Tertiary Public Hospitals, China. Chin Med J (Engl) 2015;128:3185-90.
Mosadeghrad AM, Esfahani P, Nikafshar M. Hospitals' efficiency in Iran: A systematic review and meta-analysis of two decades of research. J Payavard Salamat 2017;11:318-331.
Kiadaliri AA, Jafari M, Gerdtham UG. Frontier-based techniques in measuring hospital efficiency in Iran: A systematic review and meta-regression analysis. BMC Health Serv Res 2013;13:312.
[Table 1], [Table 2], [Table 3], [Table 4]