Evaluation of psychometric properties of scales measuring student academic satisfaction: A Systematic review
Pardis Rahmatpour1, Hamid Sharif Nia2, Hamid Peyrovi3
1 Department of Nursing, School of Nursing and Midwifery, Iran University of Medical Sciences, Tehran, Iran
2 Department of Nursing, School of Nursing and Midwifery Amol, Mazandaran University of Medical Sciences, Sari, Iran
3 Department of Critical Care, Nursing School of Nursing and Midwifery, Iran University of Medical Sciences, Tehran, Iran
|Date of Submission||14-Aug-2019|
|Date of Acceptance||07-Sep-2019|
|Date of Web Publication||31-Dec-2019|
Dr. Hamid Peyrovi
Nursing Care Research Center, School of Nursing and Midwifery, Rashid Yasami St., Vali-e Asr St., Tehran
Source of Support: None, Conflict of Interest: None
BACKGROUND: Student satisfaction has an impact on student motivation, recruitment of new students, and retention of existing students. Hence, it is important for researchers and academic institutes to assess student academic satisfaction by valid and reliable scales. The aim of this study was to rigorously assess methodological quality and psychometric properties of scales measuring student academic satisfaction.
METHODS: In this systematic review, databases including Scopus, PubMed, ProQuest, ScienceDirect, and Web of Science, and two Persian databases were searched using relevant keywords such as academic satisfaction, student satisfaction, university satisfaction, campus satisfaction, academic life experience, validation, and psychometric and factor analysis from 1970 to December 2018. Considering eligibility criteria, studies were selected after titles and abstracts screening. The methodological quality assessment was performed by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist and the Terwee quality criteria.
RESULTS: Of 814 retrieved articles, 13 studies were included in the study. Based on the COSMIN checklist, structural validity (84%), content validity (53%), and hypothesis testing (53%) were the most reported properties. One study reported cross-cultural validity, one for criterion validity, and none reported measurement error.
CONCLUSION: The results of our study showed that in spite of ≥48 years of development in student satisfaction scales; however, each scale has at least one “poor” psychometric property based on the COSMIN checklist. Quality appraisal of scales is necessary after developing and performing psychometric process.
Keywords: Academic satisfaction, psychometric testing, systematic review, university student, validation studies
|How to cite this article:|
Rahmatpour P, Sharif Nia H, Peyrovi H. Evaluation of psychometric properties of scales measuring student academic satisfaction: A Systematic review. J Edu Health Promot 2019;8:256
|How to cite this URL:|
Rahmatpour P, Sharif Nia H, Peyrovi H. Evaluation of psychometric properties of scales measuring student academic satisfaction: A Systematic review. J Edu Health Promot [serial online] 2019 [cited 2020 Jul 5];8:256. Available from: http://www.jehp.net/text.asp?2019/8/1/256/274488
| Introduction|| |
Students are the most important and main output of the universities, and it is necessary to identify what is important to students. In this regard, to improve the quality of academic services, and adopt appropriate educational policies for students, continuous monitoring of student satisfaction is imperative. According to dynamic education environment, the results of student satisfaction help higher education institutions to remain in competitive situations.,,
Satisfaction is customer's pleasure resulted from services provided by the organization. In academic setting, student satisfaction definition was referred to Oliver and DeSarbo “the favorability of a student's subjective evaluation of the various outcomes and experiences associated with education.” Academic satisfaction is also defined as “Short-term attitude that results from the evaluation of student experiences with the education service received,” and this attitude has an impact on student motivation, recruitment of new students, and retention of existing students.
Student satisfaction is correlated with some academic outcomes. Previous studies reported the relationship between student satisfaction, retention in the field of the study, and academic achievement.,, Furthermore, evidence showed that student satisfaction leads to academic success that improves student academic motivation; in other words, student with higher academic satisfaction has higher motivation and try more for top grades.
Student academic satisfaction is a multifaceted concept; Lent et al. reported a social cognitive model in engineering students that academic satisfaction would be directly predicted by self-efficacy, outcome expectations, environmental supports, and perceived goal progress. Alves and Raposo tested a conceptual model of student satisfaction in higher education and found that variables image, expectation, quality, and value influence in student satisfaction that the consequences of them were student loyalty and word of mouth from student to student.
Studies have shown that there are significant differences among academic satisfaction of students from different academic level, field of study, country of study, and time of satisfaction assessment. Student satisfaction of postgraduate students was different from undergraduate students, because of maturity, academic ability, their experience and expectations of their educational experience. Regarding the field of study, field of nursing needs more interest and competency, and evidence showed that they had different academic satisfaction levels compared to other students. It should be noted that there are significant differences in the educational system and student satisfaction in each society. Furthermore, as the concept of academic satisfaction depends on the educational structure of universities, it is expected that this concept changes overtime.
According to these factors, there are various scales that have been developed in university student; some are general,,,,,,, and some are for specific groups such as nursing students,,, international students, and sport students. Regarding multidimensional nature of academic satisfaction concept, scales are different in dimensions, and some scales do not cover all the dimensions of academic satisfaction. Hence, it becomes necessary to conduct a systematic review for evaluating the psychometric properties of scales that measuring academic satisfaction for proper selecting and better using of them in academic setting. The aims of this systematic review are to:
- Identify scales that investigate university student academic satisfaction
- Assess the methodological quality of included studies
- Analyze the psychometric properties of the scales.
| Methods|| |
A systematic review of studies that evaluate the psychometric properties of academic satisfaction scales was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.
Electronic databases were searched including Scopus, PubMed, ScienceDirect, and Web of Science, two Persian databases such as SID (https://www.sid.ir/) and MAGIRAN (http://www.magiran.com/) and finally, Google Scholar as a search engine from 1970 to 16 December 2018. Furthermore, ProQuest database was searched to identify relevant theses. Reference lists of all identified articles were also hand searched.
Keywords used in the search were as follows: academic satisfaction, student satisfaction, university satisfaction, campus satisfaction, academic life experience, validation, psychometric, and factor analysis. Persian meaning of “Student Satisfaction” and “Academic Satisfaction” was searched in Persian databases. Keywords used in the search for the different databases are provided in [Table 1].
Eligibility criteria and selection procedure
Published articles in English and Persian that describe the scales' psychometric properties/validation process/cross-cultural evaluation of student satisfaction about academic career in university student were included in this study. Articles with irrelevant subjects (student satisfaction about specific teaching method or training courses), language other than English or Persian, structural equation model or model testing articles, review/systematic review articles, and conference articles were excluded from the study. EndNote (version X8; Thomson Reuters, New York, NY, USA) was used to initially screen for duplicated results. Two authors independently involved screening titles and abstracts in the first stage. Full texts of included articles were assessed carefully for eligibility. Any discrepancy between authors was resolved through joint discussions.
Data extraction was independently conducted by two researchers (one statistical expert and one expert in concept of the study). A data extraction sheet included: first author name, publication year, name of scale, country, target population (students' major), face validity, content validity, construct validity (sample size, factor extraction method, rotation methods, selection of the number of factors, name of factors, and total variance), and reliability (consistency: Cronbach's alpha coefficient, stability: Spearman's correlation coefficient, and interclass correlation (ICC) coefficient).
Two researchers assessed the full texts of articles for methodological quality on the basis of the checklist proposed by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN). The COSMIN checklist assesses different psychometric properties (A = internal consistency, B = reliability, C = measurement error, D = content validity, E = structural validity, F = hypothesis testing, G = cross-cultural validity, H = criterion validity, and I = responsiveness). To analyze the results obtained, a four-point COSMIN score was used. Each item was classified as “excellent” when there was appropriate methodology, “good” when there was insufficient relevant information, but an adequate level of quality was reached, and “fair” when the applied methodology was questionable and “poor” when there was evidence that the methodological process was not correct. A methodological quality score per box is obtained by taking the lowest rating of any item in a box (“worst score counts”). The quality criteria for measurement properties were analyzed according to the criteria of Terwee study. Inter-reviewer consensus was evaluated according to the Cohen's Kappa value. Any discrepancies were resolved through discussion and consensus.
Because the overall analysis of psychometric properties is not possible, a narrative analysis was carried out based on the characteristics of the included articles.
| Results|| |
As shown in the PRISMA flow chart [Figure 1], 814 articles (42 articles from Persian database + 772 articles from English language databases) were found in the initial search. After excluding duplicated and irrelevant studies, 13 studies remained.,,,,,,,,,,,,
|Figure 1: Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart|
Click here to view
Included studies were published from the year 1970 to 2017, and majority of them were in the year 2012 (n = 4). One study was doctoral thesis and other was peer-review original articles that published in journals. Only one study was published in the Persian language. Half of the studies (n = 7) were conducted in the USA,,,,,,, followed by Iran (n = 1), Canada (n = 1), Brazil (n = 1), China (n = 1), India (n = 1), and Pakistan (n = 1). Majority of studies focused on college students (n = 5), undergraduate students (n = 5), and three articles were conducted on nursing students [Table 2].
All studies measured student satisfaction concept. Regard to the study design, one article was cross-cultural evaluation study, and others were studied about psychometric properties.,,,,
Number of scale items and dimensions of included studies were various. Minimum item number was 22 and maximum was 92. Minimum number of dimensions were three in two studies, and two studies had 11 dimensions.,
All studies tested for the internal consistency, two for the test–retest reliability,, two for the criterion validity,, and ten for the construct validity.
Internal consistency was conducted by calculating Cronbach's alpha in all studies. ICC and split-half Spearman–Brown coefficient were reported for stability in reliability. Criterion-related validity with Health-Related Quality of Life Scale (HRQOL-14) was used as criterion scales for criterion validity. Majority of studies had construct validity by principal components factor or principal axis factor analysis (n = 7), exploratory factor analysis (n = 3), confirmatory factor analysis (CFA) (n = 3), and other methods such as known group, inter-scale correlation, and simple common factor analysis. Scales' item explained 46.9%–68.54% of the total variance and some studies did not report it. Other psychometric characteristics of included studies are summarized in [Table 2].
The results of COSMIN quality assessment of 13 included articles are given in [Table 3]. None of these articles had “Excellent” quality in all psychometric properties.
BOX A – Internal consistency
Internal consistency is measured to determine the degree of the interrelatedness among the items on the scale. Quality criteria about internal consistency are adequate sample size (seven per items and >100) AND Cronbach's alpha (s) calculated per dimension AND Cronbach's alpha (s) between 0.70 and 0.95. The COSMIN scores for four studies were “Excellent” and five studies were evaluated as “good” because did not calculate alpha for each dimension/subscale separately,,,, or did not have adequate sample size. Four studies scored as “fair” for Cronbach's alpha (s) <0.70 or >0.95.,,,
BOX B – Reliability
According to the COSMIN checklist, reliability is the extent to which scores have not changed and are the same for repeated measurement under several conditions, for example, overtime (test–retest), by different persons on the same occasion (inter-rater), or by the same persons (i.e., raters or responders) on different occasions (intra-rater). Quality criteria of reliability are ICC or weighted Kappa ≥0.70. Two studies reported reliability criteria, and were evaluated as “Excellent,” and other studies assessed as “poor” because did not mention ICC or Kappa value for scales.
BOX C – Measurement error
The systematic and random error of a score that is not attributed to true changes in the construct is considered as measurement error. Measurement errors of all studies were not reported.
BOX D – Content validity
In COSMIN checklist, the content validity is defined as “the degree to which the content of scale is an adequate reflection of the construct to be measured.” Criteria for quality are a clear description of the measurement aim, the target population, the concepts that are being measured, and the item selection AND target population AND investigators or experts were involved in item selection. Six studies that did not mention who involved in item selection and content validity were evaluated as “good,”,,,,, and others were “Excellent.”
BOX E – Structural validity
Based on the COSMIN checklist, the structural validity is the degree to which the scores of scale are an adequate reflection of the dimensionality of the construct. Studies that perform exploratory or CFA have quality criteria. In this respect, two articles did not report factor analysis, and were evaluated as “fair.”
BOX F – Hypothesis testing
According to the COSMIN checklist, hypothesis testing is the same of construct validity. Quality criteria about this aspect are specific hypotheses were formulated, AND at least 75% of the results are in accordance with these hypotheses. Two studies did not report construct validity and were scored as “poor,”, four did not report enough results and were scored as “good,“,,, and seven studies mentioned construct validity with complete details and were scored as “excellent.”,,,,,,
BOX G – Cross-cultural
According to the COSMIN checklist, cross-cultural is the degree to which the performance of the items on a translated or culturally adapted scale is an adequate reflection of the performance of the items of the original version of the scale. Quality criteria are describing translation process, translating item forward and backward, and independently, adequate sample size, pre-testing the scale, and performing CFA. One study was cross-cultural design and was categorized as “good” because it did not report CFA.
BOX H – Criterion validity
Criterion validity is the degree to which the scores of scale are an adequate reflection of a “gold standard.” Quality criteria for criterion validity are convincing arguments that gold standard is “gold” AND correlation with gold standard is >0.70.
One study performed criterion validity and hypothesized that total score of the Brief Multidimensional Students' Life Satisfaction Scale (BMSLSS) and HRQOL-14 was negatively correlated. Since HRQOL-14 was not gold standard for student satisfaction, and in correlation between scales (BMSLSS and HRQOL-14) was not >0.70, this study was scored as “fair” in criterion validity.
Categories related to responsiveness were not analyzed, because there were no results related to that.
| Discussion|| |
This systematic review identified that the psychometric properties of 13 scales measuring academic student satisfaction. Based on the COSMIN checklist, these scales did not score “Excellent” quality in all psychometric properties. In other words, there is no robust and valid single scale for the measurement of student satisfaction.
In this systematic review, the studies were conducted in different field of study, academic level in different publication time and countries. Although the word nursing as a keyword was not used, three scales were developed for nursing students. This may show the results of the importance of nursing student satisfaction and its impact on the patient care. Some studies were specific for undergraduate students or college students, but others were general. The findings showed that the number of psychometric evaluation publications has significantly increased in the year 2012, while the first published study was in 1970. Regarding the country of publication, the majority of studies were conducted in the USA.
It should be noted that the scale for nursing student satisfaction,, had better quality and addressed essential psychometric properties. Four scales were validated for undergraduate students.,,, These scales had good quality, but two of them did not report total variance. In terms of time of publication, newly published articles had more quality scores. This could be followed by the use of journals writing tool guideline and new statistical methods for psychometrics evaluation of scales. Regard to country of the study, it should be noted that first study that was found about student satisfaction scale was conducted in the USA followed by five other studies in the year 1970 until 2012. Although the time of publication had influence on quality, it is not comparable.
In general, dimensions of scales could be categorized into four themes such as curriculum, facilities, campus, and relationship. Dimensions about teaching approach were categorized into the curriculum and were the largest proportion of total explained variance of student satisfaction in some studies.,, Campus facilities, resources,, and administrative and learning facilities were mentioned as facility subscales. Some dimensions of scales were related to campus such as campus climate,, campus organization, management, university climate, financial and fee/cost,,, and other dimensions about environment.,,,, The relationship between students, admin/staff support, social interaction, and professional social interaction was reported in some scales.
The goal of factor extraction is to maximize explained variance, but since parsimony of scale is important, the aim is to balance two goals using as few factors, as it is adequate in explaining a high proportion of variance. Regardless of the factor extraction method, explained variance in half of the included studies was ≥50%. Maximum total explained variance was 68.54% for Liu et al. study with 58 items and 6 factors. Furthermore, minimum variance explained in Zhai et al.  article with 60 items, and 11 factors were 46.9%.
The COSMIN checklist was used in this systematic review, which is the only standard tool for quality assessment of studies on psychometric properties of scales. The overall quality score was not used in quality assessment of scale, because psychometric properties are not equally important. A low-quality assessment of a scale does not imply that it is unsuitable. Some studies did not state enough information in the article clearly, so it is difficult to assess their quality. All studies have reported internal consistency as reliability, but in some studies, there was no information about other essential properties. Most scales had lack of face validity, stability, measurement error, and responsiveness evaluation, and thus future studies must consider these properties when attempting to validate scales.
Although included studies did not discuss measurement error, the highest methodological quality was the “Professional Sport University Student Satisfaction Survey Scale” in Liu et al. study  that in four boxes of COSMIN checklist scored as “Excellent,” one box “Good,” and one box “Fair.”
| Conclusion|| |
This systematic review provides an overview of 13 scales that measuring student satisfaction in university context. Based on the COSMIN checklist, each study has at least “poor” quality in one box. Results of this study help researchers, managers of educational institutions, and other decision-makers to identify appropriate scales with regard to quality and psychometric properties of them to make accurate assessments of students' academic satisfaction. All of this would help to identify areas for improvement of academic education and make better decisions for students and other stakeholders. Furthermore, it should be noted that quality appraisal of scales is necessary after developing, and future research should pay equal attention to quality of development and validation.
This study was a part of PhD thesis, and the study was approved by the Ethics Committee of Iran University of Medical Sciences (IR. IUMS. REC.1397.1311). We would like to thank Dr. Abbas Ebadi for his valuable advice and help.
Financial support and sponsorship
Iran University of Medical Sciences.
Conflicts of interest
There are no conflicts of interest.
| References|| |
Elliott KM, Shin D. Student satisfaction: An alternative approach to assessing this important concept. J High Educ Policy Manag 2002;24:197-209.
Dennison S, El-Masri MM. Development and psychometric assessment of the undergraduate nursing student academic satisfaction scale (UNSASS). J Nurs Meas 2012;20:75-89.
Haghdoost AA, Rafiei H, Raeisvandi A, Kazemzadeh Y. Satisfaction of Postgraduate students of Kerman University of medical sciences, Iran, with their training program and campus facilities. Strides Dev Med Educ 2015;12:355-65.
Mdakane M, Els CJ, Blignaut AS. An inductively derived research framework for student satisfaction in ODL: The higher education environment. Progressio 2016;38:33-57.
Hakim A. Nursing students' satisfaction about their field of study. J Adv Med Educ Prof 2014;2:82-7.
Oliver RL, DeSarbo WS. Processing of the satisfaction response in consumption: A suggested framework and research propositions. J Consum Satisf Dissatisfaction Complaining Behav 1989;2:1-16.
Onditi EO, Wechuli TW. Service quality and student satisfaction in higher education institutions: A review of literature. Int J Sci Res Publ 2017;7:328-35.
Murphy SC. The First-Year Student Experience: Examining Student Satisfaction and the Use of Learning Communities in theFirst Year of College; 2010.
Jamshidi K, Mohammadi B, Mohammadi Z, Karimi Parviz M, Poursaberi R, Mohammadi MM. Academic satisfaction level and academic achievement among students at Kermanshah University of medical sciences: Academic year 2015-2016. Res Dev Med Educ 2017;6:72-9.
Noughani F, Bayat Rizi M, Zohreh Ghorbani, Ramim T. Correlation between emotional intelligence and educational consent of students of Tehran University of medical students. Tehran Univ Med J 2015;73:110-6.
Gruber T, Fuß S, Voss R, Gläser-Zikuda M. Examining student satisfaction with higher education services: Using a new measurement tool. Int J Public Sect Manag 2010;23:105-23.
Lent RW, Singley D, Sheu HB, Schmidt JA, Schmidt LC. Relation of social-cognitive factors to academic satisfaction in engineering students. J Career Assess 2007;15:87-97.
Alves H, Raposo M. Conceptual model of student satisfaction in higher education. Total Qual Manag 2007;18:571-88.
Muijs D, Bokhove C. Postgraduate student satisfaction: A multilevel analysis of PTES data. Br Educ Res J 2017;43:904-30.
Mai LW. A comparative study between UK and US: The student satisfaction in higher education and its influential factors. J Mark Manag 2005;21:859-78.
Betz EL, Klingensmith JE, Menne JW. The measurement and analysis of college student satisfaction. Meas Eval Guid 1970;3:110-8.
Derry S, Brandenburg DC. Students' ratings of academic programs: A study of structural and discriminant validity. J Educ Psychol 1978;70:772-8.
Hussain N, Bhamani S. Development of the student university satisfaction scale: Reliability and validity. Interdiscip J Contemp Res Bus 2012;4:332-41.
Juillerat S. Investigating a Two-Dimensional Approach to the Assessment of Student Satisfaction: Validation of the Student Satisfaction Inventory; 1996.
Starr AM. College Student Satisfaction Questionnaire Manual; 1971.
Torkzadeh J, Mohtaram M. The validation of student's academic satisfaction scale. High Educ Lett 2014;7:155-76.
Zhai L. Validation of an instrument to measure community college student satisfaction. Community Coll J Res Pract 2012;36:47-58.
Zullig KJ, Huebner ES, Gilman R, Patton JM, Murray KA. Validation of the brief multidimensional students' life satisfaction scale among college students. Am J Health Behav 2005;29:206-14.
Chen HC, Farmer S, Barber L, Wayman M. Development and psychometric testing of the nursing student satisfaction scale. Nurs Educ Perspect 2012;33:369-73.
Hirsch CD, Barlem EL, Barlem JG, Dalmolin Gde L, Pereira LA, Ferreira AG. Cross-cultural adaptation and validation of the nursing student satisfaction scale for use with Brazilian nursing students. Rev Lat Am Enfermagem 2016;24:e2776.
Chadha B, Tomar VS, Rai RS, Dugar A. Measuring international students' satisfaction with higher education in India: Development & validation of INHedPERF model. Int J Appl Bus Econ Res 2017;15:251-65.
Liu L, Wang YS, Wu TJ. Student satisfaction scale development and application for sport management in China. Eurasia J Math Sci Technol Educ 2017;13:1429-44.
Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Qual Life Res 2012;21:651-7.
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al.
Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60:34-42.
Polit DF, Yang F. Measurement and the Measurement of Change: A Primer for the Health Professions. Philadelphia, PA: Wolters Kluwer Health; 2015.
[Table 1], [Table 2], [Table 3]