VALIDITY AND RELIABILITY OF POSTGRADUATE HOSPITAL EDUCATIONAL ENVIRONMENT MEASURE – BAHASA INDONESIA VERSION

Background: Educational environment is an essential factor influencing performance of medical residents, and unsupportive educational environment is associated with considerable harmful impacts. The Postgraduate Hospital Educational Environment Measure (PHEEM) is a self-administered questionnaire designed to measure residents’ perception of their educational environment. This study aims to establish the validity and reliability of PHEEM Bahasa Indonesia version. Methods: The original PHEEM is translated and modified through an expert panel discussion. Construct validity is measured by corrected item-total correlation. Internal consistency reliability is measured by Cronbach’s alpha. Sample was taken from neurology and radiology residents. Results: The first validation trial was done on 44 neurology residents while the second trial was done in 35 radiology residents. Almost all item scores are correlated with the total score with r > 0,3. Cronbach’s alpha of PHEEM is 0,89 and 0,91 in the first and second trial, respectively. Conclusion: This is the first PHEEM validation study in Indonesia; changes to the instrument were made to maintain face and content validity. While the sample size is small and not selected randomly, PHEEM Bahasa Indonesia version has acceptable validity and reliability to be used in research and evaluation of residency programs.


INTRODUCTION
Indonesian medical residents are assumed to have excellent academic potential, due to high level of competition in enrollment and rigorous selection process.However, observation reveals that a considerable proportion of residents faced problems during their period of study which may be caused by academic or non-academic reasons.Problem in academic performance is a result of complex interactions among many factors, including intrinsic character of the student and extrinsic influences from the educational environment. 1 broad terms, educational environment (EE) or educational climate describes what it is like to be a learner in an educational program which is influenced by the physical environment (safety, food, comfort), emotional climate (security, quality of feedback), and intellectual climate (education planning, participation, relevance). 2 Supportive EE is essential for learning to take place optimally.In contrast, unsupportive EE is found to be associated with students' psychological distress.In the long run, such distress may lead to impaired academic performance and competence, burnout, deficits in professionalism, and mental health problems including suicide and substance abuse. 1,3,4knowledging the importance of positive EE, the World Federation for Medical Education (WFME) requires that all medical training programs evaluate the quality of their EE.The Postgraduate Hospital Educational Environment Measure (PHEEM) is an instrument developed in Dundee University to gauge students' perception of their clinical educational environment. 5A number of studies have shown the validity and reliability of PHEEM and the instrument has been subsequently adapted for use in numerous universities worldwide and diverse residency fields.
From a local perspective, Faculty of Medicine Universitas Indonesia (FKUI) and Cipto Mangunkusumo Hospital (RSCM) regularly rates pre-clinical students' perception of their educational environment using Dundee Ready Education Environment Measure (DREEM).However, no comparable survey has ever been done in any of the university's 33 residency programs.Therefore, this study aims to establish the validity and reliability of PHEEM to measure residents' perception of their educational environment and to facilitate further research in Indonesian medical education.

METHOD
PHEEM is a self-administered questionnaire using a 5-point Likert scale (scores 0-4) with 40 items, divided into 3 domains: perception on role autonomy (14 items), social support (11), and teaching (15).The total score of the whole instrument, and also each of the 3 domains, can be interpreted to reflect the overall students' perception of their EE.For policymaking purposes, specific strength and weaknesses can be gathered from the score of individual items. 5he questionnaire is originally written in the English language.
The initial step of this validation study is developing a suitable translation of PHEEM in Bahasa Indonesia.The questionnaire is translated into a first draft by a certified translator from English to Indonesian.This draft is then discussed with a panel of 5 medical education experts from FKUI and changes to the draft are made if necessary.Face and content validity is also established in this phase.The draft is then back-translated into English and sent to the developer of the original questionnaire to ensure that the adjustments made in the translation process does not deviate from its intended purpose.The final product is PHEEM -Bahasa Indonesia version.
PHEEM is then tested on a sample of residents from two departments in FKUI-RSCM separately to establish validity and reliability.The two trials are planned so that adjustments can be made to improve questionnaire validity and reliability after the first trial.The residents are consecutively selected.Ethical clearance is obtained from FKUI-RSCM Research Ethics Committee.
Construct validity is measured using corrected itemtotal correlation to see whether each item possesses a strong enough correlation to the total score.The threshold is set at correlation coefficient greater than 0,3 (r > 0,3).Reliability analysis is done to measure internal consistency, expressed in Cronbach's alpha. 6The questionnaire is considered reliable if Cronbach's alpha for the whole instrument as well as its 3 domains is greater than 0,6.Study data is managed and analyzed using SPSS version 11.5 for Windows.

Translation
The translation was conducted according to the process outlined in previous section.The expert panel discussion concluded that 4 items need to be modified to maintain face and content validity (Table 1).All modifications were discussed with and approved by one of the developers of the original PHEEM.After the modifications were made and approved by the original developer, the authors and expert panel concluded that PHEEM -Bahasa Indonesia version has sufficient face and content validity.

Validity
The sample for validation tests are 44 neurology residents in the first trial and 35 radiology residents in the second trial.
In the first trial, the score of all but 4 items correlate with total score with r > 0,3 (Table 2).The weaklycorrelated items are number 16, 20, 24, 26, all of which are part of the social support domain.The 4 items were then rephrased for clarity and better suitability with the context of residency training in Indonesia.In the second trial, only item 20 still correlates weakly with the total score (Table 2).However, the r value of 8 other items dropped below 0,3.The correlation between the score of each item and the total score of the whole instrument generally exceeds the threshold set at 0,3.The correlation of 3 out of 4 items with r < 0,3 in the first trial improved in the second trial; however, r value of 8 other items dropped below 0,3.This variation may be caused by the difference in the populations from which the samples were taken, i.e.Neurology and Radiology Departments, and the lack of randomization when taking the samples.
Nevertheless, the authors and expert panel decided to keep all items in the PHEEM -Bahasa Indonesia version because those items represent essential aspects of residency program and can give actionable feedback to medical educators and residency administrators. 5A PHEEM validation study in Greece faced a similar condition and also decided not to remove any item from the instrument.

Reliability
The translated PHEEM questionnaire has a Cronbach's alpha of 0,89 in the first trial and 0,91 in the second trial.Each of the 3 domains also has Cronbach's alpha greater than 0,60 (Table 3).Hence, PHEEM-Bahasa Indonesia version can be considered reliable.The high Cronbach's alpha of PHEEM -Bahasa Indonesia version (0,89 and 0,91 in the first and second trial, respectively) showed that the instrument has excellent internal consistency.The value is similar to previous studies in the United Kingdom (0,91 5 and 0,921 7 ), Greece (0,933 8 ), Iran (0,86 9 and 0,94 10 ), and Sri Lanka (0,84 11 ).Yet, it has also been suggested that Cronbach's alpha greater than 0,90 signals redundancy in the items. 6e value of Cronbach's alpha is influenced by at least three factors: actual reliability of the instrument, number of items in the scale, and number of response options for each item. 12However, because of the reproducibility of Cronbach's alpha for PHEEM in several studies, the authors and expert panel concluded that the Cronbach's alpha for PHEEM -Bahasa Indonesia version reflects the reliability of the instrument itself.
This study is the first in Indonesia to validate an instrument to measure the quality of educational environment at the residency level.As shown by the results of this study, the resulting instrument, PHEEM -Bahasa Indonesia version, has acceptable validity and reliability; hence, it can be used for further research in this field.
The instrument is also practical. 12It is simple enough to be self-administered and short.Even though the authors did not record the time it took residents to complete PHEEM -Bahasa Indonesia version, our observation showed that it can be finished in less than 10 minutes without any remarkable issues.
One of the limitations of this study is the relatively small sample size with a subject-to-item ratio 0,88 and 1,1 in the first and second trial, respectively.While a survey of validation studies revealed that sample size determination is rarely justified in research papers, most studies had a subject-to-item ration equal or greater than 2. Furthermore, the sample for this validation study is not randomly selected.
Considering the modifications necessary in translating the questionnaire, the authors of this study concur with the suggestion made in another PHEEM validation study that it would be beneficial for the developer of the original PHEEM to meet with worldwide translators, validators, and users and discuss the future of PHEEM to ensure its lasting utility across cultural differences, in diverse settings, and through the future.

CONCLUSION
PHEEM -Bahasa Indonesia version has acceptable validity and reliability to measure the quality of educational environment in the residency level.
The instrument can be used to conduct further research on educational environment and to shape educational policies in Indonesian medical residency settings.

Table 1 .
Modifications on PHEEM -Bahasa Indonesia versionThe authors and expert panel decided that 4 items of PHEEM, when translated, need to be altered to better suit local conditions.Item 1 on employment contract is not applicable to Indonesian residents, their educational program; therefore, it is changed into "I am asked to do tasks that are not part of my education".Item 17 on the New Deal was modified to suit Indonesian context, where work hours are mostly decided at the discretion of the head of residency program.

Table 2 .
Corrected item-total correlation of PHEEM -Bahasa Indonesia version