CONSTRUCT VALIDITY ASSESSMENT OF DUNDEE READY EDUCATIONAL ENVIRONMENT MEASUREMENT ( DREEM ) IN A SCHOOL OF DENTISTRY

Introduction: Learning environment in educational institutions of medicine and health is an important thing to be evaluated, so it takes a valid and reliable instrument. Dundee ready Educational Environment Measurement (DREEM) is one of the most frequently used instruments to evaluate the learning environment in various countries, including Indonesia. However, some psychometric study cannot prove the construct validity of DREEM. This study aims to assess the construct validity of Indonesian version of the DREEM in Study Program of Dentistry, Faculty of Medicine, University of Sam Ratulangi. Method: A total of 352 medical students were divided into three groups asked to complete the DREEM. Evaluation of the construct validity of DREEM was assessed by calculating the score of internal consistency, testing the validity of the items by Pearson Product Moment Correlation test, and confirmatory factor analysis. Results: 50 items of DREEM proven to have good reliability score (Cronbach alpha 0.883), but the two subscales have lower internal consistency score (alpha Cönbach <0.7). Seventeen items are categorized as bad items (r <0.3), the item composition is different compare to the original instrument, and there are only two items (item No. 43 and 44) in the same subscale which has strong correlation (r > 0.6). Conclusion: Indonesian translation of DREEM proved to be reliable, but the construct validity of this instrument cannot be proven in this study. It is hoped that this DREEM psychometric research continues in various countries as a basis for improving these instruments.


INTRODUCTION
Educational environment is perceived by students as the atmosphere affecting students' behavior in learning. 1An educational institution, especially in medical and health education, should be able to provide good educational environment to create comfortable and effective learning atmosphere for students to learn.Educational environment evaluation also may help an educational institution to assess whether learning process is running effectively according to the aims of learning, so it needs to be performed. 2,3To evaluate educational environment well, a valid and reliable instrument is needed.is one of the most used instrument by the medical, dentistry, and health professions in many countries, including Indonesia.This instrument has been translated into Indonesian in a studies by Rahayu 2 and Soemantri 5 .DREEM has also been proven to have good reliability (Crönbach's alpha > 0.7). 5,6,7owever, in several psychometric studies, such as by Dimoliatis, Vasilaki, Anastassopoulos, Ioannidis, & Roff; 8 Jakobsson, Danielsen, & Edgren; 9 Yusoff; 10 and Hammond, O,Rourke, Kelly, Bennett, & O'Flynn 11 , it was demonstrated that DREEM did not have good construct validity, so that they recommended to perform psychometric studies of DREEM in many parts of the world to gain comprehensive information to review this instrument.This study aims to assess the construct validity of the Indonesian translation of DREEM in the School of Dentistry, Faculty of Medicine, Universitas Sam Ratulangi (SD FM Unsrat), Manado.

Research subjects
Total of 352 students of SD FM Unsrat divided into three groups of students (117 second-year students, 128 third-year students, and 107 fourth-year students) were the population of this study.First-year students were not included in this study because SD FM Unsrat did not accept students in the year 2013.The inclusion criteria of this study was active students (never took any academic leaves) taking classes with competence-based curriculum model.Meanwhile, the exclusion criteria was regular program students.Those students were not included as subjects because they were rarely in campus and difficult to contact.
The Indonesian translation of DREEM readability was tested once by five students of SD FM Unsrat who were not in the research population and were chosen using convenience sampling (SD FM Unsrat students who were known by the researchers).This instrument was then revised according to the suggestions from the readability test and was translated back into English to see the meaning suitability between the English revision and the original instrument.From the evaluation, a different meaning was not found in both the revision and the original versions, so the revision was used in this study.

Data collection
This study was held in May -June 2014 using cross-sectional study design.Data was collected in academic meetings at the end of term 2013/2014 and all students included as subjects were invited in those meetings.Data collection was done separately in those three student groups.Before distributing the questionnaire, researchers first informed the subjects about the background, aims, and benefits of the study for students and the school.Students were asked to fill in the questionnaire voluntarily and were informed that there would be no impact to students' learning and to the relationship between teachers and students if they were not willing to fill in the questionnaire.Students who consented to participate in the study were asked to sign an informed consent to become prospective subjects and were asked not to write their names and student numbers in the questionnaire, except for their year of entry as students in SD FM Unsrat.During the data collection, researchers were in the room to answer questions if there were things students did not understand.Questionnaire was filled by students in a twenty-minute duration.

Data analysis
In this study, construct validity test of DREEM was performed in three ways.First, instrument reliability test was performed by calculating internal consistency value (Crönbach's alpha).An instrument is said to have high reliability if its Crönbach's alpha ≥ 0.70. 12Second, the validity of each item was tested using Pearson's product moment correlation test by correlating each item score with the total score.An item is declared as valid if r > 0.3, 13,14,15 invalid items were then analyzed further.Third, confirmatory factor analysis was performed by using extraction technique of principal component analysis with Varimax rotation.
Confirmatory factor analysis was performed by observing these: 1) samples needed to validate the instrument was at least five subjects per item; 13,15 2) Kaiser-Meyer-Olkin (KMO) value > 0.6 and Barlett's test of sphericity value showed a level of significance (p < 0.05), so that data could be analyzed further; 3) eigenvalue ≥ 1 showed the number of factors formed from the analyzed items; 15 and 4) grouping the items in a factor according to the factor loading value (r > 0.3), items must have correlation value not far different from the other items in the same factor (homogeneous), determining a cutoff point (lowest value of items in a factor), and ensuring that items in the same factor were truly different from the other items in different factors by comparing items grouping in the original instrument.
Pearson's product moment correlation test was also used to assess the correlation between items and it was established that items in the same sub scale must have a strong correlation (r > 0.6).Data analysis in this study used Statistical Package for Social Science (SPSS) program version 16.

RESULTS AND DISCUSSION
Data collection session attended by 253 students, i.e. 78 second-year students (66.6%), 104 thirdyear students (81.2%), and 71 fourth-year students (66.3%).Two hundred and fifty students completely filled the DREEM questionnaire and three students only filled 20% of all the items in the questionnaire, so they could not be analyzed in this study (response rate = 71.62%).
Crönbach's alpha value of 50 items of DREEM was 0.883, but two sub scales, namely students' social self-perceptions and students' academic self-perceptions, had Crönbach's alpha values of 0.320 and 0.594, respectively.Seventeen items (items no. 2, 3, 4, 5, 9, 10, 15, 17, 19, 25, 28, 39, 46, 47, 48, 49, and 50) were invalid items (r < 0.3), so that only 33 items were analyzed further (Table 1).Correlation value (r = 0.387) was determined to be the cutoff point for the first factor.In the second factor, item no.23 (the atmosphere is relaxed during lectures) was shown to have the highest correlation (0.798), while item no.40 (the teachers are well-prepared for their classes) had the lowest correlation (0.3).Correlation value (r = 0.316) was determined to be the cutoff point for the second factor.In the third factor, item no. 8 (the teachers ridicule the students) was shown to have the highest correlation value (0.671), while item no.40 (the teachers are well-prepared for their classes) had the lowest correlation value (0.340).Correlation value (r = 0.471) was determined to be the cutoff point for the third factor.In the fourth factor, item no.27 (I am able to memorize all I need) was shown to have the highest correlation value (0.653), while item no.37 (the teachers give clear examples) had the lowest correlation value (0.313).Correlation value (r = 0.406) was determined to be the cutoff point for the fourth factor.In the fifth factor, item no. 31 (I have learned a lot about empathy in my profession) was shown to have the highest correlation (0.711), while item no.40 (the teachers are well-prepared for their classes) had the lowest correlation value (0.390).Correlation value (r = 390) was determined to be the cutoff point for the fifth factor.Items grouping according to the results of principal component analysis is presented in Table 3.
Table 3. Grouping of the 33 items according to the results of principal component analysis with Varimax rotation.
Factor I ( If factor I was named the sub scale of students' perceptions of learning, factor II was named students' perceptions of atmosphere, factor III was named students' perceptions of teachers, factor IV was named students' social self-perceptions, and factor V was named students' academic self-perceptions, it was seen that only items in the sub scale of students' perceptions of teachers that were in accordance with the original instrument.In the sub scale of students' perceptions of learning, there were three items (items no.21, 26, and 43) that were not in accordance with the original instrument.In the sub scale of students' perceptions of atmosphere, there was one item (item no.24) that was not in accordance with the original instrument.In the sub scale of students' social selfperceptions, there was only one item (item no.14) that was in accordance with the original instrument.Meanwhile, in the sub scale of students' academic self-perceptions, there were only two items (items no. 31 and 45) that were in accordance with the original instrument.Considering the meaning suitability of each statement item and considering its factor loading value, there were three items moved to other factors, although their r value were higher than the cut off points of their previous factors, i.e. item no.43 was moved from factor I to factor II, item no.40 was moved from factor V to factor III, and item no.41 was moved from factor IV to factor V (Table 4).From the results of the correlation between an item and the total item score, 17 statement items were found invalid.Six items (items no.17, 25, 46, 47, 48, 49) were also found invalid in a study by Hammond et al. 11 Meanwhile, one other item (item no.39) was also found invalid by a study by Jakobsson et al. 9 The different invalid items found in a study by Hammond et al., 11 Jakobsson et al., 9 and this study may be due to the learning environment which is very contextual (affected by local culture), so the local culture of the study might cause different students' perceptions of learning environment.DREEM was developed in another country.Therefore, the different culture of the place where the instrument was developed from the culture where this study was held might cause different results.
Thirty-three items were grouped into nine factors according to correlation analysis and this number of factors was also more than in the original instrument.
A study by Wang, Zang, & Shan 16 also demonstrated DREEM items grouped into more than five factors according to the results of factor analysis.This might be due to the small number of samples in this study.Tabbanick & Fidel, cited from Cohen et al., 15 mentioned that if the number of samples was small, the factors formed be many.However, in this study, five factors were taken because the other four factors only contained one or two items and the original instrument only grouped the items into five factors.
Items grouping according to the results of confirmatory factor analysis showed different items order from the original instrument.Only one item was grouped into the same sub scale of students' social self-perceptions like in the original instrument, i.e. item no.14.The results of factor analysis using Varimax method with Kaiser normalization also showed that there were nine items, namely items no.14, 18, 24, 32, 37, 40, 43, 44, and 45, which correlated with more than one factors.It proved that those items were ambiguous (measuring more than one construct).The small number of samples in this study might also be one of the causes items correlated with more than one factors.
Cutoff point values determined for the three sub scales in this study were also quite low (r < 0.4).An item is categorized to be able to depict a construct well if r > 0.7. 14However, Cohen et al. 15 mentioned that there was never a deal regarding the minimum value of a cutoff point for a factor.In this study, cutoff point values < 0.4 in three sub scales were determined because several items (items no. 1, 20, 40, and 42) only had correlation value (r > 0.3) in one factor, not allowing to determine a cutoff point value above 0.4.In this study, all factors had more than three items, so that each of them was allowed to be called a factor (a factor must have three items or more). 15e limitation of this study was the minimal number of samples to get good psychometric assessment results, so that the 17 items found invalid might be valid if more subjects were included.This research results showed DREEM had good reliability.Low reliability was found in two sub scales of DREEM, only 33 valid items, and a different items order compared to the original instrument, generated problems in the construct validity of this instrument.However, the results of this study are expected to add a reference about DREEM psychometric assessment in a cultural context different from previous studies and are able to be used as a consideration to review the construct validity of DREEM.

CONCLUSION
Indonesian translation of DREEM is proven reliable, but the construct validity of this instrument could not be proven in this study.

Table 1 .
Corrected item-total correlation value in each sub scale Factors analysis results showed KMO value of 0.867 and Barlett's test of sphericity showed a level of significance (p = 0.000), so that data for the 33 statement items could be analyzed further.According to the results of the total variance explained test, nine factors were formed.However, only five factors were taken in accordance with the sub scales grouping in the original instrument.Those five factors were able to explain 45.071% of all students' perceptions construct about learning environment.Items grouping in factors are presented in Table2.

Table 2 .
Principal component analysis of DREEM with Varimax rotation.
Michael Andreas Leman, Construct Validity Assessment of Dundee Ready Educational Environment Measurement (DREEM) in a School of Dentistry

Table 4 .
New grouping of the 33 items of DREEM according to the results of the confirmatory factor analysis.