DEVELOPMENT AND VALIDITY AND RELIABILITY TESTS OF PROFESSIONALISM ASSESSMENT INSTRUMENT IN PSYCHIATRY RESIDENTS

Background: The main objective of specialist training program is professionalism and competence. In modern era, the medical professions have to face legal issues. This problem can be overcome by improving professionalism education and routine evaluation of professionalism during specialist training. Psychiatric specialist training in particular is different from other specialist training due to several unique aspects during the course of the training. Therefore, an instrument to assess professionalism of psychiatric residents is required. The objective of this study was to develop an instrument to assess professionalism for psychiatric residents adapted from ACGME (Accreditation Council for Graduate Medical Education). Method: This study was a quantitative non-experimental study consisted of 3 stages (literature review, peer review, and validity and reliability testing). The subjects were residents on training at the Department of Psychiatry, Medical Faculty, Gadjah Mada University. The construct validity testing was conducted with Confimatory Factor Analysis (CFA) and content validity with Pearson’s product moment. Reliability testing was conducted with Cronbach’s alpha. Results: Statistical analyses results showed that instrument was valid (Pearson’s product moment r = 0.302 – 0.797; p = 0.001 – 0.052) and reliable (Cronbach’s alpha = 0.943). Analyses on the subjects score revealed that professionalism was associated with the stage of training. The t-test results showed that subjects in the poststasis stage showed higher professionalism score except for the 4 domain (professional behavior). Conclusion: The instrument that developed was valid and reliable to be use as a mean to assess professionalism for resident in training at the Department of Psychiatry, Medical Faculty, Gadjah Mada University.


INTRODUCTION
Professionalism is the quality and the attitude and is the characteristic of a profession or a professional.Professionalism is an area of the Standards of Competency of Indonesian Doctors which also has a paradigm in the globalization of medical education. 1rofessionalism is the basic contract to hold in the relationship between medical society and general public.[4] In Indonesia, the Medical Legal Aid (LBH Kesehatan) noted 182 cases of malpractice all over Indonesia in 2006-2012, including cases in teaching hospitals.The increased number of charges to medical professions necessitates IPDS (Institusi Pendidikan Dokter Spesialis or Medical Specialist Education Institution) to review professionalism teaching in specialist education.
One of residents' roles to participate to prevent medical errors is to work according to residency professionalism.Residents as direct caregivers to patients work under hospitals and medical schools, so that routine monitoring is needed.According to that need, an instrument is required which is easy to apply and has good validity and reliability to measure residents' professionalism.
This study developed a professionalism assessment instrument for psychiatry residents in Medical Specialty Education Program of Psychiatry, Faculty of Medicine, Universitas Gadjah Mada and was adapted from a milestone-based model.This instrument was not meant to replace already existing evaluation instruments, but to complement existing assessment methods.A milestone-based model instrument stresses on the accepting attitude of global society of medical education which is interdependent and has mutual benefits.

METHOD
This is an observational, non-experimental study with cross-sectional design to test the validity and reliability of an instrument being developed to assess psychiatry residents' professionalism.This study took place in the Department of Psychiatry, Faculty of Medicine, Universitas Gadjah Mada/Yogyakarta Dr. Sardjito Hospital in July 2016.The subjects were all psychiatry residents of Faculty of Medicine, Universitas Gadjah Mada.
The instrument used in this study was a personal data form to obtain sociodemographic and personal data of the subjects.The professionalism questionnaire being developed was a self-rating inventory adapted from Psychiatry Milestone-ACGME 2015 with 4-scale likert rating. 5Professionalism was assessed using scoring system according to the total score of all responded items, then it was converted to T-score.
Psychiatry residents' professionalism was the independent variable and instruments' validity and reliability were the dependent variables.Confounding variables in this study were age, sex, previous medical school, semester, marital status, and occupation.Sociodemographic characteristics were analyzed descriptively as frequency distribution.Frequency and percentage were used for categorical variables, while means and standard deviation were used for quantitative variables.
Construct validity test was done with factor analysis, i.e. the Confirmatory Factor Analysis (CFA).Content validity test was done with Pearson's product moment test, i.e. measuring the correlation between the score of each item and the total score.In this study, reliability test was done by analyzing Cronbach's alpha as the estimation of the consistency among items in the instrument/inventory. 6

RESULTS AND DISCUSSION
In this study, 3 validity test were done, namely the face validity, sampling validity, and the correlation between Pearson's product moment among items and the total score.The instrument being developed in this study had been reviewed by a psychiatrist and a psychologist who both had already had more than 10 years of experience.The appearance of the inventory and writer's grammar had been corrected to fulfill professional judgment.Good face validity was expected to positively correlate with the credibility of the instrument.The final results of a credible instrument were expected to get good appreciation and understanding from the subjects/respondents. Content validity was measured statistically using Pearson's product moment.The table's r value for 42 samples (n) was 0.304.This analysis showed 3 invalid questionnaire items, i.e. item no. 8, 26, and 34.These items were not included in subsequent analysis.The range of r value for invalid items was 0.307 -0.797 (p = 0.001 -0.0048).The three eliminated items were from 3 different sub competency domains.The original domain table from ACGME consists of columns which represent the first to the fifth years of residency.Each of the columns then divided into 5 rows of professionalism sub competency.The most left column is "has not achieved level 1" column and is addressed for fresh residents who are just accepted.In this study, 4 new residents were put in that "has not achieved level 1" column because subsequent analyses divided the residents into several groups: ward, psychiatry polyclinics, and post-rotation.New residents had already passed the phase of observing the environment of the Department of Psychiatry, Faculty of Medicine, Universitas Gadjah Mada/Dr.Sardjito Hospital along with their mandatory tasks.
The final domain sampling mapping (area) after three items had been eliminated (item no. 8, 26, and 34) resulted in adequate logic validity because the sub competency areas were still represented evenly.Those three items were not valid because they gave too homogeneous score contribution towards the total score in Pearson's product moment test.It might happen because the items were too common/normative they gave uniform answers in all respondents or they were too hard to understand they made the respondents difficult to respond. 6 ensure that the factors in the scale/inventory had the valid order, construct validity test was performed using a factor analysis, namely the Confirmatory Factor Analysis (CFA).Determinant of correlation test matrix showed a value of 0.001 which showed that the items in the instrument were interrelated.
The subjects in this study fulfilled the condition for factor analysis, seen from their Kaiser-Meyer-Olkin (KMO) value of 0.554 (> 0.5) and it showed that the number of samples is adequate.Bartlett's test of sphericity showed a level of significance of 0.001 (Chi square = 1163.892;df = 666; p < 0.05), this meant that the data could be predicted and further analysis could be performed.Measure of sampling adequacy (MSA) was showed with the value of antiimage correlation above 0.500.The analysis results showed that MSA value of all items were above 0.500 (the range was between 0.501 -0.812).
According to communalities test, all items showed extraction value above 50% (0.500) (the range of extraction value was 0.582 -0.869).It was concluded that all items were able to explain the factors statistically.Total variance explained test obtained 9 factors with eigenvalue ≥ 1.These 9 factors were able to explain 76.29% of all the instrument construct.Rotated factor analysis showed that the instrument was divided into 9 factors, where a majority of the components were covered in the first factor.The already developed instrument/inventory was divided into 5 analysis factors.The instrument currently being developed had already been divided into 9 factors statistically, so it was very possible to unite the arranged components and reduce them into 5 factors in accordance with the original instrument arrangement.Reliability test results showed that the items in this questionnaire were reliable with a Cronbach's alpha value of 0.943 (> 0.600).The alpha test being developed by Cronbach was the test used to test the internal consistency among items statistically, so it was an estimation.The alpha value was always between 0 -1.Alpha test gives an estimation of several items' internal consistency when applied in a certain population in a certain time and for a certain purpose.If the tested items were closely related, most likely high alpha score will be obtained. 7,8e instrument being developed, after passing through validity and reliability test, had 37 question items with Likert scale (0-4).This instrument was divided into 5 appraisal sub competency domains, namely 1) affection, reflection, and appreciation of differences; 2) ethics; 3) management of fatigue and work balance; 4) professional behavior and participation in community; and 5) concern for patients.Each item's score was added and then converted to T-score.Average cut off point was used (T-score = 50) as the basis to divide the subjects into 2 groups, i.e. subjects with less competency and with adequate competency.The characteristics of subjects' professionalism were presented in Table 3. Chi square test was performed to assess whether there was a difference of professionalism competency and sub competency according to the subjects' characteristics.The analysis results showed that there was not a relation between total score or each sub competency domain score and subjects' characteristics, aside from concern for patients and residents' rotation domains.One-way ANOVA showed that the scores of post-rotation subjects were higher compared to subjects in wards and polyclinic, except for domain 4 (professional behavior and participation in community).
From the mean scores, it was shown that subjects' scores were generally high when in ward rotation, then lower when in polyclinic rotation, and much higher when in post-rotation.This needs further review for its relation with resident supervision during the education process and where the supervision in the ward is strict because the teaching staff are always in the ward.It is different from the polyclinic rotation where the supervision is less strict.Post-rotation is the rotation where the subjects had been through other rotations, so their behavior is more careful.Muchlas reminded the importance of control function in the scope of an organization, in this case it was the organization of medical education called the committee. 9ACGME as the initiator of the concept of residents' competency assessment in psychiatry milestones indeed requires the establishment of a clinical competency committee.
ACGME's version of clinical competency committee assessed residents' progress every 6 months periodically in 6 competency domains.Assessment may be preceded by a self-assessment by the residents themselves.The development of an inventory such as the one in this study helps the implementation of an initial self-assessment.

CONCLUSIONS
Residents' professionalism assessment instrument being developed is valid and reliable to be used as one of residents' professionalism assessment instruments in the Department of Psychiatry, Faculty of Medicine, Universitas Gadjah Mada.

SUGGESTIONS
Further study is needed in other centers of psychiatry education to obtain more generalizable results.

Table 1 .
The analysis results of Pearson's product moment of 40 questionnaire items (n = 42)

Table 2 .
The results of rotated factor analysis test showed 9 factors

Table 3 .
Professionalism in psychiatry residents of Faculty of Medicine, Universitas Gadjah Mada according to the instrument being developed.