Interrater Reliability Checklist Osce Kateterisasi Urin Di Program Studi Ilmu Keperawatan Fakultas Kedokteran Universitas Gadjah Mada



Hershinta Retno(1*), Intansari Nurjannah(2)

(1) Nursing Universitas Gadjah Mada
(2) Departemen Keperawatan Jiwa Program Studi Ilmu Keperawatan Fakultas Kedokteran, Kesehatan Masyarakat dan Keperawatan Universitas Gadjah Mada
(*) Corresponding Author

Abstract


Latar Belakang: Objective Structured Clinical Examination (OSCE) merupakan salah satu metode penilaian sumatif dalam penilaian berbasis performa. Salah satu komponen yang menyusun OSCE adalah instrumen penilaian1. Instrumen checklist merupakan komponen OSCE yang mempengaruhi reliabilitas penilaian tersebut2. Selama diterapkan di PSIK FK UGM, reliabilitas checklist kateterisasi urin belum pernah diuji.

Tujuan Penelitian: Penelitian ini bertujuan untuk mengetahui interrater reliability checklist OSCE kateterisasi urin di Program Studi Ilmu Keperawatan Fakultas Kedokteran Universitas Gadjah Mada.

Metode: Penelitian ini merupakan penelitian psikometri. Dua rater yang terdiri dari mahasiswa tingkat  4 dan seorang dosen menilai performa 93 mahasiswa tahun kedua dalam stase kateterisasi urin saat OSCE. Hasil pengukuran akan dihitung dan diuji menggunakan uji kappa dan Percent Agreement (PA). Sedangkan reliabilitas tiap item kateterisasi urin akan dihitung dengan menggunakan weighted kappa, dan beberapa item yang mengalami paradoks akan dihitung menggunakan Prevalence And Bias Adjusted Kappa-Ordinal Scale (PABAK-OS) untuk menghilangkan efek bias dan prevalensi.

 Hasil: Hasil dari penghitungan nilai kappa menunjukkan bahwa checklist kateterisasi urin memiliki nilai kappa sebesar 0,57, dan PA sebesar 78,49%. Sedangkan pengukuran item menunjukkan hasil yang bervariasi. Nilai kappa item berada pada kisaran 0,24-0,96. Adapun faktor yang mempengaruhi penilaian OSCE dapat dilihat dari sudut pandang item maupun rater.

Kesimpulan: Checklist kateterisasi urin merupakan checklist dengan kategori reliabilitas sedang dan merupakan ceklis yang reliabel. Namun, terdapat 9 item pada checklist tersebut yang memerlukan perbaikan karena reliabilitasnya tidak dapat diterima.

 

 

Kata kunci: OSCE, checklist, interrater  reliability

 

Background: Objective Structured Clinical Examination (OSCE) is one of summative test method for performance-based assessment. One of component that make up an OSCE is assessment instrumen. Whereas checklist is one of OSCE’s component that affect OSCE’s reliability. As long as this checklist was implemented in Nursing Science Program, Faculty of Medicine, UGM, the reliability of urinary catheterization checklist hasn’t been tested

Objective: This study aims to assess interrater reliability of OSCE checklist instruments for urinary catheterization in Nursing Science Program, Faculty of Medicine, Universitas Gadjah Mada.

Methods: This study is a psychometric testing study. Two rater consisted of a fourth-year student and a lecturer who performed measurement on 93 second-year students who was taking the OSCE examination. The measurement result were analyzed using kappa test and percent agreement (PA). Whereas the item’s reliability were analyzed using weighted kappa dan some items which is paradox can be count with Prevalence and Bias Adjusted Kappa-Ordinal Scale (PABAK-OS) to separate the bias and prevalence effect.

Results: The results of measurement of the reliability was 0,57, which indicated that the checklist was in the moderate category, and the PA was 78,49%. According to Osborne (2008) and Stemler and Tsai (2008), this checklist reliability considered as acceptable. Meanwhile, the result of measurement of each item indicated various reliabilities. Reliability value on this checklist’s item was around 0,24-0,96. Meanwhile, some factors that affect OSCE’s rating categorized as item and rater..

Conclusion: The checklist of urinary catheterization has moderate reliability value and can be used as an instrument for the OSCE assessment. However, there were 9 items that weren’t reliable and must be improved.

 

 

Keywords: OSCE, checklist, interrater  reliability

 


Full Text:

PDF


References

McWilliam, P., Botwinski, C. Developing a Successfull Nursing Objective Structured Clinical Examination. Journal of Nursing Education. 2010; 49(1): 36-41 2. Brannick, M.T., Erol-Korkmaz, H.T., Prewett, M. 2011. A systematic review of the reliability objective structured clinical examination scores. Medical Education 2011 (45):1181-1189 3. Harden, R. McG., Stevenson, M., Downie, W.W., Wilson, G.M. Assessment of Clinical Competence using Objective Structured Examination. British Medical Journal. 1975 (1): 447-451 4. Newble, D. Techniques for Measuring Clinical Competence: Objective Structured Clinical Examination. Medical Education 2004. (38):199-203 5. Medical Council of Canada. Guidelines for the Development of Objective Structured Clinical Examination (OSCE) Cases. 2013:1-13 6. Khan, K.Z., Ramachandran, S., Gaunt, K., Pushkar, P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: An historical and theoretical perspective. Medical Teacher. 2013 (35): 1437-1446 7. McWilliam, P., Botwinski, C. Developing a Successfull Nursing Objective Structured Clinical Examination. Journal of Nursing Education. 2010; 49(1): 36-41 8. Silva, C.C.B.M., Lunardi, A.C., Mendes, F.A.R., Souza, F.F.P., Carvalho, C.R.F. Objective Structured Clinical Evaluation as an Assessment Method for Undergraduate Chest Physical Therapy Students: A Cross-Sectional Study. Rev Bras Fisioter. 2011; 15 (6): 481-486 9. Erfanian, F. dan Khadivzadeh, T. Evaluation of Midifery Students’ Competency in Providing Intrauterine Services Using Objective Structured Clinical Examination. Journal of Nurse Midwifery Research 2011; 16 (3): 191-196 10. Smith, S.A. Nurse Competence: A Concept Analysis. International Journal of Nursing Knowledge. 2011; 23(3):242-247 11. Sakurai, H., Sugiura, Y., Tomita, M., Tanabe, S. Standardization of Clinical Skill Evaluation in Physical/Occupational Thereapist Education –Effect of Introduction of an Education System Using OSCE-. Journal of Physical Therapy Science. 2013 (25): 1071-1077 12. Beckham, N.D. Objective Structured Clinical Evaluation Efectiveness in Clinical Evaluation for Family Nurse Practitioner Students. Clinical Simulation in Nursing 2013 (9):453-459 13. Mitchell, M.L., Henderson, A., Groves, M., Dalton, M. Nuity, D. The Objective Structured Clinical Examination (OSCE): Optimising its value in the undergraduate nursing curriculum. Nursing Education Today. 2009 (29): 398-404 14. Moattari, M., Zargar, A.S., Mousavinasab, M., Zare, N., Marvdast, B. Reliability and Validity of OSCE in Evaluating Clinical Skills of Nursing Students. Journal of Medical Education 2009; 13 (3): 79-85 15. Mukwato P.K., Mwape, L., Makukula, M.K., Mweemba, P., Maimbolwa, M.C. Implementation of Objective Structured Clinical Examination for Assessing Nursing Students’ Clinical Competencies: Lessons and Implications. Creative Education 2013; 4(10A): 48-53 16. Rushfort, H.E. Objective Structured Clinical Examination (OSCE): Review of Literature and Implications for Nursing Education. Nurse Education Today. 2007 (27):481-490 17. Selim, A.A., Ramadan, F.H., El-Gueneidy, M.M., Gaafer, M.M. 2012. Using Objective Structured Clinical Examination (OSCE) in Undergraduate Psychiatric Nursing Education: Is it Reliable and Valid?. Nurse Education Today 2012 (30):283-288 18. Tudiver, F., Rose, D., Banks, B., Pfortmiller, D. Reliability and Validity Testing of an Evidence-based Medicine OSCE Station. Family Medicine. 2009; 41(2):89-91 19. Miller, M.D., Linn, R.L., Gronlund, N.E. 2009. Measurement and Assessment in Teaching. United States of America. Pearson 20. Feinstein, A.R. dan Ciccheti, D.V. 1990. High Agreement but low kappa: I. The Problems of Two Paradox. 21. Warrens, M.J. 2013. Weighted Kappa for 3x3 Tables. Journal of Probability and Statistics 2013 (13):1-10 22. Graham, M., Milanowski, A., Miller, J., Westat. Measuring and Promoting Inter-rater Agreement of Teacher and Principal Performance Rating. Wisconsin: Center for Educator Compensation Reform 2012: 1-33 23. Osborne, J. Best Practice in Quantitative Methods. California: SAGE Publishing. 2008: 32-48 24. Stemler, S.E. dan Tsai, J. 3 Best Practice in Interrater Reliability: Three Common Approaches. Arizona: SAGE Publication. 2008: 1-45 25. McCray, G. Assessing inter-rater agreement for nominal judgement variables. Paper presented at the Language Testing Forum. Nottingham, November 15-17. 2013: 1-23 26. Streiner, D.L., Norman, G.R., dan Cairney, J. Health Measurement Scales: A practical guide to their development and use 5th ed.Oxford: Oxford University Press 2015:177 27. Sim, J. dan Wright, C.C. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requiremets. Phys Ther 2005 (85):257-268 28. Kottner, J. dan Dassen, T. An interrater Reliability Study of the Braden Scale in Two Nursing Homes. International Journal of Nursing Studies 2008 (45): 1501-1511 29. Besar, M.N.A., Siraj, H.H., Manap, R.A., Mahdy, Z.A., Yaman, M.N., Kamarudin, M.A., Mohamad, N. 2012. Should a single clinician examiner be used in objective structured clinical examination?. Procedia-Social and Behavioral Science 2012 (60):443-449 30. Moineau, G., Power, B. Pion, A.M.J., Wood, T.J., Murto, S.H. 2010. Comparison of Student Examiner to Faculty Examiner Scoring and Feedback in an OSCE. Medical Education 2010 (45):183-191




Article Metrics

Abstract views : 27 | views : 11

Refbacks

  • There are currently no refbacks.