DEVELOPMENT OF A MOBILE ECG APPLICATION TO IMPROVE ECG INTERPRETATION SKILLS OF GENERAL PRACTITIONERS AND MEDICAL STUDENTS

Background: Electrocardiogram (ECG) has become a crucial examination in the management of cardiac emergencies. Accordingly, improvement of ECG interpretation skills is mandatory for general practitioners as the front-liners in emergency cases. The Mobile ECG application was developed as mobile learning media to facilitate continuing improvement of ECG interpretation skills. This study aimed to investigate the impact of the Mobile ECG application toward ECG interpretation skills of general practitioners and medical students and evaluate its usability. Methods: A pilot observational study was conducted in a 1-week timeframe using the Mobile ECG application. Subjects were recruited through consecutive sampling. They met the following criteria: 1) registered as general practitioners or medical students, 2) completed the basic ECG pre and post-tests, and 3) agreed to participate in the study. The Mobile ECG is a web-based application which consists of modules, quizzes, and gallery of ECG interpretations. Pre and post-test analysis and system usability scale (SUS) questionnaire were used to evaluate the impact and usability of the application. Results: A total of 252 subjects were recruited and 80.2% were general practitioners. There was a significant increase in post-test scores compared to pre-test (p=0.000) for all subjects. General practitioners significantly gained more score increment than medical students (1.08 vs 0.16, p=0.001). Based on the SUS score of 67.5, the application was marginally accepted by the users. Conclusion: To conclude, the implementation of the Mobile ECG application did improve basic ECG interpretation skills. According to the SUS score, this application still needs improvement.


INTRODUCTION
Electrocardiogram (ECG) is the recording of electrical potentials generated by cardiac impulses as they pass through the heart and spread to adjacent tissues, all the way to the surface of the body. ECG works by capturing those electrical potentials by using electrodes which are placed on the projection of the heart on the skin surface. 1 Around the globe, ECGs are recorded approximately 200 million times each year and it has become the most frequently performed cardiovascular test. 2 ECG has been used routinely and especially in emergency settings because of its advantages as a simple, non-invasive, and inexpensive examination. 3 ECG is now one of the most useful complementary medical examinations since it can detect and evaluate heart problems quickly through waveform recordings. 4 For example, because of both its high specificity and sensitivity, ECG has been very useful as an initial diagnostic tool for heart diseases. 3 Previous studies have found that ECG utility depends heavily on doctors' interpretations where inaccurate reading may have negative impact for patients. 5 Studies showed that medical students and doctors have limited performance in terms of accuracy of ECG interpretation. In one study, it was found that among US graduating medical students, only 37% of key and potentially life-threatening ECGs were correctly identified. 6 In another study, among general practitioners, accuracy of ECG interpretation and the subsequent clinical decision was found in 69% of cases. 7 Consequently, more trainings for ECG interpretation skills are necessary. Those trainings can be considered as part of Continuing Medical Education (CME) that can improve medical treatments. 5 Rapid development of mobile technology has affected the field of medical education. As a result, mobile learning in medicine has become widely available nowadays. A cohort observational study conducted by Chase et al. 8 showed that mobile learning has positive impact on the learning process during clinical rotation. Nonetheless, that impact relies on the availability of good Internet access.
The demand for educational software is growing exponentially with the increase of interest in the Internet, educational reform, and distance learning. 9 The use of electronic media as a learning method in medicine may be beneficial in assisting medical students and clinicians to practice skills, including ECG interpretation. Because ECG interpretation is considered to be difficult for clinicians who are not specialized in cardiology, the use of electronic media to improve the ECG interpretation skills has the potential to be developed in the field of cardiology education. 4 E-learning methods can facilitate learning without temporal and physical barriers by using a case-based approach and increasing the effectivity of self-paced learning. 10 With a good design, it can also promote active learning and simulations which in turn may lead to improvement in the quality and effectiveness of education. 11 In a systematic review by Pontes et al., 4 the use of an educational application of ECG was found to have a positive impact on learning effectivity. Viljoen et al. 12 discovered that blended learning using a web-based application is associated with the improvement of ECG competence and confidence compared to conventional lectures.
Since 2019, the Department of Cardiology and Vascular Medicine, Faculty of Medicine, Public Health and Nursing Universitas Gadjah Mada has been developing an educational application namely the Mobile ECG to meet the demand of borderless and life-long learning of electrocardiography. This application was built on a web-based platform. The advantages of the application include the accessibility at the learners' own suitable time and pace. This study aimed to conduct a trial of the educational application usage as well as investigate its impact on ECG interpretation skills among general practitioners and medical students.

METHODS
This observational study was conducted using a webbased application in combination with webinars. A two-phase webinar was performed to get the subjects familiarized with the application. Recruitment of the subjects was done by consecutive sampling method until a total of 334 subjects registered as participants of the webinars. The inclusion criteria were: registered as general practitioners or medical students and agreed to participate in the study. Subjects who failed to complete the pre-test, application trial, and post-test were excluded.
Subjects were assigned to do a pre-test before joining the first webinar. The pre-test consisted of 10 questions about basic ECG knowledge and interpretation. Introduction to distance learning, tips and tricks in ECG interpretation skill, and introduction to the Mobile ECG webbased application were delivered in the webinar. Simultaneously subjects were asked to learn ECG using the Mobile ECG for 1 week. After the oneweek trial of the application, a second webinar with post-test was performed to evaluate the progress of subjects' knowledge.
Mobile ECG is a web-based learning application about ECG to facilitate self-paced learning which comprises basic ECG modules, quizzes, and gallery of ECG interpretation. Topics covered in the modules are normal electrocardiogram, basic arrhythmia, cardiac chamber enlargement, ischemia and infarct, and electrolyte-and drug-induced ECG changes. The quizzes were arranged in two levels based on difficulty. Level 1 consisted of relatively easier questions. In each attempt, it was programmed to randomly select 10 questions from the question bank based on the difficulty. Users were given 20 minutes to complete each quiz. If the correct answers were less than 50%, users had two more attempts to make a higher score. Once users were able to answer at least 50% of the questions correctly, they were not permitted to take the same quiz again. Meanwhile, the gallery of ECG interpretation is a collection of ECG cases with complete step-by-step interpretation. Users may look through the gallery to learn ECG from real cases.
Subjects were assigned to read the modules and complete the quiz with minimum 50% correct answers. After a 1-week trial, they did the post-test which consisted of the same questions as the pretest, but in a different order. Subjects had to fill out an Indonesian System Usability Scale (SUS) Questionnaire which was adapted from Sharfina and Santoso. 13 The questionnaire consists of 10 items which rate the usability of Mobile ECG on a 5-point Likert scale from the subjects' perspectives (1 -strongly disagree to 5 -strongly agree). The odd numbers are for positive items and the even numbers for the others. Table 1 outlines the items included in the SUS questionnaire. 3 Saya merasa sistem ini mudah untuk digunakan. I thought the system was easy to use. 4 Saya membutuhkan bantuan dari orang lain atau teknisi dalam menggunakan sistem ini. I think that I would need the support of a technical person to be able to use this system.

5
Saya merasa fitur-fitur sistem ini berjalan dengan semestinya. I found the various functions in the system were well integrated.

6
Saya merasa ada banyak hal yang tidak konsisten (tidak serasi) pada sistem ini. I thought there was too much inconsistency in this system.

7
Saya merasa orang lain akan memahami cara menggunakan sistem ini dengan cepat. I would imagine that most people would learn to use this system very quickly. 8 Saya merasa sistem ini membingungkan. I found the system very cumbersome to use.

9
Saya merasa tidak ada hambatan dalam menggunakan sistem ini. I felt very confident using the system.

10
Saya perlu membiasakan diri terlebih dahulu sebelum menggunakan sistem ini. I needed to learn a lot of things before I could get going with this system.
In the second phase webinar, subjects were given the opportunity to ask about the ECG-related materials they had not understood. They could ask for the explanations of the difficult ECG interpretations. Their feedback in experiencing the trial was also elaborated.
The data analysis was performed using IBM® SPSS® Statistics version 25. Normality test was used to assess the data distribution. Demographic characteristics were analyzed descriptively. To evaluate the increase in scores, pre-and post-test data were analyzed using a test for mean difference of related samples. To assess the difference in scores gained between groups, an independent test was used.
Data from the SUS questionnaire were analyzed using a specific formula (Figure 3) to calculate the score. The score contribution for positive items were calculated by subtracting 1 from the scale position while for negative items it was subtracting the scale position from 5. Each score contribution was then multiplied by 2.5. The total score was achieved by summing all of item scores after multiplication, ranging from 0 to 100. The cut-off score for good usability was 68 or above. years old. Most of the subjects in this study were 25 years old or younger. The age group represented the graduating medical students and freshly-graduated doctors. A possible explanation to this finding was because the challenging transition between school and work experience motivates them to learn more to be prepared in clinical settings. As shown by a qualitative study which assessed the perspectives of junior doctors, the transition was considered as an abrupt, stressful, and exhausting experience but served as a period of a steep learning curve and a key opportunity to learn the role of physician. 16 The mean scores of pre-and post-tests were 4.19 and 5.09, respectively. None of the subjects were able to gain a perfect score on the pre-test. Otherwise, 6 subjects answered all post-test questions correctly. The number of subjects who scored zero in the posttest were fewer than in the pre-test (2 vs. 4).
The obtained data did not show normal distribution. Therefore, a non-parametric test was performed to analyze the difference between post-and pretest scores. There was a significant increase in post-test scores compared to pre-test scores (mean difference=0.90, p=0.000). General practitioners significantly gained higher scores than medical students (Table 3).  Table 2 explains the demographic characteristics of the subjects. A total of 252 subjects participated completely in this study with the ratio of female to male subjects 2.4:1. Given that general practitioners made up the majority (80.2%) of the subjects, there may be a role involving their seeking behavior for supplemental learning sources. The recruitment of the subjects was done by open publication and anyone from the background of medicine who wanted to participate was welcome to fill out the registration form. Because of that, our findings may indicate that general practitioners are more eager to seek for more sources to improve their skills in ECG interpretation and to engage in a CME activity. To date, there is no study comparing the willingness between both groups in learning skills from external sources and pursuing CME.  Opportunity to repeat and review the materials in mobile learning at students' own pace may contribute to the enhancement of the skills learned. 18 Independence that students experience in mobile learning allows them to learn in a more flexible and convenient way thus gain better retention. 19,20 Students can also choose specific materials they want to learn more or less based on their need. Individuals with other obligations aside from studying can benefit from the ease of mobile learning. 20 Viljoen et al. 21 stated that a mobile learning application in ECG enriched students' learning experience, particularly in improving the interpretation accuracy. The application aided the ECG interpretation by providing concise and practical modules along with interpretation algorithm and many ECG learning resources. However, the application's benefit is not sustained overtime and depends on the usage frequency.

RESULTS AND DISCUSSION
Overall, the SUS score given by the subjects was 67.5 (SD 15.16) with the lowest and highest scores of 15.0 and 100.0, respectively. According to Bangor et al., 22 that score can be interpreted as marginally acceptable. It implies that the Mobile ECG application had not been able to demonstrate good usability. Compared to another study which evaluated a similar type of mobile learning application for medical students with overall SUS score of 85.3, the results of this study prompted the need for further improvement in terms of application quality. Despite that, the application had good usability among medical students, as shown by the SUS score which was 70.2 (Table 4). It may be explained by the need of more advanced materials and cases among general physicians. Our study added to the global data of utilization of mobile learning in the scope of ECG interpretation. The results of this study did not merely show the impact of the application but also its user acceptability.
This study did not provide qualitative data to support quantitative data findings and there were no data regarding any additional course or training taken by subjects before using the application that might contribute to their ECG interpretation skills.

CONCLUSION
Utilization of web-based application significantly increased the post-test score of the subjects in basic ECG interpretation. The marginal score of SUS showed that the application still needs further improvement. The results of this study open the opportunity to enhance the Mobile ECG application's quality. The intervention used in this study might be a potential model to be applied in the medical education system.

RECOMMENDATION
It is paramount to further develop the Mobile ECG application to meet the quality expected by the candidates of users. Retesting the application in larger groups is recommended to obtain more generalizable data about its effectiveness and usefulness. The Mobile ECG application may then be implemented as a complementary learning media in medical education.