Optimizing the Accuracy of the Semantic-Based Compound Emotion Classifications using the XLM-RoBERTa
Abstract
There are six basic emotions; they are anger, sadness, happiness, disgust, surprise, and fear. A combination of basic emotions creates a new type of emotion called a compound emotion. The examples of applying these compound emotions are in chatbots, translations, and text summarization. Several research on classifying these emotions based on Indonesian texts have used traditional models such as multinomial naïve Bayes, support vector machine (SVM), k-nearest neighborhood, and term frequency–inverse document frequency (TF-IDF). The previous research have a massive drawback, primarily on their less optimized performances. The models used could only classify things with the available data; thus, the text processing is required that results in a longer training time for larger This research aims to solve the issue from the previous research by using cross-lingual language model-robustly optimized bidirectional encoder representations from transformers approach (XLM-RoBERTa) model to classify compound emotions based on the semantics or meaning in words and sentences. The XLM-RoBERTa is a transformer model that can identify the meaning of a word from its attention mechanism and represent it as a vector to know the usage and position in a sentence. It is also a method to understand the meaning of a specific word. Using the attention mechanism, the model used the word position to recognize the sentence pattern and classify them even further to know the pattern and sequence to understand the semantics. The experiment result showed that the model could classify Indonesian texts into basic and compound emotion classes with an accuracy of up to 95.56%. This result is much higher than using traditional models to classify the compound emotion classes.
References
I.F. Putra and A. Purwarianti, “Improving Indonesian Text Classification Using Multilingual Language Model,” 2020 7th Int. Conf. Adv. Inform.: Concepts, Theory, Appl. (ICAICTA) 2020, pp. 1–5, doi: 10.1109/ICAICTA49861.2020.9429038.
S. Du, Y. Tao, and A.M. Martinez, “Compound Facial Expressions of Emotion,” PNAS, Vol. 111, No. 15, pp. E1454-E1462, Mar. 2014, doi: 10.1073/pnas.1322355111.
P. Ekman, “An Argument for Basic Emotions,” Cogn., Emot., Vol. 6, No. 3–4, pp. 169-200, Jan. 2008, doi: 10.1080/02699939208411068.
V. Dogra et al., “A Complete Process of Text Classification System Using State-of-the-Art NLP Models,” Comput. Intell., Neurosci., Vol. 2022, pp. 1–26, Jun. 2022, doi: 10.1155/2022/1883698.
T.H. Saputro and A. Hermawan, “The Accuracy Improvement of Text Mining Classification on Hospital Review Through the Alteration in the Preprocessing Stage,” Int. J. Comput., Inf. Technol. (IJCIT), Vol. 10, No. 4, pp. 140–146, Jul. 2021, doi: 10.24203/ijcit.v10i4.138.
W.-H. Khong, L.-K. Soon, and H.-N. Goh, “A Comparative Study of Statistical and Natural Language Processing Techniques for Sentiment Analysis,” J. Teknol., Vol. 77, No. 18, pp. 155–161, Nov. 2015, doi: 10.11113/jt.v77.6502.
Aripin, H. Haryanto, and W. Agastya, “Synthesis of Compound Facial Expressions Based on Indonesian Sentences Using Multinomial Naïve Bayes Model and Dominance Threshold Equations,” Eng. Lett., Vol. 30, No. 1, pp. 1–10, Mar. 2022.
A. Conneau et l., “Unsupervised Cross-lingual Representation Learning at Scale,” Proc. 58th Annu. Meeting Assoc. Comput. Linguist., 2020, pp. 8440–8451, doi: 10.18653/v1/2020.acl-main.747.
H. Gonen, S. Ravfogel, Y. Elazar, and Y. Goldberg, “It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT,” Proc. Third BlackboxNLP Workshop Anal., Interpreting Neural Netw. NLP, 2020, pp. 45–56, doi: 10.18653/v1/2020.blackboxnlp-1.5.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proc. 2019 Conf. North Amer. Chapter Assoc. Comput. Linguist.: Human Lang. Technol., 2019, pp. 4171–4186, doi: 10.18653/v1/n19-1423.
K. Taneja and J. Vashishtha, “Comparison of Transfer Learning and Traditional Machine Learning Approach for Text Classification,” 2022 9th Int. Conf. Comput. Sustain. Glob. Development (INDIACom), 2022, pp. 195–200, doi: 10.23919/INDIACom54597.2022.9763279.
Aripin, W. Agastya, and H. Haryanto, “Ekstraksi Emosi Majemuk Kalimat Bahasa Indonesia Menggunakan Convolution Neural Network,” J. Nas. Tek. Elekt., Teknol. Inf. (JNTETI), Vol. 10, No. 2, pp. 148–155, Mei 2021, doi:10.22146/jnteti.v10i2.1051.
W. Agastya and Aripin, “Pemetaan Emosi Dominan pada Kalimat Majemuk Bahasa Indonesia Menggunakan Multinomial Naïve Bayes,” J. Nas. Tek. Elekt., Teknol. Inf. (JNTETI), Vol. 9, No. 2, pp. 171–179, Mei 2020, doi: 10.22146/jnteti.v9i2.157.
S. Du and A.M. Martinez, “Compound Facial Expressions of Emotion: From Basic Research to Clinical Applications,” Dialogues Clin. Neurosci., Vol. 17, No. 4, pp. 443–455, 2015, doi: 10.31887/DCNS.2015.17.4/sdu.
A. Wibowo and E. Winarko, “Paper Review: Data Mining Twitter,” Konf. Nas. Sist., Inform. (KNS&I), 2014, pp. 1–6.
N. Azam, Jahiruddin, M. Abulaish, and N.A.H. Haldar, “Twitter Data Mining for Events Classification and Analysis,” 2015 Second Int. Conf. Soft Comput., Mach. Intell. (ISCMI), 2015, pp. 79–83 doi: 10.1109/ISCMI.2015.33.
R. Batool, A. Khattak, J. Hashmi, and S. Lee, “Precise Tweet Classification and Sentiment Analysis,” 2013 IEEE/ACIS 12th Int. Conf. Comput., Inf. Sci. (ICIS), 2013, pp. 461–466. doi: 10.1109/ICIS.2013.6607883.
S. Vosoughi, H. Zhou, and D. Roy, “Enhanced Twitter Sentiment Classification Using Contextual Information,” Proc. 6th Workshop Comput. Approaches Subjectivity Sentiment, Social Media Anal., 2016, pp. 16–24, doi: 10.18653/v1/W15-2904.
M.D. Samad, N.D. Khounviengxay, and M.A. Witherow, “Effect of Text Processing Steps on Twitter Sentiment Classification using Word Embedding,” 2020, arXiv:2007.13027.
A.I. Kadhim, “An Evaluation of Preprocessing Techniques for Text Classification,” Int. J. Comput. Sci., Inf. Secur., Vol. 16, No. 6, pp. 22–32, Jun. 2018.
J.J. Webster and C. Kit, “Tokenization as the Initial Phase in NLP,” Proc. 14th Conf. Comput. Linguist., 1992, Vol. 4, pp. 1106–1110, doi:10.3115/992424.992434.
N. Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” J. Mach. Learn. Res., Vol. 15, No. 56, pp. 1929–1958, Jun. 2014.
A. Labach, H. Salehinejad, and S. Valaee, “Survey of Dropout Methods for Deep Neural Networks,” 2019, arXiv:1904.13310, doi: 10.48550/arXiv.1904.13310.
S. Cai at al., “Effective and Efficient Dropout for Deep Convolutional Neural Networks,” 2020, arXiv:1904.03392, doi: 10.48550/arXiv.1904.03392.
S.R. Dubey, S.K. Singh, and B.B. Chaudhuri, “Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark,” 2022, arXiv:2109.14545, doi: 10.48550/arXiv.2109.14545.
C. Nwankpa, W. Ijomah, A. Gachagan, and S. Marshall, “Activation Functions: Comparison of Trends in Practice and Research for Deep Learning,” 2018, arXiv:1811.03378, doi: 10.48550/arXiv.1811.03378.
A.U. Ruby, P. Theerthagiri, I.J. Jacob, and Y. Vamsidhar, “Binary Cross Entropy with Deep Learning Technique for Image Classification,” Int. J. Adv. Trends Comput. Sci., Eng., Vol. 9, No. 4, pp. 5393–5397, Aug. 2020, doi: 10.30534/ijatcse/2020/ 175942020.
© Jurnal Nasional Teknik Elektro dan Teknologi Informasi, under the terms of the Creative Commons Attribution-ShareAlike 4.0 International License.
1.png)
