Sentiment Analysis Review Threads Google Play Store with RoBERTa Model

Natan Kharisma A; Dewi Lestari; Gatot T Pranoto

doi:10.22146/jnteti.v14i4.22038

Natan Kharisma A Informatics Engineering Study Program, Faculty of Engineering, Trilogi University, Jakarta 12760, Indonesia
Dewi Lestari Informatics Engineering Study Program, Faculty of Engineering, Trilogi University, Jakarta 12760, Indonesia
Gatot T Pranoto Informatics Engineering Study Program, Faculty of Engineering, Trilogi University, Jakarta 12760, Indonesia

DOI: https://doi.org/10.22146/jnteti.v14i4.22038

Keywords: Sentiment Analysis, Threads App, Google Play Store, BERT Method, User Reviews, RoBERTa Model

Abstract

The rapid development of internet technology globally, including in Indonesia, has drastically changed communication and interaction patterns between individuals. One impact is seen in the increasing use of text-based social media applications, such as Threads, developed by Meta. Within a short time, Threads managed to attract millions of users. However, the large number of user reviews on the Google Play Store presents its own challenges, particularly in manual sentiment analysis, which is very time-consuming and prone to bias. This research aims to overcome these challenges by implementing a variant of bidirectional encoder representations from transformers (BERT), the robustly optimized BERT pretraining approach (RoBERTa) model, which has been optimized for natural language processing. The research process followed the cross-industry standard process for data mining (CRISP-DM) framework, including several main stages: understanding the business context, data exploration and model building preparation, performance evaluation, and model deployment. Data were obtained directly from the Google Play Store and then cleaned through deduplication, normalization, and tokenization stages. The RoBERTa model demonstrated strong performance, with an accuracy of 88%. Precision was recorded at 92% for positive sentiment and 81% for negative sentiment, while recall was at 88% and 87%, respectively. The F1 score was also high, at 90% for positive and 84% for negative sentiment. When compared to algorithms like naïve Bayes and support vector machine (SVM), RoBERTa proved superior. This research opens opportunities for exploring other transformer models or using ensembles to improve performance in the future.

References

K. Wisnubroto, “Government,s commitment to protecting children in the digital space,” indonesia.go.id. Accessed: Mar. 29, 2025. [Online]. Available: https://indonesia.go.id/kategori/editorial/9037/komitmen-pemerintah-melindungi-anak-di-ruang-digital?lang=1

W. Meliani and D. Gustian, “Public opinion sentiment analysis of the Threads application on Twitter using the Naïve Bayes method,” in Proc. Nat. Seminar Inf. Syst. Informatics Manage., Nusa Putra Univ., Jan. 2024, pp. 197–202. Accessed: Feb. 12, 2025. [Online]. Available: https://sismatik.nusaputra.ac.id/index.php/sismatik/article/view/260

M.N. Akbar and N. Samrin, “Sentiment analysis of user comments on the Threads application on Google Playstore using the multinominal Naive Bayes classifier algorithm,” Jagti, vol. 3, no. 2, pp. 21–29, Aug. 2023, doi: 10.24252/jagti.v3i2.67.

M.F. Hanif, S.H. Wijoyo, and W.H.N. Putra, “Sentiment classification of Threads application reviews based on the Naive Bytes algorithm and root cause analysis method,” J-PTIIK, vol. 8, no. 6, Jul. 2024. Accessed: Mar. 29, 2025. [Online]. Available: https://j-ptiik.ub.ac.id/index.php/j-ptiik/article/view/13786

M. Wankhade, A.C.S. Rao, and C. Kulkarni, “A survey on sentiment analysis methods, applications, and challenges,” Artif. Intell. Rev., vol. 55, no. 7, pp. 5731–5780, Oct. 2022, doi: 10.1007/s10462-022-10144-1.

D. Naik, H. Sultana, and K.K. Jitendra, “Insight into sentimental analysis,” J. Emerg. Technol. Innov. Res., vol. 7, no. 6, pp. 1561–1566, 2020. Accessed: Mar. 25, 2025. [Online]. Available: https://www.academia.edu/download/81006839/JETIR2006559.pdf

N. Nurzaman, N. Suarna, and W. Prihartono, “Sentiment analysis of Threads app reviews on Google Playstore using the Naïve Bayes algorithm,” Inf. Eng. Student J., vol. 8, no. 1, pp. 967–974, 2024, doi: 10.36040/jati.v8i1.8708.

F. Nufairi, N. Pratiwi, and F. Herlando, “Sentiment analysis of Threads application reviews on Google Play Store using support vector machine algorithm,” JIPI, vol. 9, no. 1, pp. 339–348, Feb. 2024, doi: 10.29100/jipi.v9i1.4929.

L. Pan, C.-W. Hang, A. Sil, and S. Potdar, “Improved text classification via contrastive adversarial training,” AAAI, vol. 36, no. 10, pp. 11130–11138, Jun. 2022, doi: 10.1609/aaai.v36i10.21362.

S. Kierszbaum, T. Klein, and L. Lapasset, “ASRS-CMFS vs. RoBERTa: Comparing two pre-trained language models to predict anomalies in aviation occurrence reports with a low volume of in-domain data available,” Aerospace, vol. 9, no. 10, p. 591, Oct. 2022, doi: 10.3390/aerospace9100591.

J. Dai, H. Yan, T. Sun, P. Liu, and X. Qiu, “Does syntax matter? A strong baseline for aspect-based sentiment analysis with RoBERTa,” arXiv preprint, arXiv:2104.04986, Apr. 2021, doi: 10.48550/arXiv.2104.04986.

C.A. Deagusti, “Sentiment Analysis of the Development Plan of Indonesia’s New Capital City (IKN) Based on Twitter (X) Using the Hybrid RoBERTa-GRU Method,” Ph.D. dissertation, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia 2024.

N.A.R. Putri and Ardiansyah, “Sentiment analysis of artificial intelligence progress in Indonesia using BERT and RoBERTa,” J. Sci. Informatics, vol. 9, no. 2, pp. 136–145, Nov. 2023, doi: 10.34128/jsi.v9i2.649.

T.L. Anggana, “Sentiment analysis using RoBERTa on user reviews of the National Health Insurance (JKN) mobile application,” Doctoral dissertation, UIN Sunan Gunung Djati Bandung, 2024. Accessed: Mar. 29, 2025. [Online]. Available: https://digilib.uinsgd.ac.id/99004/

R.M.R.W.P. Kusuma and W. Yustanti, “Customer review sentiment analysis of the Ruang Guru application using the BERT method,” J. Emerg. Inf. Syst. Bus. Intel., vol. 2, no. 3, Jul. 2021, doi: 10.26740/jeisbi.v2i3.41567.

G.S. Al-Husna, D. Asmarajati, I.A. Ihsannuddin, and R. Mahmudati, “Comparison of Naive Bayes and support vector machine methods for sentiment analysis on LinkedIn application user reviews,” Storage, vol. 3, no. 2, pp. 139–144, 2024, doi: 10.55123/storage.v3i2.3602.

A.K. Dewi, U.S. Semarang, J.L. T. Lomba, J. Mugas, and S. Semarang, “Sentiment analysis of Sicepat expeditions from Google Play reviews using the Naïve Bayes algorithm,” J. Inf. Eng. Inf. Syst., vol. 9, no. 2, pp. 796–805, Jun. 2022, doi: 10.35870/jtik.v8i2.1580.

D.F. Sjoraida, B. Wibawa, K. Guna, and D. Yudhakusuma, “Sentiment analysis of the film Dirty Vote using BERT,” JTIK, vol. 8, no. 2, pp. 393–404, Mar. 2024, doi: 10.35870/jtik.v8i2.1580.

S.S. Tandiapa and G.C. Rorimpandey, “Sentiment analysis of user reviews on Threads application using lexicon-based method and Naive Bayes classifier,” JCM, vol. 3, no. 1, pp. 339–352, Jan. 2024, doi: 10.36312/jcm.v3i1.

R. Ramadhan, “Sentiment analysis on Maxim app reviews on Google Play Store with K-nearest neighbor,” JURIKOM, vol. 10, no. 3, pp. 715–724, Jul. 2023. Accessed: Mar. 29, 2025. [Online]. Available: https://repository.uin-suska.ac.id/74467/

J.U.S. Lazuardi and A. Juarna, “Sentiment analysis of Joox application user reviews on Android using the BERT method,” Sci. J. Comput. Informatics, vol. 28, no. 3, pp. 251–260, 2023, doi: 10.35760/ik.2023.v28i3.10090.

C.-Z. Liu, Y.-X. Sheng, Z.-Q. Wei, and Y.-Q. Yang, “Research of text classification based on improved TF-IDF algorithm,” in Proc. IEEE Int. Conf. Intell. Robot. Control Eng. (IRCE), Aug. 2018, pp. 218–222, doi: 10.1109/IRCE.2018.8492945.

M.A. Java, M. Syafrullah, and F. Teknologi, “Sentiment analysis of user reviews of the Threads application on the Google Play Store using multinomial Naive Bayes and support vector machine,” TICOM J. Technol. Inf. Commun., vol. 12, no. 2, 2024, doi: 10.70309/ticom.v12i2.112.

N.V. Chawla, K.W. Bowyer, L.O. Hall, and W.P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953.

F.R. Adi Pratama and S.I. Oktora, “Synthetic minority over-sampling technique (SMOTE) for handling imbalanced data in poverty classification,” Stat. J. IAOS, vol. 39, no. 1, pp. 233–239, Feb. 2023, doi: 10.3233/SJI-220080.

V. Chandradev, I.M.A.D. Suarjaya, and I.P.A. Bayupati, “Hotel review sentiment analysis using the BERT deep learning method,” Buana Inform. J., vol. 14, no. 2, pp. 107–116, Oct. 2023, doi: 10.24002/jbi.v14i02.7244.

F.I. Septian, I.L. Kharisma, H. Hermanto, and K. Kamdan, “Implementation of the Bidirectional Encoder Representations from Transformers (BERT) method for sentiment analysis of Dana application user comments on Instagram,” in Proc. TAU SNARS-TEK Nat. Seminar Eng. Technol., vol. 3, no. 1, pp. 201–210, Jan. 2023, doi: 10.47970/snarstek.v2i1.571.