Sentiment analysis of user texts based on the tuning of the training parameters of a distilled model of the BERT family
Keywords:
sentiment analysis, text sentiment classification, distillation, learning model, data preprocessing, data normalization, BERT, ruBert, Python
Abstract
The increasing complexity of neural network architectures and the increasing volume of processed data in machine learning raises the question of the need to apply more productive approaches that would optimize the development of text classification models to solve the tasks of sentiment analysis. The aim of this work is to train and optimize a approaches for data classification as part of the solution of sentiment analysis of the Russian-speaking text. This research proposes the application of pre-trained BERT bidirectional coding models as well as the ruBERT-tiny knowledge distillation model to perform multiclass text classification for sentiment analysis of user text. The application of the data compaction step for knowledge distillation models allows to optimize the training phase of the text classification models. A program is developed in the Python using machine learning libraries. The technical solution allows to test the pre-trained models of data classification, on the basis of which to create optimized models of classification for the analysis of the sentiment of the user texts, taking into account the specifics of the subject area.
Downloads
Download data is not yet available.
References
1. Koloshina V. S. and Rodionova, M. E. (2021) Analiz vliyaniya social’nyh setej v politicheskoj sfere obshchestva: ot mirovoj k rossijskoj praktike (na analize social’nyh setej rossijskih politicheskih partij) [Analysis of the impact of social networks in the political sphere of society: from world to russian practice (on the analysis of social networks of russian political parties)]. Science and Education Today: Basic and Applied Research. P. 412–420. (in Russian)
2. Cherkasov D. and Ivanov V. V. (2018) Mashinnoe obuchenie [Machine learning. Science]. Technology and Education. No. 5(46). P. 85–87. (in Russian)
3. Malchits V. S. (2019) Primenenie metodov mashinnogo obucheniya dlya klassifikacii novostej [Application of machine learning methods for news classification] Youth in the 21st century: a step into the future. P. 208–209. (in Russian)
4. Widmant O. S. (2018) Prognozirovanie finansovyh vremennyh ryadov s ispol’zovaniem rekurrentnyh nejronnyh setej LSTM [Financial time series forecasting using LSTM recurrent neural networks]. Society: politics, economics, law. (5). P. 63-66. (in Russian).
5. Galeev D. T. and Panischev V. S. (2022) Eksperimental’noe issledovanie yazykovyh modelej” transformer” v zadache nahozhdeniya otveta na vopros v russkoyazychnom tekste. [Experimental study of linguistic “transformer” models in the task of finding an answer to a question in Russian-language text]. Informatics and Automation. 21(3). P. 521–542. (in Russian)
6. Kosterin, M. A. and Paramonov I. V. (2022) Nejrosetevaya klassifikaciya russkoyazychnyh predlozhenij po tonal’nosti na chetyre klassa [Neural network classification of Russian sentences by tonality into four classes]. Modeling and analysis of information systems. 29(2). P. 116–133. (in Russian)
7. Yarushkina N. G., Moshkin V. S. and Konstantinov A. A. (2020) Primenenie yazykovyh modelej word2vec i bert v zadache sentiment-analiza tekstovyh soobshchenij social’nyh setej [Application of word2vec and bert language models in the task of sentiment analysis of social network text messages]. Automation of management processes. (3). P. 60–69. (in Russia)
8. Berezin S. A. (2020) Rekomendatel’naya sistema dlya meropriyatij na osnove yazykovoj modeli BERT [A recommendation system for events based on the BERT language model]. INSC 2020. P. 154–154. (in Russian)
9. Dale D. S. (2021) Malen’kiy i bystryy BERT dlya russkogo yazyka [Small and fast BERT for the Russian language] – Text: electronic. URL
10. Blinov P., Avetisian, M., Kokh V., Umerenkov D. and Tuzhilin A. (2020) Predicting clinical diagnosis from patient’s electronic health records using BERT-based neural networks. International Conference on Artificial Intelligence in Medicine. P. 111–121.
11. Bessmertny I. A., Vasiliev A. V., Koroleva Y. A., Platonov A. V. and Poleshchuk E. A. (2019). Metody kvantovogo formalizma v informacionnom poiske i obrabotke tekstov na estestvennyh yazykah [Methods of quantum formalism in information retrieval and text processing in natural languages]. Proceedings of higher education institutions. Instrumentation. 62(8). P. 702–709. (in Russian)
12. Karpovich S. N. (2018) Korpus tekstov russkogo yazyka dlya testirovaniya algoritmov tematicheskogo modelirovaniya [A corpus of Russian texts for testing thematic modeling algorithms]. Intelligent Technologies in Transport. 1(13). P. 11–19. (in Russia)
13. Cui, X., Scogland T. R., de Supinski B. R. and Feng W. C. Performance Evaluation of the NVIDIA Tesla P100: Our Directive-Based Partitioning and Pipelining vs. NVIDIA’s Unified Memory. Matrix. 40. P. 50.
14. RuBERT for Sentiment Analysis (2021) – Text: electronic. URL
15. Kropanev N. D. and Kotelnikova A. В. (2021) BERT dlya analiza tonal’nosti dlinnyh tekstov na primere Kaggle Russian News Dataset [BERT for analyzing the tonality of long texts on the example of Kaggle Russian News Dataset]. Society. Science. Innovations (NPC-2021). P. 256–259.
16. Shazeer N. and Stern M. (2018) Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. P. 4596–4604.
17. Kosykh N. E. (2020) Ocenka giperparametrov pri analize tonal’nosti russkoyazychnogo korpusa tekstov [Hyperparameter estimation in tone analysis of Russian-language text corpus]. Intelligent Technologies in Transport. 3 (23). P. 41– 44. (in Russain)
18. Bolshakov M. A., Molodkin I. A. and Pugachev S. В. (2020) Sravnitel’nyj analiz metodov mashinnogo obucheniya dlya ocenki kachestva IT-uslug [Comparative analysis of machine learning methods for IT service quality assessment]. Information protection. Insider. (4). P. 36–43.
2. Cherkasov D. and Ivanov V. V. (2018) Mashinnoe obuchenie [Machine learning. Science]. Technology and Education. No. 5(46). P. 85–87. (in Russian)
3. Malchits V. S. (2019) Primenenie metodov mashinnogo obucheniya dlya klassifikacii novostej [Application of machine learning methods for news classification] Youth in the 21st century: a step into the future. P. 208–209. (in Russian)
4. Widmant O. S. (2018) Prognozirovanie finansovyh vremennyh ryadov s ispol’zovaniem rekurrentnyh nejronnyh setej LSTM [Financial time series forecasting using LSTM recurrent neural networks]. Society: politics, economics, law. (5). P. 63-66. (in Russian).
5. Galeev D. T. and Panischev V. S. (2022) Eksperimental’noe issledovanie yazykovyh modelej” transformer” v zadache nahozhdeniya otveta na vopros v russkoyazychnom tekste. [Experimental study of linguistic “transformer” models in the task of finding an answer to a question in Russian-language text]. Informatics and Automation. 21(3). P. 521–542. (in Russian)
6. Kosterin, M. A. and Paramonov I. V. (2022) Nejrosetevaya klassifikaciya russkoyazychnyh predlozhenij po tonal’nosti na chetyre klassa [Neural network classification of Russian sentences by tonality into four classes]. Modeling and analysis of information systems. 29(2). P. 116–133. (in Russian)
7. Yarushkina N. G., Moshkin V. S. and Konstantinov A. A. (2020) Primenenie yazykovyh modelej word2vec i bert v zadache sentiment-analiza tekstovyh soobshchenij social’nyh setej [Application of word2vec and bert language models in the task of sentiment analysis of social network text messages]. Automation of management processes. (3). P. 60–69. (in Russia)
8. Berezin S. A. (2020) Rekomendatel’naya sistema dlya meropriyatij na osnove yazykovoj modeli BERT [A recommendation system for events based on the BERT language model]. INSC 2020. P. 154–154. (in Russian)
9. Dale D. S. (2021) Malen’kiy i bystryy BERT dlya russkogo yazyka [Small and fast BERT for the Russian language] – Text: electronic. URL
10. Blinov P., Avetisian, M., Kokh V., Umerenkov D. and Tuzhilin A. (2020) Predicting clinical diagnosis from patient’s electronic health records using BERT-based neural networks. International Conference on Artificial Intelligence in Medicine. P. 111–121.
11. Bessmertny I. A., Vasiliev A. V., Koroleva Y. A., Platonov A. V. and Poleshchuk E. A. (2019). Metody kvantovogo formalizma v informacionnom poiske i obrabotke tekstov na estestvennyh yazykah [Methods of quantum formalism in information retrieval and text processing in natural languages]. Proceedings of higher education institutions. Instrumentation. 62(8). P. 702–709. (in Russian)
12. Karpovich S. N. (2018) Korpus tekstov russkogo yazyka dlya testirovaniya algoritmov tematicheskogo modelirovaniya [A corpus of Russian texts for testing thematic modeling algorithms]. Intelligent Technologies in Transport. 1(13). P. 11–19. (in Russia)
13. Cui, X., Scogland T. R., de Supinski B. R. and Feng W. C. Performance Evaluation of the NVIDIA Tesla P100: Our Directive-Based Partitioning and Pipelining vs. NVIDIA’s Unified Memory. Matrix. 40. P. 50.
14. RuBERT for Sentiment Analysis (2021) – Text: electronic. URL
15. Kropanev N. D. and Kotelnikova A. В. (2021) BERT dlya analiza tonal’nosti dlinnyh tekstov na primere Kaggle Russian News Dataset [BERT for analyzing the tonality of long texts on the example of Kaggle Russian News Dataset]. Society. Science. Innovations (NPC-2021). P. 256–259.
16. Shazeer N. and Stern M. (2018) Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. P. 4596–4604.
17. Kosykh N. E. (2020) Ocenka giperparametrov pri analize tonal’nosti russkoyazychnogo korpusa tekstov [Hyperparameter estimation in tone analysis of Russian-language text corpus]. Intelligent Technologies in Transport. 3 (23). P. 41– 44. (in Russain)
18. Bolshakov M. A., Molodkin I. A. and Pugachev S. В. (2020) Sravnitel’nyj analiz metodov mashinnogo obucheniya dlya ocenki kachestva IT-uslug [Comparative analysis of machine learning methods for IT service quality assessment]. Information protection. Insider. (4). P. 36–43.
Published
2022-11-09
How to Cite
Kosykh, N. E. (2022). Sentiment analysis of user texts based on the tuning of the training parameters of a distilled model of the BERT family. Proceedings of Voronezh State University. Series: Systems Analysis and Information Technologies, (3), 139-151. https://doi.org/10.17308/sait/1995-5499/2022/3/139-151
Section
Computer Linguistics and Natural Language Processing
Условия передачи авторских прав in English
