# Russian sarcastic comments detection

## About the project

Sarcasm recognition topic is popular nowadays. A lot of articles devoted
to the sarcasm recognition theme have been published since 2010.
Although there was actually no one of them to be about Russian sarcastic
comments recognition.

This project proposes a solution to the problem of Russian sarcastic
comments detection. A corpus of Russian-language comments was formed
with binary markup: whether the comment is sarcastic or not. The problem
was solved using three natural language processing approaches: TF-IDF
and Logistic Regression, Recurrent Neural Networks, Transformers. A
comparative analysis of the results of these approaches was carried out.

The link to the github repository is [here](https://github.com/123-39/NLP_Project).

## Researchers
The project was prepared by [Ivan Trenev](https://ods.ai/users/f3487c760b79) and [Nadezhda Chaplinskaia](https://ods.ai/users/affffadfbb1d).

## Dataset

To build the corpus of Russian sarcastic and non-sarcastic comments we
decided to use the text-based English news headlines corpus ([here](https://github.com/sankalpg10/NLP-Sarcasm-Detection/tree/main/data/) is a link to the website, where the text-based English news headlines corpus could be downloaded). We have
translated to Russian and then we have cleared all the samples. As the
result we have got 9554 sarcastic and 10479 non-sarcastic translated
samples.

But we had the idea that the corpus we want has to be more
"Russian", has to have some exclusively Russian sarcastic
statements that appears due to Russian mentality. That's why we decided
to use also another corpus &ndash; the corpus of Russian jokes ([here](https://www.kaggle.com/datasets/konstantinalbul/russian-jokes/) is a link to the website, where the corpus of Russian jokes could be downloaded). The corpus is huge: 130204 samples. These jokes we would like to consider as Russian sarcastic statements, because every Russian joke has Russian-sarcastic emotional background.

Finally, the corpus we made consisted of 5000 translated sarcastic
samples and 5000 Russian jokes &ndash; as sarcastic comments (labelled 1),
and 10000 translated non-sarcastic samples as non-sarcastic comments
(labelled 0). Thus we have get the corpus of 20000 balanced comments.

The split of the dataset to the train, valid and test samples is
presented on the following table. 

|               | Train | Valid |  Test  |
| ------------- |:-----:| ----- |:------:|
| **Comments**  | 11200 | 2800  | 6000   |
| **Tokens**    | 190400| 36400 | 120000 |


## Methods and approaches

We decided to use three train models: TF-IDF with Logistic Regression,
Recurrent Neural Network and Transformer.

As for RNN, it was attempted to use a relatively good quality, and most
importantly, lightweight glove model "navec" ([here](https://github.com/natasha/navec) you can find navec glove model). Using the glove
model, we will build a dictionary and tokenize (converting text strings
into sequences of numbers) every comment. In this case, the size of the
vocabulary was limited to 30,000 words.

As for transformer, we used pre-trained BERT type transformer. The
rubert-tiny2 (sentence encoder model) model was chosen for
classification. As a result, the pre-trained tokenizer and model were
loaded from huggingface ([here](https://huggingface.co/cointegrated/rubert-tiny2) you can find rubert-tiny2 model). For classification, it was necessary to add
a fully connected layer.

## Results

The evaluations (F1-score) of presented methods on the testing set are
showed in a folowing table. 

|    Method    | F1-score |
| ------------ |:--------:|
| Log-Reg      | 0.7868   |
| RNN          | 0.7940   |
| Transformer  | 0.8478   |

The third approach &ndash; Transformers &ndash; has the higher score of detection,
than Recurrent Neural Networks which showed the result close to the
score of the baseline Logistic Regression.

In respect that the presented model is the first model capable to detect
Russian sarcasm, the received results are quite encouraging for the
further researches connected to this field.


## References

1. Dmitry Davidov, Oren Tsur, Ari Rappoport *Semi-supervised recognition of
sarcastic sentences in twitter and amazon* // Proceedings of the
Fourteenth Conference on Computational Natural Language Learning,
Association for Computational Linguistics, 2010.

2. Roberto González-Ibánez, Smaranda Muresan, Nina Wacholder *Identifying
sarcasm in Twitter: a closer look* // Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics: Human Language
Technologies: short papers-Volume 2, Association for Computational
Linguistics, 2011.

3. Stephanie Lukin, Marilyn Walker *Really? well. apparently bootstrapping
improves the performance of sarcasm and nastiness classifiers for online
dialogue* // Proceedings of the Workshop on Language Analysis in Social
Media, 2013.

4. Peng Liu, et al. *Sarcasm detection in social media based on imbalanced classification* // International Conference on Web-Age Information Management, Springer International Publishing, 2014.

5. Francesco Barbieri, Horacio Saggion, Francesco Ronzano *Modelling
sarcasm in twitter, a novel approach* ACL 2014, 2014.

6. Elisabetta Fersini, Federico Alberto Pozzi, Enza Messina *Detecting
irony and sarcasm in microblogs: The role of expressive signals and
ensemble classifiers* // Data Science and Advanced Analytics (DSAA), 2015.

7. Ashwin Rajadesingan, Reza Zafarani, Huan Liu *Sarcasm detection on
twitter: A behavioral modeling approach* // Proceedings of the Eighth
ACM International Conference on Web Search and Data Mining, ACM, 2015.

8. Santosh Kumar Bharti, Korra Sathya Babu, Sanjay Kumar Jena *Parsing-based sarcasm sentiment recognition in Twitter data* // 2015
IEEE/ACM International Conference on Advances in Social Networks
Analysis and Mining (ASONAM). IEEE, 2015.

9. David Bamman, Noah A. Smith *Contextualized sarcasm detection on
twitter* // Ninth International AAAI Conference on Web and Social Media, 2015. 

10. Nabeela Altrabsheh, Mihaela Cocea, Sanaz Fallahkhair *Detecting sarcasm from students' feedback in Twitter* // Design for teaching and learning in a networked world, Springer International Publishing, 2015.

11. Debanjan Ghosh, Weiwei Guo, Smaranda Muresan *Sarcastic or not: word embeddings to predict the literal or sarcastic meaning of words* // Proccedings of the 2015 Conference on Empirical Methods in Natural
Language Processing, Accociation for Computational Linguistics, 2015.

12. Mondher Bouazizi, Tomoaki Otsuki *A pattern-based approach for sarcasm detection on twitter* // IEEE Transl., 2016.

13. S.K. Bharti, et al. *Sarcastic sentiment detection in tweets streamed in real time: a big data approach* // Digital Communications and Networks, 2016.

This project proposes a solution to the problem of Russian sarcastic comments detection. A corpus of Russian-language comments was formed with binary markup: whether the comment is sarcastic or not. The problem was solved using three natural language processing approaches: TF- IDF and Logistic Regression, Recurrent Neural Networks, Transformers. A comparative analysis of the results of these approaches was carried out.

Russian sarcastic comments detectionactive, Founded 3 years ago

Russian sarcastic comments detection
active,
Founded 3 years ago