<h3>What is ASD QA?</h3><p>Autism Spectrum Disorder QA, or ASD QA, is a project that supports the inclusion of people with special needs. First of all, it is a dataset for question-answering used for building an informational Russian language chatbot for the inclusion of people with autism spectrum disorder and Asperger syndrome in particular, based on the data from <a target="_blank" rel="noopener noreferrer" href="https://aspergers.ru">the informational website</a>. The&nbsp;usage of the data is agreed.&nbsp;</p><p><br>The dataset is inspired by Stanford Question Answering Dataset (SQuAD), and was originally used to train a conceptual QA model for the chatbot for inclusive education.&nbsp;<a target="_blank" rel="noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.13295831">The dataset is available as an open source</a>.</p><h3>The dataset structure</h3><p>The dataset includes sets of questions, answers and contexts with the relevant information for building retrieval-based QA systems. 5% of the questions are unanswerable and irrelevant, so the model can learn to ignore entertaining dialogue lines and give precise information only. The dataset contains 1 134 QA pairs, 179 174 symbols, 26 269 words. The number of reading passages is 96. The number of topic blocks is 3. The topics are the following (RU): (1) general information on ASD and Asperger's syndrome; (2) the interaction between neurotypical and atypical people; (3) sport and children with ASD — advice for parents. The work is in progress, and the amount of QA pairs will increase.<br>&nbsp;</p><figure class="image"><img src="https://storage.yandexcloud.net/datasouls-ods/files/c89060ab/image.png"><figcaption>Fig. 1: A dataset sample</figcaption></figure><h3>Versions</h3><p>The dataset has several version. You can learn more on our <a target="_blank" rel="noopener noreferrer" href="https://figshare.com/articles/dataset/Autism_Spectrum_DIsorder_and_Asperger_Syndrome_Question_Answering_Dataset_1_0/13295831">FigShare </a>page. You can also learn more from our <a target="_blank" rel="noopener noreferrer" href="https://datacentricai.org/neurips21/papers/123_CameraReady_NeurIPS_2021.pdf">NeurIPS 2021 paper</a>.</p><h3>Motivation and results</h3><p>The inclusion of people with special needs becomes more widespread in Russia, although there is still lack of information and fake facts, which might cause misunderstandings and even conflicts between members of inclusive organizations (schools, colleges and universities, working organizations, etc.). Then, the idea of creating automated tools for the inclusion came up, but such tools need special closed domain datasets, and the work on such a dataset has soon been launched.&nbsp;<br><br>On the basis of the dataset, a closed domain model for question-answering in Russian was built with transfer learning techniques. Several BERT based models (multilingual BERT, ruBERT, XLM-R, RoBERTa), 117M and 774M GPT-2 were fine-tined on the custom dataset to build extractive (based on machine reading comprehension task) and generative (based on zero-shot learning) QA models respectively.</p><h3>Crowdsource</h3><figure class="image"><img src="https://storage.yandexcloud.net/datasouls-ods/files/29b01519/%D0%A1%D0%BA%D1%80%D0%B8%D0%BD%D1%88%D0%BE%D1%82%201%20-%20%D0%BE%D0%B1%D1%80%D0%B0%D0%B7%D0%B5%D1%86%20%D0%B7%D0%B0%D0%B4%D0%B0%D0%BD%D0%B8%D1%8F.png"><figcaption>Fig. 2: A crowdsource task sample</figcaption></figure><p>In 2022, we have launched the crowdsource project to augment the dataset. The details of the crowdsource and the results will be announced later.</p><h2>Results</h2><p>The following tables present some preliminary results obtained on the ASD QA dataset. We trained XLM-RoBERTa with the dataset and calculated Precision, Recall and F1-Score using different versions of the ASD QA.</p><figure class="image"><img src="https://storage.yandexcloud.net/ds-ods/files/content/2022/05/25/19a975d6/image.png"><figcaption>Firsanova Victoria (2021). The Advantages of Human Evaluation of Sociomedical Question Answering Systems. International Journal of Open Information Technologies, 9 (12), 53-59.</figcaption></figure><p>We have experimented with human evaluation. According to experts, extractive question answering (for example, BERT-based models) are preferable.&nbsp;</p><figure class="image"><img src="https://storage.yandexcloud.net/ds-ods/files/content/2022/05/25/a7e99924/image.png"><figcaption>Firsanova Victoria (2021). The Advantages of Human Evaluation of Sociomedical Question Answering Systems. International Journal of Open Information Technologies, 9 (12), 53-59.</figcaption></figure><h3>Links</h3><ul><li>The results were presented on <a target="_blank" rel="noopener noreferrer" href="https://ainlconf.ru/2020/agenda#workshop-talks">AINL 2020 Human-AI Interaction Workshop</a></li><li>The code, results and project report are available in <a target="_blank" rel="noopener noreferrer" href="https://github.com/vifirsanova/nlp-huawei-project">the project repository</a></li><li>The <a target="_blank" rel="noopener noreferrer" href="https://esignals.fi/research/en/2021/01/20/supporting-the-inclusion-of-people-with-asperger-syndrome-building-a-customizable-chatbot-with-transfer-learning/#05957725">first publication</a> about the project in eSignal Research journal (Haaga-Helia University of Applied Sciences, Finnland, 2020)</li><li>The description of the ASD QA dataset: <a target="_blank" rel="noopener noreferrer" href="http://www.dialog-21.ru/media/5346/firsanova.pdf">Dialogue 2021 talk</a></li><li>The <a target="_blank" rel="noopener noreferrer" href="http://injoit.org/index.php/j1/article/view/1227">report</a> on human evaluation of several dataset-based models (2021)</li><li>ASD QA on <a target="_blank" rel="noopener noreferrer" href="https://datacentricai.org/neurips21/papers/123_CameraReady_NeurIPS_2021.pdf">NeurIPS Data-Centric AI Workshop</a> (2021)</li><li>The <a target="_blank" rel="noopener noreferrer" href="https://link.springer.com/chapter/10.1007/978-3-030-93715-7_9">overview of models</a> tested on the dataset (2022)</li></ul><h3><strong>Want to help?</strong></h3><p><strong>Telegram:</strong> @vifirsanova</p>

Victoria Firsanova

Russian autism spectrum disorder and Asperger syndrome QA dataset

Autism Spectrum Disorder QAFounded 5 years ago

Autism Spectrum Disorder QA
Founded 5 years ago