Text Normalization for Social Media Corpus

Abstract

Text markup is the process of enriching text with metalinguistic information, such as lemmata, morphological tags, and syntactic relations. Text markup is a fundamental task in computational linguistics and plays a key role in both fundamental and applied natural language processing. The scale of modern text corpora, with over a billion word tokens, has necessitated the use of automated tagging tools. Most such solutions have been developed and tested on texts with standard orthography. Consequently, their performance can vary significantly when applied to texts from social media, which contain non-standard usages. One way to overcome this problem is through pre-normalization of the text. This task is similar to, but not equivalent to, automatic spell correction and remains significantly less studied. Text normalization requires not only the correction of typos and spelling errors but also the restoration of abbreviations, especially those common in online communication. In this article, we present a corpus of Russian-language sentences from social media and their normalized variants, available at https://huggingface.co/datasets/ruscorpora/normalization. Using this corpus, we compiled a list of typical speech distortions and compared the effectiveness of different text normalization methods.