Fajik: A Neural Encoder-Decoder Model along with Required Language Resources for an Accurate Tajiki-Persian Transliteration

Document Type : .

Authors
Mohammad Ali Sadraei Javaheri 1
Ehsan Asgari 2
Hamid Reza Rabiee 3
1 Master Student at Sharif University, Artificial Intelligence Group, Department of Computer Engineering Data Science & Machine Learning Lab
2 Applied Science and Technology, University of California Berkeley Language Processing and Digital Humanities Lab, AI Group, SUT
3 Professor at Sharif University, Artificial Intelligence Group, Department of Computer Engineering Data Science & Machine Learning Lab
10.30465/lsi.2023.43449.1639
Abstract
The Tajik language, also known as Tajiki Persian, is a variation of Persian (Farsi) spoken in Tajikistan. One of the main distinctions between Iranian Persian and Tajiki Persian is the writing script. Since the early 1900s, Tajiki has been written in the Cyrillic script. Although the difference between spoken Tajiki and spoken Persian is not significant, the script difference caused a culturalbreak between these two nations making many cultural resources (e.g., poems, stories, etc.) unavailable for the newer generations. An automatic and accurate transliterator model can again fill the gap between Persians and Tajiks and facilitate transfer learning between these two language variations. The efforts on Persian-Tajiki transliteration have primarily been rule-based and context-independent, causing many errors. Deep learning methods, particularly neural language models, revolutionized computational linguistics in the last decade by providing language understanding through contextualized representations. In this work, we create a Persian-Tajiki transliteration dataset for training and evaluation purposes. We also train an accurate neural sequence-to-sequence model transliterating between Tajiki-Persian and Persian-Tajiki.

Keywords

Language and Linguistics
Volume 18, Issue 35
February 2023
Pages 87-103