Fostering Research in Arabic NLP via Open Source Collaboration

​

Zaid Alyafeai

Outline

• Arabic NLP challenges.
• Using open source to democratize Arabic NLP.
• From open source to research.
• Beyond NLP.

Arabic Challenges

• Arabic language is written right to left and it contains many letters that are difficult to pronounce by foreigners like ض ، غ ، ح ، خ، ظ .
• Arabic language contains special characters called diacritics that help readers pronounce words correctly شَكْلُهُ.
• Compared to English, the Arabic language words letters are mostly connected اللغة العربية .
• In addition to modern standard arabic (MSA) there are different dialects in different regions around the MENA region.

Arabic Challenges

• Arabic is a highly inflected language with rich morphology.
• There are mainly two types of morphology which are derivational and inflectional.
• Inflectional morphology is used to inflect gender, tense, etc. from a given stem. For example the stem ذهب (go) could have the present tense inflection يذهب (to go) the plural inflection يذهبون (they go) and the feminine inflection ذهبت (she went) .
• Derivational morphology on the other hand causes changes to the meaning of a word or part of speech. For instance, given the stem كتب (write) could have the derivations كاتب (writer), كتابة (writing) and مكتبة library.

Hijjawi, Mohammad, and Yousef Elsheikh. "Arabic language challenges in text based conversational agents compared to the English language." International Journal of Computer Science and Information Technology (IJCSIT) 7.5 (2015): 1-13.

Democratize Arabic NLP

Goals

• Enrich the Arabic content by creating open-source projects.
• Open the community eyes on the significance of ML and NLP.
• Create interactive applications that allow novice Arabs to learn more about machine learning and appreciate its advances.
• Provide researchers and developers with model prototypes, reproducible results and datasets.
• Enrich Arabic NLP research and collaboration.

NLP pipeline

Open Source

• Foster collaboration in the Arabic NLP field.
• An open source first approach to democratize Arabic NLP.
• From open source to academic research.

Bayanat : Dataset Visualization

• Get useful stats about a text dataset.
• Show highest frequent words, chars, etc.
• Sample random sentences, chars, etc.
• Plot embeddings.

https://github.com/ARBML/bayanat

​

Tnkeeh : Dataset Cleaning

• Segmentation.
• Normalization.
• Social media data cleaning.
• Diacritics processing.
• Direct support with huggingface.

https://github.com/ARBML/tnkeeh

​

Tkseem : Dataset Tokenization

• Support for Arabic specific tokenization algorithms.
• Abstract handling of tokenizers.
• Benchmark different tokenizers.

https://github.com/ARBML/tkseem

​

Nmatheg : Training

• Supports fine tuning datasets and models from huggingface.
• Support different tasks, like sentiment analysis, sequence labeling and question answering.
• Currently supports more than 10 datasets in Arabic.

https://github.com/ARBML/nmatheg

​

Deployment Pipeline

https://github.com/ARBML/ARBML

​

Datasets

Models

​

Website

​

https://zaidalyafeai.github.io/ARBML/Interfaces/Website

​

​

​

Captioning

Open source to Research

Arabic Compact Language Modelling

​

​

Alyafeai, Zaid, and Irfan Ahmad. "Arabic Compact Language Modelling for Resource Limited Devices." Proceedings of the Sixth Arabic Natural Language Processing Workshop. 2021.

• Train a teacher model through BERT objective.
• Distill the knowledge to a teacher student weight transfer and pretraining.
• Fine-tune the model in multiple tasks like QA, PoS and sentiment analysis.

Arabic Compact Language Modelling

​

​

• Training a small model for Arabic through distillation.
• The model has 14.2 million params, compared to 135 million parameters for AraBERT.
• The model can be deployed in the browser for multiple tasks like sentiment analysis.

Evaluating tokenizers for Arabic Text classification

​

• Use bayanat to visualize the data and tnkeeh for cleaning.
• Use tkseem for training and evaluating different tokenizers
• Use nmatheg for evaluating and benchmarking the different tokenizers on different datasets.

Alyafeai, Zaid, et al. "Evaluating Various Tokenizers for Arabic Text Classification." arXiv preprint arXiv:2106.07540 (2021).

Evaluating tokenizers for Arabic Text classification

​

• For low resource tasks, the word tokenizer achieves the best results.
• As we increase the size of the vocabulary, the choices of the tokenizer doesn't matter for high resources tasks for larger vocabulary size.

Evaluating tokenizers for Arabic Text classification

​

• The performance of the tokenizer depends highly on the task.
• Poetry classification depends on the character level information.

Beyond NLP

Klaam (From text to Speech)

• Training and fine-tung ASR models using wave2vec models. We have two models for MSA and Egypetian dialect
• Text to speech where we trained a model for MSA.

https://github.com/ARBML/klaam

​

Calliar: Arabic Calligraphy

• Collected 2500 handwritten calligraphy samples.
• The dataset has been annotated for stroke, character and word.
• The dataset can be used for calligraphy generation.

Alyafeai, Zaid, et al. "Calliar: An Online Handwritten Dataset for Arabic Calligraphy." arXiv preprint arXiv:2106.10745 (2021).

https://github.com/ARBML/calliar

​

Rasm: Arabic Art

• StyleGan Model fine-tuned on Arabic Calligraphy
• Calligraphy is not readable but has some aesthetic features.
• Interesting research question, can we generate readable calligraphy ?

https://github.com/ARBML/rasm

​

Future Directions

• Working on the Big Science project for enriching data sourcing for Arabic.
• Would like to support more low resource tasks for Arabic.
• We are working on a model for poetry generation for Arabic.
• Arabic calligraphy recognition and generation.

References

1. Al-shaibani, Maged S., Zaid Alyafeai, and Irfan Ahmad. "Meter classification of arabic poems using deep bidirectional recurrent neural networks." Pattern Recognition Letters 136 (2020): 1-7.
2. Al-Shaibani, Maged S., Zaid Alyafeai, and Irfan Ahmad. "MetRec: A dataset for meter classification of arabic poetry." Data in Brief 33 (2020): 106497.
3. Alyafeai, Zaid, and Maged Al-Shaibani. "ARBML: Democratizing Arabic Natural Language Processing Tools." Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS). 2020.
4. Alyafeai, Zaid, et al. "Calliar: An Online Handwritten Dataset for Arabic Calligraphy." arXiv preprint arXiv:2106.10745 (2021).
5. Alyafeai, Zaid, et al. "Evaluating Various Tokenizers for Arabic Text Classification." arXiv preprint arXiv:2106.07540 (2021).
6. Alyafeai, Zaid, and Irfan Ahmad. "Arabic Compact Language Modelling for Resource Limited Devices." Proceedings of the Sixth Arabic Natural Language Processing Workshop. 2021.