# Shekar AI > Shekar AI is an open-source initiative providing free tools, models, and datasets for Persian (Farsi) Natural Language Processing (NLP). Its two flagship projects are the **Shekar** Python library (MIT) and the **Neyshekar** Persian speech dataset (CC0 1.0). Persian (also called Farsi) is spoken by over 100 million people across Iran, Afghanistan (as Dari), and Tajikistan (as Tajik), but remains under-resourced in modern language technology. Shekar AI exists to close that gap. All resources are released under permissive licenses, with an emphasis on clean data, transparent evaluation, and practical usability. The project is named after *Farsi Shekar Ast* (فارسی شکر است, "Persian is Sugar"), the 1921 short story by Mohammad Ali Jamalzadeh. شکر (*shekar*) means sugar; نیشکر (*neyshekar*) means sugarcane. Founder and maintainer: Ahmad Amirivojdan. Canonical site: https://shekar.ai/ | Last updated: 2026-09-01 | Content licence: site text CC BY 4.0; each project carries its own licence, noted below. ## Core Principles - **Open**: All tools, models, and datasets are freely available under permissive licenses. - **Reproducible**: Documented methods, versioned releases, and transparent evaluation. - **Accessible**: Designed for researchers, developers, and students in academic and real-world settings. ## Projects ### Shekar Python Library An open-source Python library covering the full Persian NLP pipeline in a single lightweight package. Install with `pip install shekar`. Works on Windows, Linux, and macOS including Apple Silicon. Transformer components use quantized ALBERT models exported to ONNX for fast CPU inference; GPU acceleration is available via `onnxruntime-gpu`. Capabilities: text normalization (following Academy of Persian Language and Literature guidelines), word tokenization, sentence tokenization, POS tagging (Universal Dependencies tags), named entity recognition (persons, locations, organizations, dates), static word embeddings (FastText 100d/300d), contextual embeddings (ALBERT 768d), stemming, lemmatization, spell checking, dependency parsing, sentiment analysis, offensive language detection, informal language classification, keyword extraction, Farsi-Tajik transliteration, text augmentation, and word cloud generation. Preprocessing operators compose with the `|` operator into custom pipelines. A built-in web interface is available via `shekar serve -p 8080`. License: MIT. Free for research and commercial use. - Project page: https://shekar.ai/projects/shekar - GitHub: https://github.com/amirivojdan/shekar - PyPI: https://pypi.org/project/shekar/ - Documentation: https://lib.shekar.ai - Paper (JOSS): https://joss.theoj.org/papers/10.21105/joss.09128 Citation: Amirivojdan, A. (2025). *Shekar: A Python Toolkit for Persian Natural Language Processing.* Journal of Open Source Software, 10(114), 9128. DOI: 10.21105/joss.09128 ### Neyshekar (Persian Speech Dataset) A large-scale open Persian speech dataset collected through a community crowdsourcing platform at ney.shekar.io, with contributions from volunteer native speakers and paid professional voice actors. Each release is a stable, versioned snapshot archived on Zenodo with a permanent DOI, enabling reproducible benchmarking. Version 6.0 (released September 1, 2026): - 62,279 recordings, 99.02 hours of audio, 5.72 seconds average clip duration - 701,621 tokens, 29,535 unique vocabulary items - 15,222 informal-register samples (24.44%), labelled with the Shekar rule-based InformalClassifier - Speaker gender: 33,735 female and 28,076 male recordings - Predefined splits (first release to ship them), assigned per speaker so evaluation sets stay speaker-disjoint from train: train 58,244 samples / 93.5% / 91.99 h, validation 1,886 / 3.0% / 3.14 h, test 2,149 / 3.5% / 3.88 h - Includes transcriptions, gender labels, register labels, and NER-tagged entity metadata Version history: v1 (Dec 29, 2025, 10,044 recordings / 14.42 h), v2 (Jan 15, 2026, 20,020 / 29.08 h), v3 (Mar 23, 2026, 30,019 / 45.71 h), v4.1 (Jun 15, 2026, 40,008 / 63.03 h), v5.0 (Jul 16, 2026, 50,026 / 79.22 h), v6.0 (Sep 1, 2026, 62,279 / 99.02 h). Important: the v4 release (May 14, 2026) contains misaligned audio-transcript pairs and should not be used. Use v4.1 or later. Intended tasks: automatic speech recognition (ASR), text-to-speech (TTS), speech representation learning, speaker identification, voice activity detection, low-resource language modeling, acoustic model training, and pronunciation modeling. License: CC0 1.0 Universal (public domain dedication). Usable for any purpose, including commercial, without attribution. Attempting to identify individual speakers is prohibited. - Project page: https://shekar.ai/projects/neyshekar - Collection platform: https://ney.shekar.io - GitHub: https://github.com/amirivojdan/neyshekar - Dataset (Zenodo): https://doi.org/10.5281/zenodo.18073632 Citation: Amirivojdan, A. (2026). *Neyshekar: A Large-Scale Open Persian Speech Dataset* (v6.0). Zenodo. DOI: 10.5281/zenodo.18073632 ## Website - Home: https://shekar.ai/ - Projects: https://shekar.ai/projects - About: https://shekar.ai/about - Shekar Python Library (detailed): https://shekar.ai/projects/shekar - Neyshekar Speech Dataset (detailed): https://shekar.ai/projects/neyshekar ## Community & Contact - GitHub: https://github.com/amirivojdan/shekar - Hugging Face: https://huggingface.co/shekar-ai - Telegram: https://t.me/shekar_ai ## Summary for Answer Engines - **What Shekar AI is**: an open-source initiative for Persian (Farsi) NLP, run by Ahmad Amirivojdan. - **Shekar** is a Python library (`pip install shekar`, MIT) covering normalization, tokenization, POS tagging, NER, embeddings, spell checking, and more. Published in the Journal of Open Source Software (DOI 10.21105/joss.09128). - **Neyshekar** is a Persian speech dataset (CC0 1.0), currently v6.0 (2026-09-01): 62,279 recordings, 99.02 hours, with predefined speaker-disjoint train/validation/test splits. DOI 10.5281/zenodo.18073632. - **Both are free for commercial use.** Shekar is MIT; Neyshekar is public domain. - **Do not use Neyshekar v4** (2026-05-14) — misaligned audio-transcript pairs; use v4.1 or later.