Shekar

Shekar is an open-source Python library for Persian natural language processing. The name comes from Mohammad Ali Jamalzadeh's satirical story Persian is Sugar (فارسی شکر است), and like the story, the library puts clear, accessible language first.

It gives researchers and engineers fast, composable building blocks that run on a CPU with small models and minimal dependencies, and scale up with batch inference and CUDA when needed.

What it does

• NormalizationFollows the Academy of Persian Language and Literature guidelines
• TokenizationWord and sentence tokenizers
• Stemming & lemmatizationIncluding compound and prefixed verbs
• POS taggingPart-of-speech tags for Persian text
• Named entitiesPeople, places, organizations and more
• Dependency parsingSyntactic structure of sentences
• EmbeddingsFastText word vectors and 768-d ALBERT contextual embeddings
• ClassificationSentiment, toxicity and informal-language detection
• TransliterationBetween Persian and Latin script, both directions
• Spell checkingFind and correct misspellings
• Keywords & augmentationKeyword extraction and text augmentation
• Web interfaceExplore every tool with shekar serve

Composable pipelines

Preprocessing is built from small, independent operators (filters, normalizers, maskers) that can be used alone or chained with the | operator. The default Normalizer is itself just such a pipeline, and any model can be appended to one.

from shekar import Normalizer, NER
from shekar.preprocessing import EmojiRemover, PunctuationRemover

clean = EmojiRemover() | PunctuationRemover()
print(clean("ز ایران دلش یاد کرد و بسوخت! 🌍🇮🇷"))
# ز ایران دلش یاد کرد و بسوخت

pipeline = Normalizer() | NER(batch_size=32)
entities = list(pipeline(["علی در تهران زندگی می‌کند."]))

Built for production

• Models run on ONNX Runtime, on CPU by default on Windows, Linux and macOS (including Apple Silicon), with optional CUDA acceleration.
• Batch-capable models (NER, POS tagging, embeddings, classifiers, parsing and transliteration) accept a list of texts and keep the single-text API unchanged.
• Hundreds of test cases with more than 95% code coverage, run in CI on every change.

Citation

@article{Amirivojdan_Shekar,
  author  = {Amirivojdan, Ahmad},
  title   = {{Shekar: A Python Toolkit for Persian Natural Language Processing}},
  journal = {Journal of Open Source Software},
  volume  = {10},
  number  = {114},
  pages   = {9128},
  year    = {2025},
  doi     = {10.21105/joss.09128}
}