Shiraz
The FineWeb of Persian
Shiraz is a quality-filtered Persian web corpus with synthetic retrieval queries attached: 6,254,329 documents from 14 sources spanning Wikipedia, academic abstracts, news, magazines and blogs. Blog content is scored by an LLM judge and filtered for depth, coherence and spam. The clean text suits LLM pretraining, and most documents also come with LLM-generated questions and search queries for training and evaluating retrieval and embedding models.
Dataset Statistics
What Shiraz Is For
LLM Pretraining
Use the text column, 6.25M documents spanning encyclopedic, academic, news and informal blog registers, for pretraining or continued pretraining of LLMs. The judge_* scores on blog content let you select stricter, higher-quality subsets.
Embeddings & Retrieval
The synthetic questions and search queries pair with their source documents as ready-made training pairs for embedding and retrieval models, and as a base for hard-negative mining. Noql was trained this way.
How Shiraz Was Built
LLM-Judged Blogs
Each document from the four blog platforms (mihanblog, virgool, blogsky, blogfa) was scored by google/gemma-4-26B-A4B-it on content depth, boilerplate-freeness, spam-freeness, coherence and retrieval value, then kept or rejected by a threshold on the overall score.
Trusted Sources As-Is
The other ten domains are established news, reference, academic or editorial sources (Wikipedia, BBC Persian, Civilica, Zoomit and others) and were included without judging.
Synthetic Retrieval Queries
For every row except Civilica, an LLM generated natural-language questions (formal and informal) and short search-style queries a user might issue to find that document, scaled in number to the document's length.
Columns
title,text,url,tags,idDocument content and identityjudge_*LLM-judge scores and verdict (null for trusted, unjudged sources)query_questions,query_search_queriesSynthetic retrieval queries generated per document
Usage
Shiraz is about 6 GB, so streaming is the easiest way to start:
Use Cases
License & Attribution
ODC-BY 1.0
Shiraz is released under the Open Data Commons Attribution License (ODC-BY 1.0) for the dataset compilation itself. This covers Shiraz's selection, structure and annotations (judge scores and synthetic queries). Attribute it as "Shiraz" if used.
Note: the license does not grant rights to the underlying web content, which remains subject to its original authors' rights and each source site's terms.
Frequently Asked Questions
What is Shiraz?
Shiraz is an open, quality-filtered Persian (Farsi) web corpus of 6,254,329 documents from 14 sources, published on Hugging Face as shekar-ai/Shiraz. Blog content is filtered by an LLM judge, and most documents come with synthetic search queries. It is suited both to LLM pretraining and to training and evaluating retrieval and embedding models.
Can Shiraz be used to train LLMs?
Yes. The text column is a large, quality-filtered Persian corpus for pretraining or continued pretraining of LLMs. The judge_* scores on blog content can be used to select stricter, higher-quality subsets.
Why is it called "the FineWeb of Persian"?
Like FineWeb for English, Shiraz is a web-scale corpus built by filtering noisy web text for quality rather than simply collecting everything. Low-quality blog posts are scored and removed by an LLM judge, while established news, reference, academic and editorial sites are kept as-is.
How were documents filtered?
Documents from the four blog platforms (mihanblog.com, virgool.io, blogsky.com and blogfa.com) were scored by an LLM judge (google/gemma-4-26B-A4B-it) on content depth, boilerplate-freeness, spam-freeness, coherence and retrieval value, then kept or rejected by a threshold on the overall score. The other ten sources are established news, reference, academic or editorial sites and were included without judging.
What are the synthetic queries?
For every row except Civilica, an LLM generated natural-language questions (formal and informal) and short search-style queries that a user might issue to find that document, scaled in number to the document's length. They are stored in query_questions and query_search_queries; 4,016,386 rows (64.2%) have them.
How do I load Shiraz?
Use the Hugging Face datasets library: load_dataset("shekar-ai/Shiraz", split="train", streaming=True). Streaming is recommended because the dataset is about 6 GB on the Hub.
What license does Shiraz use? Can I use it commercially?
The compilation (its selection, structure, judge scores and synthetic queries) is released under ODC-BY 1.0, which allows commercial use with attribution; attribute it as "Shiraz". ODC-BY does not grant rights to the underlying web content, which remains subject to its original authors' rights and each source site's terms.
Which models were trained on Shiraz?
Noql, Shekar AI's 12M-parameter Persian text embedding model, was trained on Shiraz query–document pairs and mined hard negatives.