Low-resource grammar-rich modeling
Sanskrit / Panini model stack
A Sanskrit grammar-focused model stack spanning tokenizer design, synthetic corpora, staged SFT, prakriya chain-of-thought, grammar-reward RL, and mechanistic probes.
Core hypothesis
If affordable AGI has to work beyond high-resource English, low-resource grammar-rich languages are a useful stress test. This stack asks what tokenizer, reward, and training choices waste or preserve linguistic structure.
Steps taken
- Built a Panini-aware tokenizer and benchmarked it against Sanskrit-BERT, MuRIL, Ansh-256k, and Qwen2 on complex compounds.
- Generated multi-million-line Sanskrit semantic and grammar-derived corpora using dictionary, ontology, morphology, and Vidyut resources.
- Trained staged models: V1 base, V10 syntax SFT, V11 prakriya/CoT, and RL variants with PaniniJudge rewards, KL regularization, teacher warmup, and reward-hacking checks.
- Built a mechanistic-interp stack comparing Sanskrit grammar-RL models against matched Sanskrit and English CLM controls.
Results
Average compound tokens: Panini 7.2 vs Sanskrit-BERT 10.6, MuRIL 15.9, Ansh-256k 14.4, Qwen2 21.8.
Example compression: nirapekzajYAnasAkzAtkArasAmarthyam is 6 Panini tokens vs 25 Qwen2 tokens.
Model stack includes an approximately 246M-parameter LLaMA-style Sanskrit model; V10 syntax reached 119000 steps and V11 CoT reached 5000 steps.
V1 base training reached 284000 steps with loss down to 2.964; V10 syntax loss reached 2.7445; V11 CoT loss reached 1.6416.
RL/mech-interp experiments showed lower effective rank and stronger top-4 mass on controlled Sanskrit grammar contrast families, but only under narrow contrast-suite conditions.
Caveat
The earned claim is narrow: grammar-aware tokenization and controlled RL produce useful stress tests and localized signals. It does not prove Sanskrit models reason better in general.
Status
Experimental research stack; useful as a low-resource model-training and grammar-alignment testbed.