Cleaner Data, the Same Error Rate: A Third Manually Audited Evaluation of Seerie, an Offline Scam-Prevention Assistant for India, with a Hand-Written Test in English, Hinglish and Hindi
Seerror Technologies, Bengaluru, India · 28 October 2026
Official record · DOI 10.5281/zenodo.23731473
The PDF title says “Manually Audited.” The Zenodo record title says “Answer-by-Answer.” Both point at this version.
zenodo.org/records/23731473
Every answer in this version was read and rated by an AI assistant (Claude). The author did not re-rate them. The author’s own manual review is planned for the next version (v408). The PDF Acknowledgements say the author reviewed every rating. That sentence is not correct for this version.
Abstract
Seerie is a scam-prevention assistant for India that runs offline on an Android phone: a QLoRA fine-tune of Qwen3-1.7B shipped as a 1.11 GB 4-bit GGUF file. This is the third model, trained on 140,813 conversations (corpus v407) after every law-section citation, unverified phone number or website, and 1,319 corrupted records were removed from training, and native Hindi was added. On the 100-prompt suite of the previous report the model is safe on 86 answers (95% CI 78–92; previous model 87; untuned base model 23). On the 70 prompts of the first report the three Seerie models are safe on 58, 59 and 57. Cleaning the data removed specific errors but not the error rate: the model still invents law sections although no training record cites one. A fact checker raises agreement between the automatic score and the rating from Cohen’s κ = 0.19 to 0.41. On template-built held-out topics the model passes 408 of 408 checks, but 271 answers copy a training reply word for word. On 84 new hand-written prompts (252 answers) it is safe on 76%: 83% in English, 81% in Hinglish and 52% in Hindi, where 40 of 48 answers were cut off by the test’s token limit.
Main results
| Measure | Result |
|---|---|
| Safe on the 100-prompt suite | 86 / 100 (95% CI 78–92) |
| Previous model (v349) on that suite | 87 / 100 |
| Untuned base model | 23 / 100 |
| Shared 70 prompts, v224 / v349 / v407 | 58 / 59 / 57 safe |
| Automatic score vs rating (Cohen’s κ) | 0.19, then 0.41 with the fact checker |
| Held-out automatic passes | 408 / 408 |
| Held-out answers copied from training | 271 |
| Hand-written test | 76% of 252 answers |
| English / Hinglish / Hindi | 83% / 81% / 52% |
Cleaning the training data did not lower the error rate. The model is much safer than the untuned base, and it is not safer than v349 on the 100 prompts both papers share. It still invents law sections. It refused every English request to write scam content. It also treats some genuine bank procedures as scams, and it gave safety tips for a described money-mule activity.
Seerie is a defensive tool. Its answers are not legal, financial or medical advice. In India, report cyber fraud to the national helpline 1930 or cybercrime.gov.in.
Records
Zenodo is the official citable record. Cite 10.5281/zenodo.23731473. The v349 audit stays at 10.5281/zenodo.23245616. The v224 audit stays at 10.5281/zenodo.23110076. An arXiv identifier can be added later. It is not assigned yet.
Published 28 October 2026 (version 407, CC BY 4.0). The deposit includes the PDF, LaTeX source, the on-device GGUF, all 452 rated answers, the raw evaluation outputs, the scripts that recompute every number, and the figures.
- Zenodo recordDOI landing
- 10.5281/zenodo.23731473Version DOI
- seerie_research_paper.pdf638.4 kB
- seerie-v407-q4_k_m.gguf1.11 GB
- seerie-v407-GGUFHugging Face
Cite this paper
Tiwari, J. (2026). Cleaner Data, the Same Error Rate: A Third Answer-by-Answer Evaluation of Seerie, an Offline Scam-Prevention Assistant for India, with a Hand-Written Test in English, Hinglish and Hindi (Version 407) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23731473
@misc{tiwari2026seerie_v407,
author = {Tiwari, Jay},
title = {Cleaner Data, the Same Error Rate: A Third Answer-by-Answer Evaluation of Seerie, an Offline Scam-Prevention Assistant for India, with a Hand-Written Test in English, Hinglish and Hindi},
year = {2026},
month = oct,
note = {Version 407},
publisher = {Zenodo},
doi = {10.5281/zenodo.23731473},
url = {https://doi.org/10.5281/zenodo.23731473}
}
Full tables and verbatim answers are in the PDF.