📋
Evaluation Cheat Sheet by Task — Which Metric to Use at a Glance
Classification→Accuracy/F1, Generation→BLEU/ROUGE, OCR→CER/WER, Separation→SDR — with code
All via pip install evaluate. 3 lines of code.
Cheat Sheet
| Task | Primary | Good Level | Key |
|---|---|---|---|
| Classification | Accuracy, F1 | 85%+ | accuracy, f1 |
| Translation | BLEU | 0.3+ | bleu |
| Summarization | ROUGE-L | 0.4+ | rouge |
| OCR | CER | <3% | cer |
| STT | WER | <10% | wer |
| Source separation | SDR | 10+ dB | mir_eval |
| Text generation | BERTScore | 0.85+ | bertscore |
Key Concepts
1
Classification → load("accuracy"), load("f1")
2
Translation/Summary → load("bleu"), load("rouge")
3
OCR/STT → load("cer"), load("wer")
4
Semantic similarity → load("bertscore")
5
Source separation → mir_eval.separation.bss_eval_sources()
Use Cases
Fine-tuning monitoring — auto-evaluate per epoch with Trainer compute_metrics
Model selection — compare CER of TrOCR vs EasyOCR vs PaddleOCR on same data
Production criteria — quality gates like "deploy if CER <3%, BLEU >0.4"