SerbFin-Sentiment
Fine-tuned Serbian financial-sentiment model, on Hugging Face
Macro-F1 0.74 against 0.64 zero-shot and 0.54 for translate-then-FinBERT, on 240 human-labelled headlines
The problem
There was no public model for Serbian financial sentiment. The task is investor-perspective sentiment, not emotion: a headline about cutting 5,000 jobs to protect margins is positive for the market. The usual workaround, translate to English and run FinBERT, loses exactly that nuance.
What I built
- A headline corpus with an LLM-labelled training pool (351 train, 150 validation) and a 240-headline gold test set labelled by a human annotator, balanced across the three classes and held out from training
- A QLoRA fine-tune of Llama 3.2 3B Instruct: 4-bit NF4 quantisation with bf16 compute, LoRA rank 16, alpha 32, dropout 0.05 on all linear projections
- Training with TRL's SFTTrainer, loss on label tokens only, three epochs at learning rate 2e-4 with cosine decay, on a rented RTX A5000 in under 90 seconds
- An evaluation harness that scores the fine-tuned adapter and both baselines on the same gold set and writes macro-F1, per-class metrics, confusion matrices and the malformed-output rate
- Dataset (741 examples, CC BY 4.0) and adapter weights published on the Hugging Face Hub with a generated model card
Results
| System | Macro-F1 | Accuracy | Malformed outputs |
|---|---|---|---|
| Fine-tuned Llama 3.2 3B, QLoRA adapter | 0.74 | 0.73 | 0% |
| Zero-shot Llama 3.2 3B Instruct | 0.64 | 0.64 | 0% |
| Translate to English, then FinBERT | 0.54 | 0.54 | 0% |
Error analysis: 45 of the 64 mistakes are positive or negative headlines predicted as neutral. The training pool is 53% neutral while the test set is balanced, which points at the fix.
Limitations
- The checkpoint was chosen by validation loss on LLM-labelled data, not by macro-F1 on the gold set
- 240 test headlines is enough to rank the three systems, not to resolve differences of a point or two
- Short headlines only; longer texts were not evaluated
Next experiment
Class-weighted loss or a rebalanced training pool, and checkpoint selection by validation macro-F1.
What this means for your project
When an API model is too slow or too expensive for a classification you run thousands of times a day, a small fine-tuned model with a human-labelled test set is the business case. This project is the template: gold test set first, baselines second, fine-tune third, numbers published.