SMS scam & spam detection
A Transformer encoder, built from scratch in PyTorch, that separates scam and spam messages from legitimate ones on a heavily imbalanced dataset, and gets brand-new, hand-written messages right too.
Reading intent, not just keywords.
This project is a text classifier that decides whether an SMS is a scam/spam message or legitimate. Rather than reaching for a pre-trained library model, I implemented a Transformer encoder from scratch in PyTorch: tokenisation, embeddings, self-attention, and a classification head, to understand the architecture end to end.
On a highly imbalanced dataset it reaches 98% accuracy and a 0.95 macro-F1, and, the part that matters most, it correctly labels fresh messages I wrote by hand that never appeared in training.
Try to trick it.
The clip above is a recording. Below is the real model, embedded live: type any message and the Transformer encoder I built entirely from scratch in PyTorch classifies it in real time and shows you which words drove the decision.
When 98% accuracy can still be useless.
Scam and spam SMS drive real financial fraud, but scam messages are rare compared to normal ones: the dataset is heavily imbalanced. A lazy model can score high accuracy simply by calling almost everything "legit" and quietly missing the scams, which is exactly the failure that hurts people.
So the real target wasn't raw accuracy; it was a strong macro-F1 that treats the rare scam class as seriously as the common one, plus the ability to generalise to unseen phrasing, because scammers never reuse the same wording twice.
Built the model, top to bottom.
This was an individual deep-learning project, so I owned every layer:
Data & tokenisation
Prepared the imbalanced corpus and a tokenizer + vocabulary for the encoder.
Model
Implemented the Transformer encoder from scratch: embeddings, multi-head attention, FFN, classifier head.
Training
Trained and tuned for the minority class, optimising for macro-F1 rather than accuracy alone.
Evaluation
Validated on held-out data and on new hand-written messages to test real generalisation.
What it's built with.
Modelling
ML utilities
Domain
Inside the encoder.
A message becomes tokens, tokens become embeddings, and stacked self-attention blocks build a representation that a small head turns into a scam probability.
What it does.
- From-scratch Transformer encoder in PyTorch: attention and positional encoding implemented directly, not imported.
- Imbalance-aware training and evaluation, optimised for macro-F1 so rare scams aren't ignored.
- Probability output: every message gets a scam probability, not just a hard label.
- Generalises to unseen text: correctly classifies brand-new hand-written messages at inference time.
The hard parts, and how I solved them.
Building attention by hand
Implementing the encoder from scratch meant getting embeddings, positional information, multi-head self-attention, and the classification head all working together correctly: a deep dive that made the architecture genuinely mine rather than a black box.
Scammers never repeat themselves
To prove it learned intent and not memorised phrasing, I tested it on new messages I wrote by hand. As the demo shows, it flags obvious scams with high probability and passes normal messages through: evidence it generalises beyond the training set.
Strong where it counts.
The classifier pairs high accuracy with a macro-F1 that shows it isn't cheating on the imbalance, and it holds up on unseen, hand-written messages, which is the real test of a scam detector.