This is the ninth post in our AI & ML Foundations series, building on training vs. inference. Start with What Is Machine Learning, Really? if you're joining partway through.
Maya built fraud-detection models for a payments company, and the model review meeting for her latest version opened well: 99.9% accuracy on a held-out test set of 100,000 real transactions. The head of risk started drafting the rollout announcement before Maya had even finished her slide.
She stopped him with one number she'd deliberately saved for last: of the roughly 100 genuinely fraudulent transactions in that same test set, the model had correctly flagged four. Four. It had achieved 99.9% accuracy by doing something almost embarrassingly simple — predicting "not fraud" for nearly everything, which is correct 99.9% of the time in a dataset where fraud is genuinely that rare, and catches almost none of the fraud that actually matters.
Why accuracy quietly stops meaning anything
Accuracy — the fraction of predictions that were correct — is a completely reasonable metric when the two outcomes you're predicting are roughly balanced. It becomes actively misleading the moment one outcome is rare, because a model can score extremely well by simply always guessing the common answer.
from sklearn.metrics import accuracy_score
# 100,000 transactions, only 100 are actually fraud
y_true = [0] * 99900 + [1] * 100
y_pred_lazy = [0] * 100000 # a model that predicts "not fraud" for literally everything
accuracy_score(y_true, y_pred_lazy)
# 0.999A model that never even looks at the input data scores 99.9% here. That's the exact trap Maya's review meeting almost fell into, and it's a genuinely common one in any real-world problem where the interesting outcome — fraud, a rare disease, a mechanical failure — is naturally uncommon.
The confusion matrix: the actual full picture
Instead of collapsing everything into one number, a confusion matrix breaks predictions into four distinct categories, and the distinction between them is where the real information actually lives:
Maya's "lazy" model had zero true positives and zero false positives — it never predicted fraud at all — and a wall of true negatives, which is exactly what let it look nearly perfect on accuracy while being operationally useless.
Precision and recall: the two questions that actually matter
Two metrics, built directly from the confusion matrix, ask the two questions a fraud team actually cares about:
from sklearn.metrics import precision_score, recall_score
precision_score(y_true, y_pred)
# "Of everything we flagged as fraud, what fraction actually was?"
recall_score(y_true, y_pred)
# "Of all the actual fraud that existed, what fraction did we catch?"- Precision answers "when the model says fraud, can we trust it?" Low precision means a lot of false alarms — legitimate customers getting their cards blocked, a real cost in complaints and lost business.
- Recall answers "of all the fraud that actually happened, how much did we catch?" Low recall — Maya's original 4-out-of-100 — means the model is missing the vast majority of what it exists to catch.
These two pull in genuinely opposite directions. A model that flags every transaction as fraud achieves perfect recall (100% of real fraud gets caught, trivially) and terrible precision (almost every flag is a false alarm). A model that only ever flags the single most obviously fraudulent transaction pattern achieves excellent precision and terrible recall. Neither extreme is actually useful — the real decision is where between them a specific business actually needs to land.
The threshold: the dial that trades one for the other
Most classifiers don't output a hard yes/no — they output a probability, and a chosen threshold decides where "probably fraud" becomes an actual flag:
probabilities = model.predict_proba(X_test)[:, 1] # probability of fraud, per transaction
threshold = 0.3 # lower threshold: flag more aggressively
predictions = (probabilities > threshold).astype(int)Lowering the threshold from, say, 0.5 to 0.3 flags more transactions as fraud — recall goes up (catching more real fraud), precision goes down (more false alarms mixed in). This is a genuine business decision, not a purely technical one: a payments company that loses more from undetected fraud than it loses from annoyed, wrongly-blocked customers should set a lower threshold; one where blocked legitimate transactions are the costlier problem should set it higher. No single "correct" threshold exists independent of what a false positive versus a false negative actually costs the business.
F1 score: one number, when you genuinely need one
For cases where precision and recall need to be balanced into a single comparable number — ranking several candidate models, say — the F1 score combines them:
from sklearn.metrics import f1_score
f1_score(y_true, y_pred)F1 is the harmonic mean of precision and recall, which — unlike a simple average — penalizes a model heavily if either precision or recall is very low, even if the other is excellent. A model with 99% precision and 2% recall (Maya's original model, roughly) gets a low F1 score, correctly reflecting that it's not actually a good model despite one of its two component numbers looking great in isolation.
What Maya's team actually did
Rather than deploying on accuracy alone, Maya's team picked a threshold using a precision-recall curve — plotting precision against recall across every possible threshold — and chose the point that matched their actual business tolerance for false alarms versus missed fraud, then monitored precision and recall separately in production going forward, never accuracy alone again.
AUC-ROC: evaluating across every possible threshold at once
Choosing one threshold and reporting precision and recall at that single point is useful, but it only tells you about one specific operating point. The ROC curve plots the trade-off between catching real fraud and generating false alarms across every possible threshold simultaneously, and AUC (area under that curve) summarizes it into one number:
from sklearn.metrics import roc_auc_score
roc_auc_score(y_true, probabilities)
# 0.5 = no better than random guessing
# 1.0 = perfect separation between fraud and legitimate transactionsAn AUC of 0.5 means the model's fraud-probability scores are no more informative than a coin flip, regardless of what threshold you'd choose to apply; an AUC close to 1.0 means fraudulent transactions genuinely tend to score higher than legitimate ones across the board. This is worth checking specifically because it's threshold-independent — a useful way to compare two candidate models' underlying discriminative quality, before ever getting into the business decision of exactly where to set the operating threshold for either one.
Comparing models honestly requires the same test set
A detail worth being explicit about: comparing precision, recall, or AUC across two candidate models only means something if both were evaluated against the exact same held-out test set, never seen during either model's training. Evaluating Model A against one sample of transactions and Model B against a different sample — even if both samples are drawn from the same underlying population — introduces noise that can easily make a genuinely worse model look better, purely by chance, the same "compare on identical, controlled conditions" discipline that matters for A/B testing any other system change.
What to actually remember from this post
- Accuracy is meaningless, or actively misleading, on imbalanced data — a model that always predicts the common outcome can score deceptively well while being operationally useless.
- The confusion matrix's four categories (true/false positive/negative) contain the real information that accuracy alone collapses away.
- Precision and recall ask two different, often competing questions — "can we trust a positive prediction?" versus "did we catch everything that mattered?"
- The classification threshold is a genuine business decision, not a purely technical default — it trades precision against recall based on what a false positive actually costs versus what a false negative costs.
- F1 score combines precision and recall into one number, specifically penalizing a model that's excellent on one and poor on the other.
Next in the series: Generative Models: From GPT to Diffusion, the final post — covering why text generation and image generation don't actually work the same way under the hood.