What Is a Confusion Matrix and How Precision and Recall Measure AI Models

A confusion matrix is a table that shows how a classification model’s predictions compare with the real answers. Instead of a single accuracy score, it breaks results into four counts: what the model got right and the two different ways it got things wrong. From those counts come precision and recall, the two metrics that tell you whether an AI model is actually useful for the job.

Why it matters now: AI features are shipping into fraud checks, content moderation, lead scoring, support triage, and medical tools, and many of them are classifiers under the hood, including LLM-based ones that label text. A model that is “95% accurate” can still be useless or harmful. Reading a confusion matrix is how AI engineers and analysts find out.

The four cells of a confusion matrix

Take a model that flags transactions as fraud (positive) or legitimate (negative). Every prediction lands in one of four cells:

• True positive (TP): the model flagged fraud, and it really was fraud.

• False positive (FP): the model flagged fraud, but the transaction was legitimate. Also called a false alarm or Type I error.

• False negative (FN): the model said legitimate, but it was fraud. A miss, or Type II error.

• True negative (TN): the model said legitimate, and it was.

Rows usually represent the actual class and columns the predicted class, but tools differ, so always check the labels. The scikit-learn model evaluation guide documents its conventions and the metrics built on top of the matrix.

Why accuracy alone misleads

Accuracy is (TP + TN) divided by all predictions. Suppose 1 in 100 transactions is fraud. A model that always predicts “legitimate” is 99% accurate and catches zero fraud. This is the class imbalance problem, and it shows up everywhere: rare diseases, churn, defects, security alerts. Google’s Machine Learning Crash Course makes the same point in its lesson on accuracy, precision, and recall.

Precision and recall, explained

Precision: when the model says yes, how often is it right?

Precision = TP / (TP + FP). High precision means few false alarms. It matters when acting on a wrong positive is costly, such as blocking a real customer’s card or auto-deleting a legitimate post.

Recall: of all the real positives, how many did the model catch?

Recall = TP / (TP + FN), also called sensitivity or true positive rate. High recall means few misses. It matters when missing a positive is costly, such as an undetected fraud case or an untriaged urgent support ticket.

F1 score: one number that balances both

F1 is the harmonic mean of precision and recall. It is useful for comparing models quickly, but it hides which kind of error you are making, so look at the matrix too.

The precision–recall trade-off

Most classifiers output a score or probability, and you choose a threshold to turn it into yes or no. Lower the threshold and you catch more positives (recall rises) but raise more false alarms (precision falls). Raise it and the opposite happens. Google’s lesson on thresholds and the confusion matrix shows this visually, and ROC curves and AUC summarize performance across all thresholds.

The right threshold is a business decision, not a purely technical one. Put a cost on each false positive and each false negative, and pick the threshold that minimizes total cost.

A worked example

A support-triage model reviews 1,000 tickets, 100 of which are truly urgent. It flags 120 tickets as urgent, and 80 of those are correct.

• TP = 80, FP = 40, FN = 20, TN = 860

• Accuracy = (80 + 860) / 1,000 = 94%

• Precision = 80 / 120 ≈ 67%

• Recall = 80 / 100 = 80%

The 94% accuracy sounds great, but the matrix tells a sharper story: one in three “urgent” flags is a false alarm, and 20 urgent tickets slip through. Now the team can decide which of those errors to reduce.

Where confusion matrices show up in AI work

• Evaluating LLM classifiers: when an LLM labels sentiment, intent, or policy violations, compare its labels with a human-labeled set and build a matrix. It is a core part of AI evals.

• Catching overfitting: compare the matrix on training data with the matrix on held-out data. A big gap is a classic sign of overfitting.

• Multi-class problems: with more than two classes, the matrix grows to N×N and reveals which classes the model confuses with each other.

• Fairness checks: computing precision and recall separately for different user groups can expose uneven error rates.

Frequently asked questions

Is higher precision or higher recall better?

Neither by default. Favor recall when misses are expensive or dangerous, and precision when false alarms are expensive or annoying. Many teams set a minimum for one and maximize the other.

How do I create a confusion matrix in Python?

With scikit-learn, call confusion_matrix(y_true, y_pred) and classification_report(y_true, y_pred) to get the matrix plus precision, recall, and F1 per class.

Do confusion matrices apply to regression models?

No. They apply to classification, where outputs are categories. Regression models that predict numbers use metrics like mean absolute error or RMSE instead.

Keep learning

If you want to evaluate, tune, and ship AI models with confidence, and analyze the data behind them, explore these programs:

• AI Engineering Course

• Data Analytics Course