What Is the Bias-Variance Tradeoff and Why Machine Learning Models Underfit or Overfit

The bias-variance tradeoff is the tension between two ways a machine learning model can get predictions wrong. Bias is error that comes from a model being too simple to capture the real pattern in the data. Variance is error that comes from a model being so sensitive to its training data that it learns noise and changes a lot when the data changes. Making a model more flexible usually lowers bias but raises variance, and the goal is to find the level of complexity that keeps total error on new data as low as possible.

It is one of the first concepts anyone learning machine learning runs into, and it still explains most day-to-day modeling decisions: how deep a decision tree should be, how many features to keep, how much regularization to apply, and when collecting more data will actually help. As IBM’s explainer on the bias-variance tradeoff puts it, the concept is about balancing underfitting and overfitting so a model generalizes well.

Bias and variance in plain terms

High bias: the model is too simple

Imagine fitting a straight line to data that clearly curves. No matter how much data you give it, the line will miss the curve. That systematic miss is bias. High-bias models underfit: they perform poorly on the training data and on new data.

High variance: the model is too flexible

Now imagine a model that bends to pass through every single training point, including the random noise. It looks perfect on training data, but if you trained it on a slightly different sample, you would get a very different model. That instability is variance. High-variance models overfit: great training scores, disappointing results in production. We cover that failure mode in detail in what overfitting is and why models fail on new data.

Irreducible error

For problems measured with squared error, expected prediction error can be split into three parts: bias squared, variance, and irreducible error (the noise in the data itself). You can trade bias against variance, but no model removes the irreducible part. Knowing that keeps expectations realistic.

How to tell which problem you have

The practical diagnostic is to compare training error with validation error:

Both errors high and close together: likely high bias. The model is not capturing the pattern.

Training error low, validation error much higher: likely high variance. The model memorized the training set.

Both low and close: you are in a good spot.

Learning curves and validation curves make this visible. The scikit-learn guide to validation and learning curves shows how to plot training and validation scores as you change a hyperparameter or the amount of training data. Google’s Machine Learning Crash Course module on overfitting uses the same idea of comparing loss curves. To get trustworthy validation scores, use cross-validation rather than a single lucky split.

How to fix high bias

Use a more expressive model (for example, gradient-boosted trees instead of a linear model), add better features through feature engineering, reduce regularization strength, or train longer if the model has not converged. Adding more data rarely fixes high bias on its own, because the model cannot represent the pattern regardless of how many examples it sees.

How to fix high variance

Get more training data, simplify the model (shallower trees, fewer features), add or increase regularization, use early stopping, or use ensembles. Random forests are a classic variance-reduction trick: averaging many high-variance trees produces a much more stable prediction. Most of these levers are hyperparameters, which is why tuning them with a validation set is so central to the workflow.

Does the tradeoff still apply to deep learning and LLMs?

Mostly, with an important nuance. Very large neural networks can have far more parameters than training examples and still generalize well, a behavior researchers describe as “double descent,” where test error can drop again after models become large enough. That does not make the tradeoff useless: in practice, teams still watch training vs validation loss, use regularization techniques like dropout and weight decay, and stop training early when validation performance stops improving. When you fine-tune a large language model on a small dataset, overfitting to that dataset is a very real risk, so the same diagnostic habits apply.

A quick example

Suppose you are predicting house prices. A linear model with only square footage gives similar, mediocre error on training and validation data: high bias. A decision tree with no depth limit nails the training data but misses badly on new houses: high variance. A depth-limited gradient-boosting model with tuned learning rate and number of trees lands in between, with validation error close to training error and lower than both extremes. That middle ground is the tradeoff in action.

FAQ

Is bias in the bias-variance tradeoff the same as algorithmic bias?

No. Here, bias is a statistical term for systematic prediction error caused by model assumptions. Algorithmic or social bias refers to unfair outcomes for groups of people, which is a separate (and also important) topic.

Can you have low bias and low variance at the same time?

Yes, that is the goal. Better features, more data, and well-tuned models can lower both. The tradeoff describes what happens when you change model complexity while everything else stays fixed.

Which matters more, bias or variance?

Neither by default. What matters is total error on new data. Use validation results to decide which one is currently limiting your model.

Keep learning

Understanding bias and variance helps you debug models instead of guessing. If you want to build that skill hands-on:

• AI Engineering Course — train, evaluate, and deploy machine learning and AI models with sound validation habits.

• Data Analytics Course — build the statistics and data foundations that make model evaluation make sense.