Overfitting vs Underfitting: Understanding the Balance in Machine Learning

As you started reading this blog from the title overfitting and underfitting I am guessing that you bit know what is overfitting and what is underfitting But what exactly do they mean, and why are they so important? Let’s make these concepts easy to understand with a simple example of two students
preparing for an exam.

Overfitting:

Ovvi’s Approach to Studying
Ovvi is determined to ace her exam, so she decides to memorize everything. She learns every question and answers word-for-word, including all the examples and practice problems. She even memorizes details that might not matter, just in case they come up on the test. Instead of focusing on understanding the concepts, Ovvi spends all her time memorizing the exact answers. When exam day came, Ovvi felt ready — but only for questions that matched exactly what she memorized. If a question is even slightly different, she struggles because she never fully understood the ideas behind the material. Ovvi put in a lot of hard work, but instead of truly understanding the material, she focused on memorizing everything. She’s well-prepared for the types of questions she’s already seen, but when faced with something new or different, she struggles because
she doesn’t take the time to grasp the underlying concepts or patterns. So, we can conclude that Ovvi’s approach is similar to overfitting in machine learning. Overfitting occurs when a model learns the training data (just like Ovvi memorizing her study materials) very well but fails to generalize. This means it struggles to handle new data or situations that it hasn’t encountered before. In Ovvi’s case, she was excellent with familiar questions but struggled with anything slightly different, just as an overfitted model fails on new or “test” data.

underfitting:

Uvvi’s Approach to Studying
Now let’s look at Uvvi’s approach. Uvvi is more relaxed and takes a casual attitude toward studying.
He glances through the notes, getting a general sense of the topics, questions, and answers, but
doesn’t spend much time studying. He skips the details, thinking a rough understanding will be
enough to get by.
When exam day comes, Uvvi realizes he can’t even answer the basic questions he saw in his notes
because he didn’t prepare. His lack of understanding and minimal practice means he struggles
with both familiar questions and new ones. Uvvi’s approach is a classic example of underfitting in
machine learning.

Conclusion

Uvvi’s approach represents underfitting. In machine learning, an underfit model is too simple to capture the important patterns in the data, which means it performs poorly on both the training data (what it has seen) and new data (what it hasn’t seen). Just like Uvvi didn’t put in enough effort to
truly understand the material, an underfit model lacks the complexity needed to grasp the data well enough to make accurate predictions.

The Balanced Approach
The Ideal Student Now imagine a third student who takes a balanced approach to studying. This student doesn’t just memorize but also works on understanding the core concepts. They spend enough time practicing to get comfortable with the material, but they don’t waste time on irrelevant
details. This balanced approach allows the student to handle both familiar and new types of questions with confidence.

Conclusion

This balanced approach represents a generalized model in machine learning. A generalized model performs well on both training data (seen data) and test data (unseen data). Just like a well-prepared student who can handle both familiar and unfamiliar exam questions, a generalized model adapts
well to new data and makes accurate predictions without overfitting or underfitting.

Definitions:
Overfitting: When a model performs very well on training data (seen data) but poorly on new data (test data). This is like a student who memorized but didn’t truly understand the material.
Underfitting: When a model performs poorly on both training and test data. This is similar to a student who didn’t study enough to understand the material at all.
Generalized Model: A model that performs well on both training and test data, effectively handling
new, unseen data without overfitting or underfitting.

Now understand some new terms:

High Bias: Bais refers to when a model makes too many assumptions about the data and doesn’t pay enough attention to details in the data, and as a result, it starts making mistakes because it’s too simple to understand complex Parten

Causes of underfitting:

1. Too simple
2. Not enough training time
3. Lack of relevant features
So High bias occurs when the model is too simple to adapt to the data, so performs poorly Now understand High Variance: if the model has high variance, it means the model is overfitting and High Variance refers to the model that is more focused on the noise which is not much relevant for the
model learning in the training data

Causes of Overfitting

1. Too complex model
2. Too many features
3. Small Training Dataset
So high Variance occurs when the model gets too sensitive to the training data and fails to generalize on unseen data