Gradient Descent and Overfitting: Machine Learning's Two Core Ideas, Worked Out
Most of machine learning rests on two ideas: learn by repeatedly reducing an error, and don't memorize your training data. Both are easier than the textbooks make them look.
Gradient descent in one picture
You have a function that measures your model's error, called the loss. You want the inputs (the model's weights) that make the loss as small as possible. Gradient descent says: look at the slope where you stand, and take a small step downhill.
Update rule:
w_new = w_old − η × (slope at w_old)
where η (eta) is the learning rate, the size of each step.
A worked example by hand
Take the simplest possible loss: f(w) = (w − 3)². Its minimum is obviously at w = 3, but pretend we do not know that.
The slope (derivative) is f′(w) = 2(w − 3).
Start at w = 0 with learning rate η = 0.1.
| Step | w | Slope 2(w − 3) | New w = w − 0.1 × slope |
|---|---|---|---|
| 0 | 0 | −6 | 0 + 0.6 = 0.6 |
| 1 | 0.6 | −4.8 | 0.6 + 0.48 = 1.08 |
| 2 | 1.08 | −3.84 | 1.08 + 0.384 = 1.464 |
| 3 | 1.464 | −3.072 | 1.464 + 0.3072 = 1.7712 |
| 4 | 1.7712 | −2.4576 | 1.7712 + 0.2458 = 2.017 |
Look at what happened: w climbs toward 3, and the steps shrink as it approaches, because the slope shrinks near the minimum. Nobody told it to slow down; the maths does it.
What a wrong learning rate does
Same function, same start (w = 0), but η = 1.1:
| Step | w |
|---|---|
| 0 | 0 |
| 1 | 6.6 |
| 2 | −1.32 |
| 3 | 8.184 |
Each step overshoots the minimum by more than the last. w bounces from side to side and the swings grow: it diverges. And if η is tiny, say 0.001, it converges, but so slowly that training never finishes. Picking the learning rate is a real engineering decision, not a detail.
Three flavours you will be asked about
| Type | Uses per update | Trade-off |
|---|---|---|
| Batch gradient descent | The whole dataset | Stable, but slow on big data |
| Stochastic (SGD) | One example | Fast and noisy; the noise can help escape poor spots |
| Mini-batch | A small group (32, 64…) | The practical default |
Overfitting: memorizing instead of learning
Imagine a student who memorizes last year's question paper word for word. Perfect score on last year's paper. Fail on this year's. A model can do the same.
- Underfitting: the model is too simple to capture the pattern. It does badly on training data and new data. (Fitting a straight line to curved data.)
- Overfitting: the model is so flexible that it learns the noise in the training data. It does very well on training data but badly on new data.
- Good fit: it does well on both, and the two scores are close.
How to spot it
Hold some data back as a validation set that the model never trains on. Then compare:
| Training accuracy | Validation accuracy | Diagnosis |
|---|---|---|
| Low | Low | Underfitting |
| High | Much lower | Overfitting |
| High | Similarly high | Good fit |
A widening gap between training and validation performance as training continues is the classic overfitting signature.
How to reduce it
- More data, the most reliable cure.
- A simpler model (fewer parameters, a shallower tree).
- Regularization, which adds a penalty for large weights so the model prefers simple explanations (L1 / L2).
- Early stopping: stop training when validation error starts rising.
- Dropout in neural networks: randomly switch off units during training.
- Cross-validation, to check your result is not a lucky split.
Two-line viva answers
- Gradient descent: repeatedly move the weights opposite to the gradient of the loss, in steps scaled by the learning rate, until the loss stops falling.
- Overfitting: the model fits the training data, noise included, so it fails to generalize to unseen data. You detect it by a gap between training and validation performance.
Sumantrak™ is an AI mentor built for Indian engineering, diploma, BCA and BSc CS/IT students — Machine Learning, Data Science, and the rest of your syllabus, explained like this. Free to start →