← Computer Science blog
Concept guide10 October 2026 · 4 min read

Gradient Descent and Overfitting: Machine Learning's Two Core Ideas, Worked Out

CSEITAI&DSR&AIAN

Most of machine learning rests on two ideas: learn by repeatedly reducing an error, and don't memorize your training data. Both are easier than the textbooks make them look.

Gradient descent in one picture

You have a function that measures your model's error, called the loss. You want the inputs (the model's weights) that make the loss as small as possible. Gradient descent says: look at the slope where you stand, and take a small step downhill.

Update rule:

w_new = w_old − η × (slope at w_old)

where η (eta) is the learning rate, the size of each step.

A worked example by hand

Take the simplest possible loss: f(w) = (w − 3)². Its minimum is obviously at w = 3, but pretend we do not know that.

The slope (derivative) is f′(w) = 2(w − 3).

Start at w = 0 with learning rate η = 0.1.

StepwSlope 2(w − 3)New w = w − 0.1 × slope
00−60 + 0.6 = 0.6
10.6−4.80.6 + 0.48 = 1.08
21.08−3.841.08 + 0.384 = 1.464
31.464−3.0721.464 + 0.3072 = 1.7712
41.7712−2.45761.7712 + 0.2458 = 2.017

Look at what happened: w climbs toward 3, and the steps shrink as it approaches, because the slope shrinks near the minimum. Nobody told it to slow down; the maths does it.

What a wrong learning rate does

Same function, same start (w = 0), but η = 1.1:

Stepw
00
16.6
2−1.32
38.184

Each step overshoots the minimum by more than the last. w bounces from side to side and the swings grow: it diverges. And if η is tiny, say 0.001, it converges, but so slowly that training never finishes. Picking the learning rate is a real engineering decision, not a detail.

Three flavours you will be asked about

TypeUses per updateTrade-off
Batch gradient descentThe whole datasetStable, but slow on big data
Stochastic (SGD)One exampleFast and noisy; the noise can help escape poor spots
Mini-batchA small group (32, 64…)The practical default

Overfitting: memorizing instead of learning

Imagine a student who memorizes last year's question paper word for word. Perfect score on last year's paper. Fail on this year's. A model can do the same.

  • Underfitting: the model is too simple to capture the pattern. It does badly on training data and new data. (Fitting a straight line to curved data.)
  • Overfitting: the model is so flexible that it learns the noise in the training data. It does very well on training data but badly on new data.
  • Good fit: it does well on both, and the two scores are close.

How to spot it

Hold some data back as a validation set that the model never trains on. Then compare:

Training accuracyValidation accuracyDiagnosis
LowLowUnderfitting
HighMuch lowerOverfitting
HighSimilarly highGood fit

A widening gap between training and validation performance as training continues is the classic overfitting signature.

How to reduce it

  1. More data, the most reliable cure.
  2. A simpler model (fewer parameters, a shallower tree).
  3. Regularization, which adds a penalty for large weights so the model prefers simple explanations (L1 / L2).
  4. Early stopping: stop training when validation error starts rising.
  5. Dropout in neural networks: randomly switch off units during training.
  6. Cross-validation, to check your result is not a lucky split.

Two-line viva answers

  • Gradient descent: repeatedly move the weights opposite to the gradient of the loss, in steps scaled by the learning rate, until the loss stops falling.
  • Overfitting: the model fits the training data, noise included, so it fails to generalize to unseen data. You detect it by a gap between training and validation performance.

Sumantrak™ is an AI mentor built for Indian engineering, diploma, BCA and BSc CS/IT students — Machine Learning, Data Science, and the rest of your syllabus, explained like this. Free to start →