DEEP LEARNING CAPSTONE · PARTS 1 TO 8

Before
optimization

One baseline and three depths, all trained on the same split with the same seed. Depth moved best validation accuracy by 0.76 points and widened the generalization gap by 17.75. The problem was never capacity.

52.22%baseline validation accuracy, 1 × 512
10%random-guess floor the baseline had to beat
0.76 ptsspread across all three depths, 38 images
0.15 to 0.33generalization gap, shallow to deep
One rule shapes everything: No CNN, no Conv2D, no pretrained weights, no transfer learning.

Every image must be flattened before it reaches the first layer, so 32 × 32 × 3 becomes 3,072 × 1. That single decision sets the ceiling for every experiment that follows.

The data and the splitParts 1 and 2

60,000Images, 50,000 train + 10,000 test
32 × 32 × 3Image shape, flattened to 3,072 inputs
10Classes, 6,000 images each, perfectly balanced
10%Random-guess floor, the number every model must beat

Split once, shared by every experiment

seed 42, indices committed to splits.npz
X_train(45000, 32, 32, 3)training images, pixels scaled to [0, 1]
X_val(5000, 32, 32, 3)stratified, 500 per class, drives every decision
X_test(10000, 32, 32, 3)sealed until Part 16, opened exactly once

The validation set is carved out of the training half only, stratified so every class contributes exactly 500 images. Nothing from the test set influences any architecture or hyperparameter decision.

Examples per class (Train vs Test)

Part 1 · Original Notebook Plot
Examples per class (Train vs Test)
Balanced distribution of 5,000 training and 1,000 test images for each of the 10 CIFAR-10 classes.

Raw pixel-intensity distribution (2,000 sample images)

Part 1 · Original Notebook Plot
Raw pixel-intensity distribution (2,000 sample images)
Distribution of unnormalized pixel intensities [0, 255] across 2,000 random training images (mean 120.71, std 64.15).

Normalized pixel-intensity distribution (2,000 sample images)

Part 2 · Original Notebook Plot
Normalized pixel-intensity distribution (2,000 sample images)
Distribution of scaled pixel values [0.0, 1.0] after division by 255.0 (mean 0.47, std 0.25).

One example of each of the 10 classes

Part 1 · Original Notebook Plot
One example of each of the 10 classes
First training example from each of the 10 CIFAR-10 classes illustrating visual diversity and low resolution.

15 random training images

Part 1 · Original Notebook Plot
15 random training images
Random 32×32 pixel images sampled from the training set with their corresponding class labels.

The baselinePart 3

1 hidden layer × 512 units

Best validation accuracy52.22%
Minimum validation loss1.3877
Final training accuracy67.46%
Trainable parameters1,578,506
OptimizerAdam @ 0.0001
Batch size / epochs128 / 50

About 5× the 10% random-guess floor, and 99.7% of those parameters sit in the very first layer, between the 3,072 inputs and the 512 hidden units.

Why these hyperparameters

512 neurons

Chosen by testing. At 32 units the model stalled at 19% accuracy with 31 of 32 hidden units dead. ReLU had collapsed the layer to a single effective neuron. 512 keeps the first hidden layer wide enough that it compresses the 3,072-value input only six-fold.

Learning rate 0.0001

The value the Part 9 sweep selects, adopted up front so every architecture comparison runs at the same setting the tuned model uses.

Batch size 128

Above the range swept in Part 10. At 45,000 training images it gives 352 weight updates per epoch.

50 epochs

Validation loss reaches its minimum near epoch 42. Extending to 100 epochs gained 0.4 points of validation accuracy while training accuracy gained 1.1. That is memorisation, not learning.

Baseline - 1 hidden layer, 128 units

Part 3 · Original Notebook Plot
Baseline - 1 hidden layer, 128 units
Training vs validation loss and accuracy curves for the initial 1-hidden-layer 128-unit baseline with min val loss marker.

Three depths, one widthParts 4 to 7, same data and seed, only depth changes

Shallow

1 × 512
Best val accuracy
52.22%
Gap
0.1524
Final train accuracy
67.46%
Parameters
1,578,506

Underfitting. Validation loss never meaningfully rises: it ends at 1.3896 against a minimum of 1.3877. Not memorising, just too small.

Medium

3 × 512
Selected for Parts 8 to 16
Best val accuracy
52.40%
Gap
0.2959
Final train accuracy
81.99%
Parameters
2,103,818

Good fit for 13 epochs, then overfitting. Validation loss bottoms at 1.4024 and rises 27% to 1.7835 while training loss keeps falling.

Deep

5 × 512
Best val accuracy
52.98%
Gap
0.3299
Final train accuracy
85.97%
Parameters
2,629,130

Strongest overfitting. Validation loss bottoms at epoch 12 then rises 74% to 2.4324, the largest rise of the three, despite the highest training accuracy.

Gap here is final train accuracy minus best validation accuracy, the definition Parts 4 to 7 report. The regularization experiments in tab 02 use a different one, and it is labelled there.

Validation accuracy by depth

50 epochs, from Results/*.json
36%41%45%50%54%11020304050Epoch
Shallow (1 × 512)Medium (3 × 512)Deep (5 × 512)

Validation loss by depth

lower is better, and only shallow stays down
1.301.611.912.212.5211020304050Epoch
Shallow (1 × 512)Medium (3 × 512)Deep (5 × 512)

Validation loss & accuracy - all architectures

Part 7 · Original Notebook Plot
Validation loss & accuracy - all architectures
Direct comparison of validation loss and validation accuracy across shallow, medium, and deep networks.

Depth bought overfitting, not accuracy. All three land within 38 images of each other on a 5,000-image validation set, where the standard error is about 0.7 points.

The diagnosisPart 8

Overfitting, beginning around epoch 13

Medium: 3 hidden layers of 512 neurons, 2,103,818 parameters

  1. Validation loss reaches its minimum of 1.4024 at epoch 13 and rises steadily afterwards: 1.402, 1.412, 1.522, 1.655 and 1.784 at epochs 13, 20, 30, 40 and 50.
  2. Training loss falls across all 50 epochs, from 1.845 to 0.542, and training accuracy climbs from 0.343 to 0.820.
  3. The gap widens from 0.0759 at epoch 13, the lowest-validation-loss epoch, to 0.2993 at epoch 50.
  4. Validation accuracy peaks at 0.5240 on epoch 30 and then falls back to 0.5206.

Not underfitting

The model reaches 0.820 training accuracy and its training loss is still falling at epoch 50, so it clearly has the capacity to keep fitting.

Not optimization instability

The curves are smooth: epoch-to-epoch validation loss varies by only 0.016, with a largest single jump of 0.089.

The fix carried into Part 15

Early Stopping on validation loss, with the best weights restored.

Medium: training against validation accuracy

the two curves separate and never rejoin
30%44%58%72%86%11020304050Epoch
Training accuracyValidation accuracy

Medium: training against validation loss

validation loss bottoms at epoch 13, then reverses
0.440.821.191.571.9511020304050Epoch
Training lossValidation loss

Medium Baseline - Accuracy

Part 8 · Original Notebook Plot
Medium Baseline - Accuracy
Training vs validation accuracy for the medium baseline showing divergence.

Medium Baseline - Loss

Part 8 · Original Notebook Plot
Medium Baseline - Loss
Training vs validation loss for the medium baseline showing overfitting after epoch 13.

Training loss falls for all 50 epochs. Validation loss bottoms at epoch 13 and ends 27.2% higher. Everything after epoch 13 is memorisation.