Set Trainable Layers For Resnet 18
Note: I wrote an app to classify an object using transfer learning, and the model was trained on CIFAR-10 dataset, all by myself.Here's the app:
Revisiting the Choice of Trainable Layers
Defining a Consistent Unfreezing Strategy
Now thatResNet-18 has been selected as the target backbone for transfer learning,
it is worth revisiting how I define and apply TRAINABLE_LAYERS.In practice, there are multiple ways to interpret this parameter. One approach is to count all architectural blocks regardless of whether they contain trainable parameters. Another, more principled approach is to consider only layers that actually include learnable weights and to skip structural or functional layers (e.g., pooling operations) that do not contribute parameters.
In this project, I adopt the latter strategy. This choice ensures that:
– Each increment in
TRAINABLE_LAYERS corresponds to a meaningful increase in model capacity– Fine-tuning depth is comparable across different architectures
– Reported results more accurately reflect the effect of optimization rather than implementation artifacts
Baseline: TRAINABLE_LAYERS = 3
Previously, withTRAINABLE_LAYERS = 3, the validation accuracies were:
– ResNet-50: 73.2%
– ResNet-34: 77.4%
– ResNet-18: 81.4%
This configuration served as our reference point for further exploration.
Reducing Fine-Tuning Depth: TRAINABLE_LAYERS = 2
I next reduced the fine-tuning depth by settingTRAINABLE_LAYERS = 2.
The corresponding training log is shown below:
# ====================== # Config # ====================== NUM_CLASSES = 10 TRAINING_SAMPLE_PER_CLASS = 100 VALIDATION_SAMPLE_PER_CLASS = 100 BATCH_SIZE = 256 EPOCHS = 20 LR = 1e-3 TRAINABLE_LAYERS = 2 ============================== Training model: resnet50 ============================== Epoch [01/20] Train Acc: 0.4250 | Val Acc: 0.3980 Epoch [02/20] Train Acc: 0.8440 | Val Acc: 0.6060 Epoch [03/20] Train Acc: 0.9320 | Val Acc: 0.6610 Epoch [04/20] Train Acc: 0.9650 | Val Acc: 0.7070 Epoch [05/20] Train Acc: 0.9730 | Val Acc: 0.6240 Epoch [06/20] Train Acc: 0.9890 | Val Acc: 0.7450 Epoch [07/20] Train Acc: 0.9830 | Val Acc: 0.7400 Epoch [08/20] Train Acc: 0.9940 | Val Acc: 0.7780 Epoch [09/20] Train Acc: 0.9890 | Val Acc: 0.7960 Epoch [10/20] Train Acc: 0.9910 | Val Acc: 0.7530 Epoch [11/20] Train Acc: 0.9980 | Val Acc: 0.7400 Epoch [12/20] Train Acc: 0.9910 | Val Acc: 0.7980 Epoch [13/20] Train Acc: 0.9940 | Val Acc: 0.7910 Epoch [14/20] Train Acc: 0.9930 | Val Acc: 0.7730 Epoch [15/20] Train Acc: 0.9930 | Val Acc: 0.7830 Epoch [16/20] Train Acc: 0.9920 | Val Acc: 0.7430 Epoch [17/20] Train Acc: 0.9940 | Val Acc: 0.7780 Epoch [18/20] Train Acc: 0.9940 | Val Acc: 0.7860 Epoch [19/20] Train Acc: 0.9980 | Val Acc: 0.7930 Epoch [20/20] Train Acc: 0.9950 | Val Acc: 0.7480 ============================== Training model: resnet34 ============================== Epoch [01/20] Train Acc: 0.5310 | Val Acc: 0.6040 Epoch [02/20] Train Acc: 0.8850 | Val Acc: 0.6530 Epoch [03/20] Train Acc: 0.9450 | Val Acc: 0.7550 Epoch [04/20] Train Acc: 0.9610 | Val Acc: 0.7280 Epoch [05/20] Train Acc: 0.9830 | Val Acc: 0.7480 Epoch [06/20] Train Acc: 0.9790 | Val Acc: 0.7850 Epoch [07/20] Train Acc: 0.9920 | Val Acc: 0.7460 Epoch [08/20] Train Acc: 0.9870 | Val Acc: 0.7270 Epoch [09/20] Train Acc: 0.9910 | Val Acc: 0.7670 Epoch [10/20] Train Acc: 0.9870 | Val Acc: 0.8060 Epoch [11/20] Train Acc: 0.9970 | Val Acc: 0.7790 Epoch [12/20] Train Acc: 0.9940 | Val Acc: 0.8030 Epoch [13/20] Train Acc: 0.9950 | Val Acc: 0.7880 Epoch [14/20] Train Acc: 0.9950 | Val Acc: 0.7900 Epoch [15/20] Train Acc: 0.9970 | Val Acc: 0.7730 Epoch [16/20] Train Acc: 0.9970 | Val Acc: 0.7870 Epoch [17/20] Train Acc: 0.9970 | Val Acc: 0.8160 Epoch [18/20] Train Acc: 0.9970 | Val Acc: 0.7960 Epoch [19/20] Train Acc: 0.9980 | Val Acc: 0.7880 Epoch [20/20] Train Acc: 1.0000 | Val Acc: 0.7940 ============================== Training model: resnet18 ============================== Epoch [01/20] Train Acc: 0.5010 | Val Acc: 0.5600 Epoch [02/20] Train Acc: 0.8570 | Val Acc: 0.5900 Epoch [03/20] Train Acc: 0.9250 | Val Acc: 0.7060 Epoch [04/20] Train Acc: 0.9760 | Val Acc: 0.7350 Epoch [05/20] Train Acc: 0.9830 | Val Acc: 0.7490 Epoch [06/20] Train Acc: 0.9900 | Val Acc: 0.7620 Epoch [07/20] Train Acc: 0.9950 | Val Acc: 0.7790 Epoch [08/20] Train Acc: 0.9980 | Val Acc: 0.7840 Epoch [09/20] Train Acc: 0.9970 | Val Acc: 0.7940 Epoch [10/20] Train Acc: 0.9980 | Val Acc: 0.8060 Epoch [11/20] Train Acc: 0.9990 | Val Acc: 0.8060 Epoch [12/20] Train Acc: 1.0000 | Val Acc: 0.8040 Epoch [13/20] Train Acc: 1.0000 | Val Acc: 0.8070 Epoch [14/20] Train Acc: 1.0000 | Val Acc: 0.7960 Epoch [15/20] Train Acc: 0.9980 | Val Acc: 0.7970 Epoch [16/20] Train Acc: 1.0000 | Val Acc: 0.8030 Epoch [17/20] Train Acc: 1.0000 | Val Acc: 0.8040 Epoch [18/20] Train Acc: 0.9990 | Val Acc: 0.8080 Epoch [19/20] Train Acc: 1.0000 | Val Acc: 0.8190 Epoch [20/20] Train Acc: 1.0000 | Val Acc: 0.8170
The resulting validation accuracies were:
– ResNet-50: 79.8%
– ResNet-34: 81.6%
– ResNet-18: 81.9%
Interestingly, this setting improved performance for deeper models and produced a marginal gain for
ResNet-18, suggesting that moderate fine-tuning can help control overfitting
while still adapting higher-level features to the target domain.
Minimal Fine-Tuning: TRAINABLE_LAYERS = 1
I then evaluated a highly constrained setup withTRAINABLE_LAYERS = 1,
where only the classifier head is trainable:
============================== Training model: resnet50 ============================== Epoch [01/20] Train Acc: 0.1600 | Val Acc: 0.3420 Epoch [02/20] Train Acc: 0.3320 | Val Acc: 0.4900 Epoch [03/20] Train Acc: 0.5340 | Val Acc: 0.6550 Epoch [04/20] Train Acc: 0.6640 | Val Acc: 0.6880 Epoch [05/20] Train Acc: 0.7100 | Val Acc: 0.6950 Epoch [06/20] Train Acc: 0.7450 | Val Acc: 0.7010 Epoch [07/20] Train Acc: 0.7420 | Val Acc: 0.7060 Epoch [08/20] Train Acc: 0.7670 | Val Acc: 0.7290 Epoch [09/20] Train Acc: 0.7760 | Val Acc: 0.7230 Epoch [10/20] Train Acc: 0.7750 | Val Acc: 0.7280 Epoch [11/20] Train Acc: 0.7960 | Val Acc: 0.7210 Epoch [12/20] Train Acc: 0.7930 | Val Acc: 0.7310 Epoch [13/20] Train Acc: 0.8000 | Val Acc: 0.7350 Epoch [14/20] Train Acc: 0.8160 | Val Acc: 0.7270 Epoch [15/20] Train Acc: 0.7990 | Val Acc: 0.7420 Epoch [16/20] Train Acc: 0.8120 | Val Acc: 0.7400 Epoch [17/20] Train Acc: 0.8190 | Val Acc: 0.7410 Epoch [18/20] Train Acc: 0.8110 | Val Acc: 0.7460 Epoch [19/20] Train Acc: 0.8330 | Val Acc: 0.7440 Epoch [20/20] Train Acc: 0.8330 | Val Acc: 0.7440 ============================== Training model: resnet34 ============================== Epoch [01/20] Train Acc: 0.0910 | Val Acc: 0.1540 Epoch [02/20] Train Acc: 0.2020 | Val Acc: 0.3080 Epoch [03/20] Train Acc: 0.3130 | Val Acc: 0.3960 Epoch [04/20] Train Acc: 0.4170 | Val Acc: 0.4660 Epoch [05/20] Train Acc: 0.5090 | Val Acc: 0.5240 Epoch [06/20] Train Acc: 0.5610 | Val Acc: 0.5630 Epoch [07/20] Train Acc: 0.5890 | Val Acc: 0.6090 Epoch [08/20] Train Acc: 0.6320 | Val Acc: 0.6330 Epoch [09/20] Train Acc: 0.6870 | Val Acc: 0.6570 Epoch [10/20] Train Acc: 0.7050 | Val Acc: 0.6680 Epoch [11/20] Train Acc: 0.7130 | Val Acc: 0.6780 Epoch [12/20] Train Acc: 0.7320 | Val Acc: 0.6950 Epoch [13/20] Train Acc: 0.7460 | Val Acc: 0.6820 Epoch [14/20] Train Acc: 0.7520 | Val Acc: 0.7030 Epoch [15/20] Train Acc: 0.7630 | Val Acc: 0.7090 Epoch [16/20] Train Acc: 0.7750 | Val Acc: 0.7120 Epoch [17/20] Train Acc: 0.7780 | Val Acc: 0.7140 Epoch [18/20] Train Acc: 0.7890 | Val Acc: 0.7230 Epoch [19/20] Train Acc: 0.7690 | Val Acc: 0.7180 Epoch [20/20] Train Acc: 0.7960 | Val Acc: 0.7210 ============================== Training model: resnet18 ============================== Epoch [01/20] Train Acc: 0.1280 | Val Acc: 0.1900 Epoch [02/20] Train Acc: 0.2090 | Val Acc: 0.2700 Epoch [03/20] Train Acc: 0.3320 | Val Acc: 0.3480 Epoch [04/20] Train Acc: 0.4510 | Val Acc: 0.4420 Epoch [05/20] Train Acc: 0.5300 | Val Acc: 0.5340 Epoch [06/20] Train Acc: 0.5700 | Val Acc: 0.5670 Epoch [07/20] Train Acc: 0.6280 | Val Acc: 0.6100 Epoch [08/20] Train Acc: 0.6650 | Val Acc: 0.6180 Epoch [09/20] Train Acc: 0.6880 | Val Acc: 0.6230 Epoch [10/20] Train Acc: 0.7220 | Val Acc: 0.6420 Epoch [11/20] Train Acc: 0.7170 | Val Acc: 0.6620 Epoch [12/20] Train Acc: 0.7330 | Val Acc: 0.6520 Epoch [13/20] Train Acc: 0.7440 | Val Acc: 0.6650 Epoch [14/20] Train Acc: 0.7530 | Val Acc: 0.6830 Epoch [15/20] Train Acc: 0.7670 | Val Acc: 0.6820 Epoch [16/20] Train Acc: 0.7710 | Val Acc: 0.6870 Epoch [17/20] Train Acc: 0.7640 | Val Acc: 0.6820 Epoch [18/20] Train Acc: 0.7890 | Val Acc: 0.6910 Epoch [19/20] Train Acc: 0.7760 | Val Acc: 0.6930 Epoch [20/20] Train Acc: 0.7570 | Val Acc: 0.6920
The validation accuracies dropped substantially:
– ResNet-50: 74.6%
– ResNet-34: 72.3%
– ResNet-18: 69.3%
This confirms that freezing too much of the network prevents sufficient feature adaptation, especially when the source and target domains are not perfectly aligned.
More Aggressive Fine-Tuning: TRAINABLE_LAYERS = 4
Finally, I explored a more aggressive setting withTRAINABLE_LAYERS = 4:
============================== Training model: resnet50 ============================== Epoch [01/20] Train Acc: 0.3670 | Val Acc: 0.1830 Epoch [02/20] Train Acc: 0.7580 | Val Acc: 0.4330 Epoch [03/20] Train Acc: 0.8430 | Val Acc: 0.3220 Epoch [04/20] Train Acc: 0.9010 | Val Acc: 0.4750 Epoch [05/20] Train Acc: 0.9120 | Val Acc: 0.5610 Epoch [06/20] Train Acc: 0.9500 | Val Acc: 0.6220 Epoch [07/20] Train Acc: 0.9640 | Val Acc: 0.6220 Epoch [08/20] Train Acc: 0.9680 | Val Acc: 0.6430 Epoch [09/20] Train Acc: 0.9770 | Val Acc: 0.6620 Epoch [10/20] Train Acc: 0.9770 | Val Acc: 0.6290 Epoch [11/20] Train Acc: 0.9820 | Val Acc: 0.6940 Epoch [12/20] Train Acc: 0.9800 | Val Acc: 0.6410 Epoch [13/20] Train Acc: 0.9850 | Val Acc: 0.6630 Epoch [14/20] Train Acc: 0.9860 | Val Acc: 0.6680 Epoch [15/20] Train Acc: 0.9880 | Val Acc: 0.7210 Epoch [16/20] Train Acc: 0.9910 | Val Acc: 0.7010 Epoch [17/20] Train Acc: 0.9850 | Val Acc: 0.7370 Epoch [18/20] Train Acc: 0.9850 | Val Acc: 0.7080 Epoch [19/20] Train Acc: 0.9900 | Val Acc: 0.7050 Epoch [20/20] Train Acc: 0.9840 | Val Acc: 0.6860 ============================== Training model: resnet34 ============================== Epoch [01/20] Train Acc: 0.4790 | Val Acc: 0.1240 Epoch [02/20] Train Acc: 0.8170 | Val Acc: 0.4420 Epoch [03/20] Train Acc: 0.9180 | Val Acc: 0.4350 Epoch [04/20] Train Acc: 0.9420 | Val Acc: 0.5310 Epoch [05/20] Train Acc: 0.9570 | Val Acc: 0.5730 Epoch [06/20] Train Acc: 0.9700 | Val Acc: 0.6290 Epoch [07/20] Train Acc: 0.9740 | Val Acc: 0.6660 Epoch [08/20] Train Acc: 0.9790 | Val Acc: 0.6110 Epoch [09/20] Train Acc: 0.9780 | Val Acc: 0.6470 Epoch [10/20] Train Acc: 0.9890 | Val Acc: 0.6250 Epoch [11/20] Train Acc: 0.9880 | Val Acc: 0.6990 Epoch [12/20] Train Acc: 0.9840 | Val Acc: 0.7120 Epoch [13/20] Train Acc: 0.9890 | Val Acc: 0.6540 Epoch [14/20] Train Acc: 0.9910 | Val Acc: 0.6430 Epoch [15/20] Train Acc: 0.9850 | Val Acc: 0.7400 Epoch [16/20] Train Acc: 0.9850 | Val Acc: 0.6790 Epoch [17/20] Train Acc: 0.9850 | Val Acc: 0.7200 Epoch [18/20] Train Acc: 0.9860 | Val Acc: 0.7380 Epoch [19/20] Train Acc: 0.9800 | Val Acc: 0.7120 Epoch [20/20] Train Acc: 0.9830 | Val Acc: 0.6940 ============================== Training model: resnet18 ============================== Epoch [01/20] Train Acc: 0.4930 | Val Acc: 0.3480 Epoch [02/20] Train Acc: 0.8380 | Val Acc: 0.4220 Epoch [03/20] Train Acc: 0.9060 | Val Acc: 0.6590 Epoch [04/20] Train Acc: 0.9710 | Val Acc: 0.6470 Epoch [05/20] Train Acc: 0.9730 | Val Acc: 0.6670 Epoch [06/20] Train Acc: 0.9860 | Val Acc: 0.7260 Epoch [07/20] Train Acc: 0.9930 | Val Acc: 0.7410 Epoch [08/20] Train Acc: 0.9940 | Val Acc: 0.7660 Epoch [09/20] Train Acc: 0.9960 | Val Acc: 0.7710 Epoch [10/20] Train Acc: 0.9950 | Val Acc: 0.7550 Epoch [11/20] Train Acc: 0.9990 | Val Acc: 0.7600 Epoch [12/20] Train Acc: 0.9990 | Val Acc: 0.7390 Epoch [13/20] Train Acc: 0.9990 | Val Acc: 0.7630 Epoch [14/20] Train Acc: 0.9990 | Val Acc: 0.7570 Epoch [15/20] Train Acc: 1.0000 | Val Acc: 0.7470 Epoch [16/20] Train Acc: 1.0000 | Val Acc: 0.7580 Epoch [17/20] Train Acc: 0.9990 | Val Acc: 0.7460 Epoch [18/20] Train Acc: 1.0000 | Val Acc: 0.7520 Epoch [19/20] Train Acc: 1.0000 | Val Acc: 0.7510 Epoch [20/20] Train Acc: 0.9980 | Val Acc: 0.7590
The results were:
– ResNet-50: 73.7%
– ResNet-34: 74.0%
– ResNet-18: 77.1%
Contrary to intuition, unfreezing additional layers led to degraded performance across all models, indicating that excessive fine-tuning introduces optimization instability and overfitting under our dataset constraints.
Summary of TRAINABLE_LAYERS Configurations
| TRAINABLE_LAYERS | Unfrozen Layers | ResNet-50 Val Acc |
ResNet-34 Val Acc |
ResNet-18 Val Acc |
|---|---|---|---|---|
| 1 | fc |
74.6% | 72.3% | 69.3% |
| 2 | layer4 + fc |
79.8% | 81.6% | 81.9% |
| 3 | layer3 + layer4 + fc |
73.2% | 77.4% | 81.4% |
| 4 | layer2 + layer3 + layer4 + fc |
73.7% | 74.0% | 77.1% |
Key Insights
Taken together, these experiments highlight a classic transfer learning trade-off:With too few trainable layers, the model lacks expressive power to adapt to the target task. With too many trainable layers, the model overfits and becomes harder to optimize reliably.
For
ResNet-18, TRAINABLE_LAYERS = 2 provides the optimal balance between
feature reuse and task-specific adaptation. It enables sufficient representational flexibility
while preserving the robustness of pretrained lower-level features.
As a result, I retain
TRAINABLE_LAYERS = 2 as the default configuration for all
subsequent experiments.
Final Decision on TRAINABLE_LAYERS
Based on the full set of experiments, I ultimately selectTRAINABLE_LAYERS = 2 as the final configuration.
Although TRAINABLE_LAYERS = 3 performed well in earlier runs,
TRAINABLE_LAYERS = 2 consistently produced the best overall results
across models, particularly for ResNet-18.To further validate this conclusion, I reran the experiment with
TRAINABLE_LAYERS = 2 and observed similar validation accuracies.
This confirmation step is important, as transfer learning performance can be
sensitive to randomness introduced by dataset sampling, data augmentation,
and weight initialization.The repeated results indicate that the observed performance gains are not artifacts of a single lucky run, but rather reflect a stable and reliable fine-tuning configuration. As a result,
TRAINABLE_LAYERS = 2
is adopted for all subsequent experiments.
Any comments? Feel free to participate below in the Facebook comment section.
