Deep Learning Fundamentals for GATE
Neural networks let the model discover the feature shape itself. GATE tests conceptual clarity — what each architecture assumes about data — more than training tricks.
Important concepts
- Perceptron, activations (sigmoid, ReLU, softmax), universal approximation idea.
- Backpropagation as chain rule + gradient descent; vanishing gradients.
- Optimisers: SGD with momentum, Adam; regularisation via dropout/weight decay.
- CNNs: local connectivity, sharing, pooling, translation structure.
- RNNs/LSTMs vs transformers and self-attention.
- Transfer learning and when pretraining wins.
GATE relevance
Activation choice, why depth helps, CNN parameter counting, attention intuition, and overfitting diagnosis (train loss down, validation loss up).
Important formulas
ŷ=σ(w·x+b) · softmax_i=e^zi/Σe^zj · Attention=softmax(QKᵀ/√d)V
Worked example
A 3×3 image with a 2×2 filter at stride 1 gives a 2×2 output; with 4 filters that is 2·2·4=16 activations from (2·2+1)·4=20 parameters — sharing is why CNNs scale.
Practice questions
Practice questions (Neural networks) + mock test. Theory: Guide Unit 7.
Related topics
Machine learning · Linear algebra · Statistics
FAQ
Why did transformers displace RNNs?
Self-attention gives constant path length plus full parallelism on GPUs — RNNs process tokens serially and lose long-range signal.
Train loss falls but validation rises — what now?
Classic overfitting: early stopping, dropout/weight decay, augmentation, smaller model, and check for distribution mismatch.