Practice exam
Three long problems. Each follows one network from start to finish: the forward pass, the gradients, then the updates, so every part builds on the previous one. There is no formula sheet. When a later part needs an earlier result, the question gives the value to reach (“show that…”), so you can carry on if you get stuck.
| Problem | Lectures | Points | Suggested time |
|---|---|---|---|
| 1 · A small CNN, end to end | 3, 4, 6 | 9 | 40 min |
| 2 · Designing a photo classifier | 4, 5, 6 | 5 | 20 min |
| 3 · Reading sequences | 4, 7 | 6 | 30 min |
- σ is the logistic sigmoid (slope parameter 1), tanh the hyperbolic tangent, ln the natural logarithm.
- Convolution as in the course: the filter slides over the input without being flipped, and each output is the sum of the element-wise products plus the filter’s bias.
- Error signal: the solutions write \(\delta\) for the derivative of the loss with respect to a neuron’s weighted input. The slides’ \(\delta\) has the opposite sign. Either convention is fine if you stick to it.
- Backpropagate with the weights before the update. Biases are never regularized.
- Keep 4 decimal places. Small rounding differences from the given values don’t matter.
A small CNN, end to end
A 3 × 3 single-band patch of a satellite image is classified into three land-cover classes: water, vegetation and urban. The network, in order:
- A convolution layer with two 2 × 2 filters \(K_1\) and \(K_2\), stride 1, no padding, biases \(b_1\) and \(b_2\).
- ReLU, then max-pooling with a 2 × 2 window and stride 2 on each feature map. The two pooled values form the feature vector \(f = (f_1, f_2)\).
- A hidden dense layer of 2 sigmoid neurons. Weight \(v_{ij}\) links \(f_j\) to hidden neuron \(i\), and \(a_i\) is its bias. Its outputs are \(h = (h_1, h_2)\).
- An output layer of 3 neurons with softmax. Weight \(w_{kj}\) links \(h_j\) to output \(k\) (\(k\) = water, veg, urban), \(c_k\) is its bias, \(z_k\) its weighted input (logit) and \(p_k\) its probability.
- The loss \(L\) is the cross-entropy with a one-hot target. This patch is vegetation.
1.1Architecture
- Give the size of each feature map before and after pooling.
- Count the trainable parameters of each layer and show that the network has 25 in total. Which operations have no parameters?
- Before pooling: \(\lfloor (3-2+0)/1\rfloor+1 = 2\), so each map is 2 × 2. After pooling: \(\lfloor (2-2)/2\rfloor+1 = 1\), so each map becomes a single value. Two maps give \(f=(f_1,f_2)\).
- \[ \begin{aligned} \text{conv} &: (2\cdot 2\cdot 1+1)\cdot 2 = 10\\ \text{hidden} &: 2\cdot 2 + 2 = 6\\ \text{output} &: 3\cdot 2 + 3 = 9\\ \text{total} &= 10+6+9 = \mathbf{25} \end{aligned} \] ReLU, max-pooling and softmax have no parameters: they are fixed functions.
1.2Convolution, ReLU, pooling
- Compute the two feature maps (before ReLU).
- Apply ReLU and max-pooling. Show that \(f = (2,\ 1)\), and say which position of each map wins the pooling. You will need it in 1.7.
- Slide each filter over the four 2 × 2 windows of \(X\). Top-left window \(\left(\begin{smallmatrix}1&2\\0&0\end{smallmatrix}\right)\), top-right \(\left(\begin{smallmatrix}2&1\\0&1\end{smallmatrix}\right)\), bottom-left \(\left(\begin{smallmatrix}0&0\\0&1\end{smallmatrix}\right)\), bottom-right \(\left(\begin{smallmatrix}0&1\\1&1\end{smallmatrix}\right)\). \[ \begin{aligned} \text{map 1, top-left} &= 1\cdot1 + 2\cdot(-1) + 0\cdot 0 + 0\cdot 1 + 0 = -1\\ \text{map 1, top-right} &= 2 - 1 + 0 + 1 = 2 \end{aligned} \] and so on, giving \[ M_1=\begin{pmatrix}-1&2\\1&0\end{pmatrix},\qquad M_2=\begin{pmatrix}1&-1\\-2&0\end{pmatrix} \] For \(M_2\), remember the bias \(b_2=-1\): top-left is \(0+2+0+0-1 = 1\).
- ReLU: \(\left(\begin{smallmatrix}0&2\\1&0\end{smallmatrix}\right)\) and \(\left(\begin{smallmatrix}1&0\\0&0\end{smallmatrix}\right)\). Pooling takes the maximum of each 2 × 2 map: \(f_1 = 2\), won by the top-right of map 1; \(f_2 = 1\), won by the top-left of map 2. So \(f=(2,1)\).
1.3Dense layers and the loss
- Compute the hidden outputs and show that \(h \approx (0.5000,\ 0.7311)\).
- Compute the logits and the softmax probabilities. Show that \(p_\text{veg} \approx 0.2016\). Which class does the network predict?
- Compute the loss and show that \(L \approx 1.6012\). Compare it with the loss of a network that answers uniformly at random, and comment.
- \[ \begin{aligned} u_1 &= 0.5\cdot 2 - 1.5\cdot 1 + 0.5 = 0 &&\Rightarrow\ h_1=\sigma(0)=0.5\\ u_2 &= 0.5\cdot 2 + 0.5\cdot 1 - 0.5 = 1 &&\Rightarrow\ h_2=\sigma(1)=0.7311 \end{aligned} \]
- \[ \begin{aligned} z_\text{water} &= 1\cdot 0.5 - 1\cdot 0.7311 = -0.2311\\ z_\text{veg} &= -1\cdot 0.5 + 0.5\cdot 0.7311 = -0.1345\\ z_\text{urban} &= 0.5\cdot 0.5 + 1\cdot 0.7311 = 0.9811 \end{aligned} \] Exponentials: 0.7937, 0.8742, 2.6673, sum 4.3351. So \[ p = (0.1831,\ 0.2016,\ 0.6153) \] The network predicts urban, which is wrong.
- With a one-hot target only the true class counts: \(L = -\ln p_\text{veg} = -\ln 0.2016 = 1.6012\). A uniform guess gives \(-\ln(1/3) = \ln 3 \approx 1.0986\). The network does worse than guessing because it is fairly confident in the wrong class, and cross-entropy punishes exactly that.
1.4The gradient of softmax with cross-entropy
- Starting only from the definitions of the softmax and of the cross-entropy, prove that \(\partial L/\partial z_k = p_k - y_k\) for every output \(k\), where \(y\) is the one-hot target. Treat the cases \(j = k\) and \(j \ne k\) separately when you differentiate the softmax.
- Give the numerical value of \(\partial L/\partial z_k\) for the three outputs, and explain the meaning of their signs.
- With \(p_k = e^{z_k}/\sum_m e^{z_m}\), the quotient rule gives \[ \frac{\partial p_k}{\partial z_j}=\begin{cases}p_k(1-p_k) & j=k\\ -p_k\,p_j & j\ne k\end{cases}\quad=\ p_k(\mathbf{1}_{j=k}-p_j) \] With \(L = -\sum_k y_k \ln p_k\): \[ \frac{\partial L}{\partial z_j} = -\sum_k \frac{y_k}{p_k}\,p_k(\mathbf{1}_{j=k}-p_j) = -y_j + p_j\sum_k y_k = p_j - y_j \] because the one-hot target sums to 1.
- \[ \frac{\partial L}{\partial z} = (0.1831 - 0,\ 0.2016 - 1,\ 0.6153 - 0) = (0.1831,\ -0.7984,\ 0.6153) \] A positive gradient means gradient descent will lower that logit (water and urban). The only negative one is the true class: its logit will be raised. The three values sum to 0.
1.5Output-layer gradients
- Compute the gradient of \(L\) with respect to the six weights \(w_{kj}\) and the three biases \(c_k\). Show that \(\partial L/\partial w_{\text{veg},2} \approx -0.5836\).
- Without computing anything, what would the gradients of the weights leaving \(h_j\) be if \(h_j\) were exactly 0? What does this tell you about how a weight learns?
- Each weight’s gradient is the error at the neuron it feeds times the input it carries: \(\partial L/\partial w_{kj} = (p_k-y_k)\,h_j\) and \(\partial L/\partial c_k = p_k - y_k\). \[ \frac{\partial L}{\partial W}=\begin{pmatrix}0.1831\cdot0.5 & 0.1831\cdot0.7311\\ -0.7984\cdot0.5 & -0.7984\cdot0.7311\\ 0.6153\cdot0.5 & 0.6153\cdot0.7311\end{pmatrix}=\begin{pmatrix}0.0915&0.1338\\-0.3992&\mathbf{-0.5836}\\0.3076&0.4498\end{pmatrix} \] \[ \frac{\partial L}{\partial c} = (0.1831,\ -0.7984,\ 0.6153) \]
- All zero. A weight’s gradient is proportional to the input it carries, so a weight whose input is 0 does not learn from that sample. This is why dead ReLU units, which always output 0, stop learning.
1.6Back into the hidden layer
- Compute \(\partial L/\partial h_1\) and \(\partial L/\partial h_2\). Show that \(\partial L/\partial h_1 \approx 1.2891\).
- Compute the error signals \(\delta_1\) and \(\delta_2\) of the hidden neurons. Show that \(\delta_1 \approx 0.3223\).
- Compute the gradients of the four weights \(v_{ij}\) and of the two biases \(a_i\).
- \(\delta_2\) is about 50 times smaller than \(\delta_1\). Give the two separate reasons, using the numbers above.
- Each hidden output reaches the loss through all three outputs, so its derivative sums over them, using the old weights: \[ \begin{aligned} \frac{\partial L}{\partial h_1} &= 1\cdot 0.1831 + (-1)(-0.7984) + 0.5\cdot 0.6153 = 1.2891\\ \frac{\partial L}{\partial h_2} &= (-1)\cdot 0.1831 + 0.5\cdot(-0.7984) + 1\cdot 0.6153 = 0.0330 \end{aligned} \]
- Multiply by the sigmoid’s slope, computed from the output, \(h(1-h)\): \[ \begin{aligned} \delta_1 &= 1.2891\cdot 0.5\cdot 0.5 = 1.2891\cdot 0.25 = 0.3223\\ \delta_2 &= 0.0330\cdot 0.7311\cdot 0.2689 = 0.0330\cdot 0.1966 = 0.00649 \end{aligned} \]
- \(\partial L/\partial v_{ij} = \delta_i\,f_j\) and \(\partial L/\partial a_i = \delta_i\): \[ \frac{\partial L}{\partial V}=\begin{pmatrix}0.3223\cdot2 & 0.3223\cdot1\\ 0.00649\cdot2 & 0.00649\cdot1\end{pmatrix}=\begin{pmatrix}0.6445&\mathbf{0.3223}\\0.0130&0.0065\end{pmatrix},\qquad \frac{\partial L}{\partial a}=(0.3223,\ 0.0065) \]
- (1) Cancellation: \(h_2\) feeds water, vegetation and urban with contributions −0.1831, −0.3992 and +0.6153, which nearly cancel. Raising \(h_2\) helps vegetation and water but helps urban even more, so on balance it hardly matters. (2) A flatter sigmoid: \(u_2 = 1\) is further from 0 than \(u_1 = 0\), so the slope is 0.1966 instead of the maximum 0.25.
1.7Back to the filters
- Compute \(\partial L/\partial f_1\) and \(\partial L/\partial f_2\). Show that \(\partial L/\partial f \approx (0.1644,\ -0.4802)\).
- Explain how the gradient passes through the max-pooling and through the ReLU. Which entries of each feature map receive a non-zero gradient?
- Compute the gradient of the four weights and of the bias of \(K_1\), then of \(K_2\). Why is the gradient of the bottom-left weight of \(K_1\) exactly zero?
- Here only one window contributes to each filter’s gradient. In general a filter is used at every position. How is the gradient of one filter weight formed then?
- \(f_j\) feeds both hidden neurons: \[ \begin{aligned} \frac{\partial L}{\partial f_1} &= v_{11}\delta_1 + v_{21}\delta_2 = 0.5\cdot0.3223 + 0.5\cdot0.00649 = 0.1644\\ \frac{\partial L}{\partial f_2} &= v_{12}\delta_1 + v_{22}\delta_2 = -1.5\cdot0.3223 + 0.5\cdot0.00649 = -0.4802 \end{aligned} \]
- Max-pooling sends the whole gradient to the position that won the maximum and 0 to the others: it has no weights, it only routes. ReLU passes the gradient unchanged where its input was positive and blocks it where the input was negative. The winners (2 and 1) are positive, so only top-right of map 1 receives \(0.1644\) and only top-left of map 2 receives \(-0.4802\).
- The winning output of map 1 is the dot product of \(K_1\) with the top-right window \(\left(\begin{smallmatrix}2&1\\0&1\end{smallmatrix}\right)\), so each weight’s gradient is \(0.1644\) times the pixel under it: \[ \frac{\partial L}{\partial K_1}=0.1644\begin{pmatrix}2&1\\0&1\end{pmatrix}=\begin{pmatrix}0.3288&0.1644\\0&0.1644\end{pmatrix},\quad \frac{\partial L}{\partial b_1}=0.1644 \] \[ \frac{\partial L}{\partial K_2}=-0.4802\begin{pmatrix}1&2\\0&0\end{pmatrix}=\begin{pmatrix}-0.4802&-0.9603\\0&0\end{pmatrix},\quad \frac{\partial L}{\partial b_2}=-0.4802 \] The bottom-left weight of \(K_1\) sat over a pixel equal to 0 in the winning window. As in 1.5(b), a weight with a zero input gets a zero gradient.
- Because the weights are shared, the filter weight is used once per position. Its gradient is the sum over all positions of (gradient arriving at that output) × (input pixel under that weight). This is the same idea as an RNN weight summing its gradient over time steps.
1.8Updates: gradient descent, momentum, regularization
We follow two weights: \(w_{\text{veg},2} = 0.5\) and \(v_{12} = -1.5\). Learning rate \(\eta = 0.5\).
- Give their values after one step of plain gradient descent.
- With momentum \(\alpha = 0.9\), the previous update of \(v_{12}\) was \(\Delta v_{12}(n-1) = -0.04\). Compute \(\Delta v_{12}(n)\) and the new \(v_{12}\).
- Now drop momentum and add L2 regularization: the training objective becomes the loss plus \(\tfrac{\lambda}{2}\) times the sum of the squared weights, with \(\lambda = 0.1\). Show that one gradient step first multiplies every weight by a constant factor, then applies the usual step. Give the factor and the new values of the two weights. What is this effect called?
- With L1 instead (same \(\lambda\)), consider a weight equal to 0.01 whose loss gradient is 0 on this sample. Compare one L1 step and one L2 step on it. Which penalty produces sparse weights, and why?
- \[ \begin{aligned} w_{\text{veg},2} &\leftarrow 0.5 - 0.5\cdot(-0.5836) = 0.7918\\ v_{12} &\leftarrow -1.5 - 0.5\cdot 0.3223 = -1.6611 \end{aligned} \] The vegetation logit gets more weight from \(h_2\), and \(v_{12}\) moves to lower \(h_1\), which mostly fed the wrong classes.
- \[ \Delta v_{12}(n) = 0.9\cdot(-0.04) - 0.5\cdot 0.3223 = -0.0360 - 0.1611 = -0.1971,\qquad v_{12} = -1.6971 \] The previous step pointed the same way, so momentum lengthens the step.
- The objective’s gradient is the loss gradient plus \(\lambda w\), so \[ w \leftarrow w - \eta\left(\frac{\partial L}{\partial w}+\lambda w\right) = (1-\eta\lambda)\,w - \eta\,\frac{\partial L}{\partial w},\qquad 1-\eta\lambda = 1 - 0.05 = 0.95 \] \[ \begin{aligned} w_{\text{veg},2} &\leftarrow 0.95\cdot 0.5 + 0.2918 = 0.7668\\ v_{12} &\leftarrow 0.95\cdot(-1.5) - 0.1611 = -1.5861 \end{aligned} \] This is weight decay: every weight shrinks by 5 % before the usual step. The biases are not touched.
- L1 adds \(\lambda\,\mathrm{sign}(w)\) to the gradient, a fixed push of \(\eta\lambda = 0.05\) whatever the weight’s size: \(0.01 - 0.05 = -0.04\), so it crosses 0. Implementations stop it at exactly 0 instead. L2 only scales it: \(0.95\cdot 0.01 = 0.0095\). The push does not shrink as the weight gets smaller, so it will never reach 0. L1 gives sparse weights (feature selection); L2 gives small ones.
1.9The same weight with Adam
We now train \(w_{\text{veg},2}\) (back at 0.5, no regularization) with Adam: \(\eta = 0.01\), \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\), both moment estimates starting at 0, with bias correction. Its first gradient is \(g_1 = -0.5836\) (part 1.5).
- Compute the first and second moment estimates, their bias-corrected values, the update and the new weight after step \(t = 1\).
- Show that, whatever the non-zero first gradient, the first Adam step has size almost exactly \(\eta\). How large would the first step be without bias correction?
- The next mini-batch gives \(g_2 = -0.05\). Compute step \(t = 2\). Show that the update is about \(+0.0073\), and give the new weight.
- Plain gradient descent with the same \(\eta = 0.01\) would move this weight by how much over these two steps? Comment on what Adam does differently.
- \[ \begin{aligned} m_1 &= 0.9\cdot 0 + 0.1\cdot(-0.5836) = -0.05836\\ v_1 &= 0.999\cdot 0 + 0.001\cdot 0.5836^2 = 3.4064\times10^{-4}\\ \hat m_1 &= m_1/(1-0.9) = -0.5836,\qquad \hat v_1 = v_1/(1-0.999) = 0.34064\\ \Delta w &= -\eta\,\frac{\hat m_1}{\sqrt{\hat v_1}+\epsilon} = -0.01\cdot\frac{-0.5836}{0.5836} = +0.0100\\ w_{\text{veg},2} &= 0.5100 \end{aligned} \]
- At \(t = 1\), the bias correction gives back exactly \(\hat m_1 = g_1\) and \(\hat v_1 = g_1^2\), so the step is \(\eta\,g_1/(|g_1|+\epsilon)\), which is \(\eta\) in size with the sign of \(g_1\), whatever \(|g_1|\) is. Without correction it would be \(\eta\cdot 0.1\,|g_1|/\sqrt{0.001\,g_1^2} = \eta\cdot 0.1/0.0316 = 3.16\,\eta \approx 0.0316\). The zero start would distort the first steps.
- \[ \begin{aligned} m_2 &= 0.9\cdot(-0.05836) + 0.1\cdot(-0.05) = -0.05753\\ v_2 &= 0.999\cdot 3.4064\times10^{-4} + 0.001\cdot 0.0025 = 3.4280\times10^{-4}\\ \hat m_2 &= -0.05753/(1-0.9^2) = -0.05753/0.19 = -0.30278\\ \hat v_2 &= 3.4280\times10^{-4}/(1-0.999^2) = 3.4280\times10^{-4}/0.001999 = 0.17148\\ \Delta w &= -0.01\cdot\frac{-0.30278}{\sqrt{0.17148}} = -0.01\cdot\frac{-0.30278}{0.41411} = +0.00731\\ w_{\text{veg},2} &= 0.51731 \end{aligned} \]
- Plain descent: \(0.01\cdot 0.5836 + 0.01\cdot 0.05 = 0.00584 + 0.0005 = 0.00634\) in total, against \(0.0173\) for Adam. The second gradient is ten times smaller, yet Adam’s step barely shrinks. Its running mean of past gradients (momentum) keeps the direction, and dividing by the root of the mean squared gradient gives each weight its own scale-free step size. Plain descent’s step is tied to the raw gradient size.
Designing a photo classifier
A recycling plant puts a camera above its conveyor belt to sort household waste into 5 classes: plastic, paper, glass, metal and organic. Each photo is resized to 96 × 96 pixels in colour (RGB). The proposed CNN:
| Layer | Specification |
|---|---|
| Input | 96 × 96 × 3 |
| Conv1 | 16 filters 5 × 5, stride 1, zero-padding 2, ReLU |
| Pool1 | max-pooling 2 × 2, stride 2 |
| Conv2 | 32 filters 3 × 3, stride 1, no padding, ReLU |
| Pool2 | max-pooling 2 × 2, stride 2 |
| Conv3 | 64 filters 3 × 3, stride 2, no padding, ReLU |
| Pool3 | max-pooling 2 × 2, stride 2 |
| Flatten | – |
| FC1 | 128 neurons, ReLU, dropout with rate 0.2 |
| FC2 | 5 neurons, softmax |
2.1Output volumes
- Give the output volume (height × width × depth) of every layer from Conv1 to Flatten. Show that the flattened vector has 1,600 components.
- Pool3 receives an 11 × 11 map. What happens to its last row and last column? How could the padding of Conv3 be changed so that nothing is lost, and what would that cost?
- Each step uses \(\lfloor (W - K + 2P)/S\rfloor + 1\). The depth after a convolution equals its number of filters, and pooling keeps the depth. \[ \begin{aligned} \text{Conv1} &: (96-5+4)/1+1 = 96 &&\to 96\times96\times16\\ \text{Pool1} &: (96-2)/2+1 = 48 &&\to 48\times48\times16\\ \text{Conv2} &: (48-3)/1+1 = 46 &&\to 46\times46\times32\\ \text{Pool2} &: (46-2)/2+1 = 23 &&\to 23\times23\times32\\ \text{Conv3} &: \lfloor(23-3)/2\rfloor+1 = 11 &&\to 11\times11\times64\\ \text{Pool3} &: \lfloor(11-2)/2\rfloor+1 = \lfloor 4.5\rfloor+1 = 5 &&\to 5\times5\times64\\ \text{Flatten} &: 5\cdot5\cdot64 = \mathbf{1600} \end{aligned} \]
- The five 2 × 2 windows cover rows and columns 1 to 10. Row 11 and column 11 never fall inside a window, so those values never reach the classifier: that is what the rounding down means. With padding 1 on Conv3, \(\lfloor(23-3+2)/2\rfloor+1 = 12\), so Pool3 gives 6 × 6 × 64 and covers everything. The cost: Flatten grows to 2,304, and FC1 to \(2304\cdot128+128 = 295{,}040\) parameters.
2.2Parameter count
- Count the trainable parameters of every layer and show that the network has 229,925 in total.
- Which layer holds most of them, and what fraction? Why do pooling, ReLU, flatten and dropout add none?
- A filter spans the full input depth, and each filter has one bias: \[ \begin{aligned} \text{Conv1} &: (5\cdot5\cdot3+1)\cdot16 = 76\cdot16 = 1{,}216\\ \text{Conv2} &: (3\cdot3\cdot16+1)\cdot32 = 145\cdot32 = 4{,}640\\ \text{Conv3} &: (3\cdot3\cdot32+1)\cdot64 = 289\cdot64 = 18{,}496\\ \text{FC1} &: 1600\cdot128+128 = 204{,}928\\ \text{FC2} &: 128\cdot5+5 = 645\\ \text{total} &= \mathbf{229{,}925} \end{aligned} \]
- FC1: \(204{,}928/229{,}925 \approx 89\%\). The three convolutions together hold only 24,352 (about 11 %). Pooling, ReLU, flatten and dropout are fixed operations (a max, a threshold, a reshape, a random mask), so they have nothing to learn.
2.3Why convolution?
- Suppose Conv1 were replaced by a fully connected layer producing the same 96 × 96 × 16 output from the same input. How many weights would it have (biases excluded)? Compare with Conv1.
- How many positions is each Conv1 filter applied at, so how many times is each of its weights reused per photo?
- Name the two properties of images that justify replacing the dense layer by a convolution.
- Inputs: \(96\cdot96\cdot3 = 27{,}648\). Outputs: \(96\cdot96\cdot16 = 147{,}456\). Weights: \(27{,}648\cdot147{,}456 \approx 4.08\times10^{9}\), about 3.4 million times Conv1’s 1,216 parameters.
- The output is 96 × 96, so each filter is applied at \(9{,}216\) positions, and each of its weights is reused 9,216 times per photo.
- (1) Patterns are small compared with the image (an edge, a bottle cap, a label corner), so a neuron only needs a small receptive field (local connectivity). (2) The same pattern can appear anywhere on the belt, so one detector can be reused at every position (weight sharing).
2.4How much data?
- Using the course’s rule of thumb that links training-set size, number of free parameters and tolerated test error, how many labelled photos would this network need for a 10 % test error?
- The plant has 15,000 labelled photos. Conclude, and propose two remedies from the course: one acting on the data, one on the model.
- \(N \approx W/\varepsilon = 229{,}925/0.1 \approx 2.3\) million photos.
- 15,000 is about 150 times too few: the network will overfit. Data: augmentation. Objects on a belt have no fixed orientation, so rotations and flips keep the class, and so do small changes of scale or brightness. Adding noise is another option. Model: shrink FC1 (it holds 89 % of the parameters), keep dropout and add L2 weight decay, or start from a network pre-trained on a large photo dataset (transfer learning). Early stopping on a validation set also helps.
2.5“It has no parameters, so it doesn’t matter”
An intern argues: “Pool1 has no parameters, so removing it cannot change the size of the model.”
- Recompute the volumes from Conv2 to Flatten without Pool1.
- Give the new total number of parameters. Was the intern right? Explain which layer changes and why.
- Conv2 now sees 96 × 96 × 16: \((96-3)+1 = 94 \to 94\times94\times32\). Pool2: \(47\times47\times32\). Conv3: \(\lfloor(47-3)/2\rfloor+1 = 23 \to 23\times23\times64\). Pool3: \(\lfloor(23-2)/2\rfloor+1 = 11 \to 11\times11\times64\). Flatten: \(7{,}744\).
- FC1 becomes \(7744\cdot128+128 = 991{,}360\), and the total becomes \(1{,}216+4{,}640+18{,}496+991{,}360+645 = 1{,}016{,}357\), about 4.4 times larger. The intern is wrong. A convolution’s parameter count depends only on the filter size, the input depth and the number of filters, so Conv2 and Conv3 do not change. A dense layer depends on how many values it receives, and pooling is what shrinks that number. Every convolution after it also runs at about 4 times more positions, so the computation grows too.
2.6Reusing the trained network
The network (with Pool1) is trained. Two new projects want to reuse it. A: another plant wants to separate three kinds of plastic from the same kind of camera photos, with 400 labelled photos. B: a hospital wants to classify grayscale chest X-rays as normal or pneumonia, with 300 labelled scans.
- For project A, what do you replace, and how many parameters are trainable if everything else is frozen?
- For project B, give two reasons why reusing this network is much harder, and say what you would do.
- Replace FC2 by a new 3-neuron softmax layer: \(128\cdot3+3 = 387\) trainable parameters. The other \(229{,}925 - 645 = 229{,}280\) are frozen: they still run in the forward pass but are not updated. Same kind of input, a closely related task and very little data: this is the case for freezing everything else.
- (1) Different input: an X-ray has 1 channel, but Conv1’s filters have depth 3. Either copy the grayscale image into the three channels or replace and retrain Conv1. (2) Very different task: the high-level features of waste photos (bottle shapes, labels) do not help with lungs. That is domain mismatch, with a risk of negative transfer. Keep only the early layers, because edges and textures transfer best. Remove the later layers (Conv3, FC1, FC2), add new ones, and with 300 scans train only those. Better still, start from a network pre-trained on a much larger and more general dataset.
2.7Dropout at test time
Use the slides’ version of dropout, with no rescaling during training. For some photo, all 128 FC1 outputs equal 1.5, and every weight from FC1 to the output neuron “glass” equals 0.05 (ignore its bias).
- What is the expected weighted input of “glass” during training? What is it at test time without any correction?
- What correction makes the two match? What would you get by multiplying by the dropout rate instead? What does inverted dropout do?
- During training each FC1 neuron is kept with probability \(1-0.2 = 0.8\): expected input \(128\cdot0.8\cdot1.5\cdot0.05 = 7.68\). At test time nothing is dropped: \(128\cdot1.5\cdot0.05 = 9.6\), which is 25 % more than the network saw in training.
- Multiply the FC1 outputs at test time by the keep probability 0.8, which gives back 7.68. Multiplying by the rate 0.2 would give 1.92, far too small. The slides’ wording (“scaled by the dropout rate”) only works when the rate is 0.5. Inverted dropout divides the kept outputs by 0.8 during training instead, so nothing changes at test time.
Reading sequences
Part A · A vanilla RNN for sentiment
A sentiment classifier reads short product reviews word by word. To keep the arithmetic small, each word is a single number (a 1-dimensional embedding). The review “great but slow” gives \(x = (x_1, x_2, x_3) = (1,\ 0.5,\ -1)\). Its label is negative: \(t = 0\).
The RNN has hidden size 1, activation tanh, initial state \(a_0 = 0\), recurrent weight \(w_{aa} = 0.6\), input weight \(w_{ax} = 1\) and bias \(b_a = 0\). Only the last step has an output: one sigmoid neuron with weight \(w_{ya} = 1.5\) and bias \(b_y = 0.8\) on \(a_3\), giving \(\hat y\), the probability that the review is positive. The loss \(J\) is the binary cross-entropy.
3.1Forward pass
- Which type of RNN is this, in the course’s classification? How many trainable parameters does it have, and how would that number change for a 40-word review?
- Compute \(a_1, a_2, a_3\). Show that \(a_3 \approx -0.5037\).
- Compute \(\hat y\) and the loss \(J\). Is the review classified correctly (threshold 0.5)?
- Many to one: a sequence in, one label out. Sentiment analysis is the slides’ own example. Five parameters: \(w_{aa}, w_{ax}, b_a, w_{ya}, b_y\). The same weights are reused at every step, so a 40-word review still has 5.
- \[ \begin{aligned} a_1 &= \tanh(0.6\cdot0 + 1) = \tanh(1) = 0.7616\\ a_2 &= \tanh(0.6\cdot0.7616 + 0.5) = \tanh(0.9570) = 0.7429\\ a_3 &= \tanh(0.6\cdot0.7429 - 1) = \tanh(-0.5543) = -0.5037 \end{aligned} \]
- \(\hat y = \sigma(1.5\cdot(-0.5037) + 0.8) = \sigma(0.0445) = 0.5111\), so the review is classified as positive, by a hair: wrong. With \(t = 0\), \(J = -\ln(1-\hat y) = -\ln 0.4889 = 0.7156\).
3.2Backpropagation through time
- Prove that the derivative of \(J\) with respect to the output neuron’s weighted input equals \(\hat y - t\), for either value of \(t\). Deduce \(\partial J/\partial w_{ya}\) and \(\partial J/\partial b_y\).
- Show that \(\partial J/\partial a_3 \approx 0.7667\).
- Let \(d_t\) be the derivative of \(J\) with respect to the weighted input of \(a_t\) (the argument of tanh). Compute \(d_3\), \(d_2\) and \(d_1\).
- Compute \(\partial J/\partial w_{aa}\), \(\partial J/\partial w_{ax}\) and \(\partial J/\partial b_a\). Explain why each is a sum over the time steps.
- With \(J = -[t\ln\hat y + (1-t)\ln(1-\hat y)]\) and \(\hat y=\sigma(z)\): \[ \frac{\partial J}{\partial \hat y} = -\frac{t}{\hat y}+\frac{1-t}{1-\hat y} = \frac{\hat y - t}{\hat y(1-\hat y)},\qquad \frac{\partial \hat y}{\partial z}=\hat y(1-\hat y)\ \Rightarrow\ \frac{\partial J}{\partial z} = \hat y - t \] This is the two-class version of 1.4. Numerically \(\hat y - t = 0.5111\), so \(\partial J/\partial w_{ya} = 0.5111\cdot(-0.5037) = -0.2574\) and \(\partial J/\partial b_y = 0.5111\).
- \(\partial J/\partial a_3 = (\hat y - t)\,w_{ya} = 0.5111\cdot 1.5 = 0.7667\).
- The derivative of tanh is \(1-a^2\): \(1-a_3^2 = 0.7463\), \(1-a_2^2 = 0.4481\), \(1-a_1^2 = 0.4200\). Going back one step multiplies by \(w_{aa}\) and by the slope there: \[ \begin{aligned} d_3 &= 0.7667\cdot 0.7463 = 0.5722\\ d_2 &= d_3\cdot w_{aa}\cdot(1-a_2^2) = 0.5722\cdot0.6\cdot0.4481 = 0.1538\\ d_1 &= d_2\cdot w_{aa}\cdot(1-a_1^2) = 0.1538\cdot0.6\cdot0.4200 = 0.0388 \end{aligned} \]
- The same weight is used at every step, so its gradient adds up one contribution per step: \(d_t\) times the input that weight carried at step \(t\). \[ \begin{aligned} \frac{\partial J}{\partial w_{aa}} &= d_3a_2 + d_2a_1 + d_1a_0 = 0.4251 + 0.1171 + 0 = 0.5422\\ \frac{\partial J}{\partial w_{ax}} &= d_3x_3 + d_2x_2 + d_1x_1 = -0.5722 + 0.0769 + 0.0388 = -0.4565\\ \frac{\partial J}{\partial b_a} &= d_3 + d_2 + d_1 = 0.7647 \end{aligned} \] Notice how the contributions shrink going back in time (0.57, 0.15, 0.04): the first word, “great”, barely influences the update. That is 3.3 in miniature.
3.3Vanishing gradients
- Compute \(\partial a_3/\partial a_1\).
- Show that, for this network and whatever the inputs, \(|\partial a_T/\partial a_1| \le 0.6^{\,T-1}\). Evaluate it for a 30-word review and explain the consequence, for instance when the first word is “not”.
- What could happen instead if \(w_{aa} = 3\)?
- \(\partial a_3/\partial a_1 = [w_{aa}(1-a_3^2)]\cdot[w_{aa}(1-a_2^2)] = (0.6\cdot0.7463)(0.6\cdot0.4481) = 0.4478\cdot0.2688 = 0.1204\).
- Each factor is \(w_{aa}(1-a_t^2)\) with \(0 < 1-a_t^2 \le 1\), so its size is at most 0.6. The chain from \(a_1\) to \(a_T\) has \(T-1\) factors, hence the bound. For \(T = 30\): \(0.6^{29} \approx 3.7\times10^{-7}\). The error at the end of the review sends almost nothing back to the first word. A leading “not” that flips the meaning cannot be learned: the network fails at long-term dependencies.
- Factors can then be as large as 3 (as long as tanh is not saturated). Their product can grow exponentially, so the gradients explode, the weights change wildly and training diverges. When tanh saturates, the factors fall back below 1, so the same network can swing between both problems.
Part B · An LSTM that must remember, then forget
A language model has to remember that the subject of the sentence is plural, to write “are” rather than “is” several words later. This fact is stored in the cell state of an LSTM cell with hidden size 1, which starts with \(h_0 = 0\) and \(C_0 = 1.5\). It reads \(x_1 = 0.5\) (another word of the sentence), then \(x_2 = -1\) (a full stop, which ends the sentence). Each gate looks at \(h_{t-1}\) and \(x_t\):
| Gate | Activation | Weight on \(h_{t-1}\) | Weight on \(x_t\) | Bias |
|---|---|---|---|---|
| forget \(f\) | σ | 0.5 | 2 | 0.5 |
| input \(i\) | σ | 1 | 1 | −1 |
| candidate \(\tilde C\) | tanh | −1 | 1 | 0 |
| output \(o\) | σ | 1 | 0.5 | 0.5 |
3.4Two steps of an LSTM
- Step 1: compute the four gate values, then \(C_1\) and \(h_1\). Show that \(C_1 \approx 1.4008\) and \(h_1 \approx 0.6014\).
- Step 2: compute \(C_2\) and \(h_2\). What happened to the stored fact, which gate did it, and is that what we want here?
- Compute the derivative of \(C_2\) with respect to \(C_0\) along the cell-state path only, and compare it with \(\partial a_3/\partial a_1\) from 3.3. The LSTM is barely better on these two steps. Explain why, and what still makes its memory fundamentally different from the RNN’s.
- With \(h_0 = 0\), only the input weights and biases matter: \[ \begin{aligned} f_1 &= \sigma(2\cdot0.5+0.5) = \sigma(1.5) = 0.8176, & i_1 &= \sigma(0.5-1) = \sigma(-0.5) = 0.3775\\ \tilde C_1 &= \tanh(0.5) = 0.4621, & o_1 &= \sigma(0.25+0.5) = \sigma(0.75) = 0.6792\\ C_1 &= f_1C_0 + i_1\tilde C_1 = 0.8176\cdot1.5 + 0.3775\cdot0.4621 = 1.4008\\ h_1 &= o_1\tanh(C_1) = 0.6792\cdot0.8855 = 0.6014 \end{aligned} \]
- \[ \begin{aligned} f_2 &= \sigma(0.5\cdot0.6014 - 2 + 0.5) = \sigma(-1.1993) = 0.2316, & i_2 &= \sigma(0.6014 - 1 - 1) = \sigma(-1.3986) = 0.1980\\ \tilde C_2 &= \tanh(-0.6014 - 1) = \tanh(-1.6014) = -0.9219, & o_2 &= \sigma(0.6014 - 0.5 + 0.5) = \sigma(0.6014) = 0.6460\\ C_2 &= 0.2316\cdot1.4008 + 0.1980\cdot(-0.9219) = 0.3244 - 0.1826 = 0.1419\\ h_2 &= 0.6460\cdot\tanh(0.1419) = 0.6460\cdot0.1409 = 0.0910 \end{aligned} \] The memory has been wiped: \(C\) drops from 1.40 to 0.14. The forget gate closed (0.23) because the input was the full stop. That is what we want: the sentence is over, so the subject’s number no longer matters.
- Along the cell state, each step only multiplies by the forget gate: \(\partial C_2/\partial C_0 = f_1f_2 = 0.8176\cdot0.2316 = 0.1894\), against 0.1204 for the RNN. It is barely better because the gate chose to forget at step 2. At step 1, where the fact was still needed, the factor was 0.82, against 0.27 to 0.45 per step for the RNN. The difference: the RNN’s factor \(w_{aa}(1-a^2)\) is set by one fixed weight and the tanh slope, so it can never exceed 0.6 here, whatever the data. The LSTM’s factor is recomputed from the input at every step, and the cell is updated by addition rather than squashed through a weight and a tanh. The network can therefore hold it near 1 for as long as the information is needed, and drop it when it is not.
3.5Real-size recurrent layers
- A real model uses word embeddings of dimension 50 and hidden size 100. Show that an LSTM layer has 60,400 parameters (biases included) and that this is exactly four times a vanilla RNN layer of the same size.
- How many would a GRU layer have, biases included?
- A vanilla layer: \(100\cdot100\) recurrent weights \(+\ 100\cdot50\) input weights \(+\ 100\) biases \(= 10{,}000+5{,}000+100 = 15{,}100\). An LSTM has four such blocks (forget, input, candidate, output), each looking at \([h_{t-1}, x_t]\): \(4\cdot15{,}100 = 60{,}400\).
- Three blocks (update, reset, candidate): \(3\cdot15{,}100 = 45{,}300\). The slide’s GRU equations have no biases, which would give \(3\cdot15{,}000 = 45{,}000\).
Part C · Attention in translation
A translation model reads the French sentence “le chat dort”. Its encoder produced \(h_1 = (1, 1, 0)\) for “le”, \(h_2 = (0, 2, 1)\) for “chat” and \(h_3 = (1, 0, 2)\) for “dort”. The decoder has already written “the”, and its current state is \(s = (0, 1, 0.5)\). The score is the dot product.
3.6Attention weights and context
- Compute the three scores, the attention weights and the context vector. Show that \(\alpha_2 \approx 0.6914\). Which French word is the decoder looking at, and which English word is it probably about to write?
- Recompute the weights with the scores divided by \(\sqrt 3\). What changes, and why do Transformers do this?
- Why does attention help with long sentences, compared with a plain encoder–decoder?
- \[ \begin{aligned} e &= (s\cdot h_1,\ s\cdot h_2,\ s\cdot h_3) = (0+1+0,\ 0+2+0.5,\ 0+0+1) = (1,\ 2.5,\ 1)\\ e^{e} &= (2.7183,\ 12.1825,\ 2.7183),\quad \text{sum } 17.6191\\ \alpha &= (0.1543,\ 0.6914,\ 0.1543)\\ c &= 0.1543\,h_1 + 0.6914\,h_2 + 0.1543\,h_3 = (0.3086,\ 1.5372,\ 1.0000) \end{aligned} \] The decoder looks mostly at “chat”, so it is about to write “cat”. The softmax is over the scores of the one decoder state against all encoder states, and the context is a weighted sum of the encoder states.
- Scaled scores \((0.5774,\ 1.4434,\ 0.5774)\) give \(\alpha = (0.2284,\ 0.5431,\ 0.2284)\): flatter weights. Dot products grow with the dimension \(d\), and large scores push the softmax towards one-hot weights with tiny gradients. Dividing by \(\sqrt d\) keeps it in a range where it still learns.
- A plain encoder–decoder squeezes the whole sentence into one fixed-size context vector, a bottleneck that loses information on long inputs. With attention, the decoder builds a new context at every output word and can look straight at the relevant source words, however far back they are. That is how it aligns “cat” with “chat”.