MIAI 504 · Neural Networks · Practice exam
MIAI 504 · Neural Networks & Deep Learning · Final exam training

Practice exam

Three long problems. Each follows one network from start to finish: the forward pass, the gradients, then the updates, so every part builds on the previous one. There is no formula sheet. When a later part needs an earlier result, the question gives the value to reach (“show that…”), so you can carry on if you get stuck.

90 minDuration
20 ptsTotal
3Problems, 22 parts
CalculatorNon-programmable. No documents.
ProblemLecturesPointsSuggested time
1 · A small CNN, end to end3, 4, 6940 min
2 · Designing a photo classifier4, 5, 6520 min
3 · Reading sequences4, 7630 min
Conventions
  • σ is the logistic sigmoid (slope parameter 1), tanh the hyperbolic tangent, ln the natural logarithm.
  • Convolution as in the course: the filter slides over the input without being flipped, and each output is the sum of the element-wise products plus the filter’s bias.
  • Error signal: the solutions write \(\delta\) for the derivative of the loss with respect to a neuron’s weighted input. The slides’ \(\delta\) has the opposite sign. Either convention is fine if you stick to it.
  • Backpropagate with the weights before the update. Biases are never regularized.
  • Keep 4 decimal places. Small rounding differences from the given values don’t matter.
How to use this paper
  1. Work on paper, like the real exam. Start the timer when you begin; it keeps running if you close the tab.
  2. No solutions while you work. When you are done, submit at the end of the paper. The timer stops and the solutions to every part appear at once.
  3. Then mark yourself: full, half or zero points for each part. Your total out of 20 is at the end of the paper.
  4. Printing gives the paper without the solutions.
Problem 1 · 9 points · about 40 minutes

A small CNN, end to end

A 3 × 3 single-band patch of a satellite image is classified into three land-cover classes: water, vegetation and urban. The network, in order:

  1. A convolution layer with two 2 × 2 filters \(K_1\) and \(K_2\), stride 1, no padding, biases \(b_1\) and \(b_2\).
  2. ReLU, then max-pooling with a 2 × 2 window and stride 2 on each feature map. The two pooled values form the feature vector \(f = (f_1, f_2)\).
  3. A hidden dense layer of 2 sigmoid neurons. Weight \(v_{ij}\) links \(f_j\) to hidden neuron \(i\), and \(a_i\) is its bias. Its outputs are \(h = (h_1, h_2)\).
  4. An output layer of 3 neurons with softmax. Weight \(w_{kj}\) links \(h_j\) to output \(k\) (\(k\) = water, veg, urban), \(c_k\) is its bias, \(z_k\) its weighted input (logit) and \(p_k\) its probability.
  5. The loss \(L\) is the cross-entropy with a one-hot target. This patch is vegetation.
Network of Problem 1: a 3 by 3 input, two 2 by 2 feature maps, ReLU and max-pooling to f1 and f2, two sigmoid hidden neurons, three softmax outputs. INPUT X 121 001 011 K₁ K₂ MAP 1 MAP 2 ReLU max-pool f₁ f₂ V, a h₁ h₂ σ σ W, c z₁ z₂ z₃ softmax water vegetation urban
The feature maps are left empty: computing them is part 1.2.
\[ X=\begin{pmatrix}1&2&1\\0&0&1\\0&1&1\end{pmatrix} \]
\[ K_1=\begin{pmatrix}1&-1\\0&1\end{pmatrix},\ b_1=0 \]
\[ K_2=\begin{pmatrix}0&1\\1&-1\end{pmatrix},\ b_2=-1 \]
\[ V=\begin{pmatrix}v_{11}&v_{12}\\v_{21}&v_{22}\end{pmatrix}=\begin{pmatrix}0.5&-1.5\\0.5&0.5\end{pmatrix},\quad a=\begin{pmatrix}0.5\\-0.5\end{pmatrix} \]
\[ W=\begin{pmatrix}w_{\text{water},1}&w_{\text{water},2}\\w_{\text{veg},1}&w_{\text{veg},2}\\w_{\text{urban},1}&w_{\text{urban},2}\end{pmatrix}=\begin{pmatrix}1&-1\\-1&0.5\\0.5&1\end{pmatrix},\quad c=\begin{pmatrix}0\\0\\0\end{pmatrix} \]

1.1Architecture

  1. Give the size of each feature map before and after pooling.
  2. Count the trainable parameters of each layer and show that the network has 25 in total. Which operations have no parameters?

1.2Convolution, ReLU, pooling

  1. Compute the two feature maps (before ReLU).
  2. Apply ReLU and max-pooling. Show that \(f = (2,\ 1)\), and say which position of each map wins the pooling. You will need it in 1.7.

1.3Dense layers and the loss

  1. Compute the hidden outputs and show that \(h \approx (0.5000,\ 0.7311)\).
  2. Compute the logits and the softmax probabilities. Show that \(p_\text{veg} \approx 0.2016\). Which class does the network predict?
  3. Compute the loss and show that \(L \approx 1.6012\). Compare it with the loss of a network that answers uniformly at random, and comment.

1.4The gradient of softmax with cross-entropy

  1. Starting only from the definitions of the softmax and of the cross-entropy, prove that \(\partial L/\partial z_k = p_k - y_k\) for every output \(k\), where \(y\) is the one-hot target. Treat the cases \(j = k\) and \(j \ne k\) separately when you differentiate the softmax.
  2. Give the numerical value of \(\partial L/\partial z_k\) for the three outputs, and explain the meaning of their signs.

1.5Output-layer gradients

  1. Compute the gradient of \(L\) with respect to the six weights \(w_{kj}\) and the three biases \(c_k\). Show that \(\partial L/\partial w_{\text{veg},2} \approx -0.5836\).
  2. Without computing anything, what would the gradients of the weights leaving \(h_j\) be if \(h_j\) were exactly 0? What does this tell you about how a weight learns?

1.6Back into the hidden layer

  1. Compute \(\partial L/\partial h_1\) and \(\partial L/\partial h_2\). Show that \(\partial L/\partial h_1 \approx 1.2891\).
  2. Compute the error signals \(\delta_1\) and \(\delta_2\) of the hidden neurons. Show that \(\delta_1 \approx 0.3223\).
  3. Compute the gradients of the four weights \(v_{ij}\) and of the two biases \(a_i\).
  4. \(\delta_2\) is about 50 times smaller than \(\delta_1\). Give the two separate reasons, using the numbers above.

1.7Back to the filters

  1. Compute \(\partial L/\partial f_1\) and \(\partial L/\partial f_2\). Show that \(\partial L/\partial f \approx (0.1644,\ -0.4802)\).
  2. Explain how the gradient passes through the max-pooling and through the ReLU. Which entries of each feature map receive a non-zero gradient?
  3. Compute the gradient of the four weights and of the bias of \(K_1\), then of \(K_2\). Why is the gradient of the bottom-left weight of \(K_1\) exactly zero?
  4. Here only one window contributes to each filter’s gradient. In general a filter is used at every position. How is the gradient of one filter weight formed then?

1.8Updates: gradient descent, momentum, regularization

We follow two weights: \(w_{\text{veg},2} = 0.5\) and \(v_{12} = -1.5\). Learning rate \(\eta = 0.5\).

  1. Give their values after one step of plain gradient descent.
  2. With momentum \(\alpha = 0.9\), the previous update of \(v_{12}\) was \(\Delta v_{12}(n-1) = -0.04\). Compute \(\Delta v_{12}(n)\) and the new \(v_{12}\).
  3. Now drop momentum and add L2 regularization: the training objective becomes the loss plus \(\tfrac{\lambda}{2}\) times the sum of the squared weights, with \(\lambda = 0.1\). Show that one gradient step first multiplies every weight by a constant factor, then applies the usual step. Give the factor and the new values of the two weights. What is this effect called?
  4. With L1 instead (same \(\lambda\)), consider a weight equal to 0.01 whose loss gradient is 0 on this sample. Compare one L1 step and one L2 step on it. Which penalty produces sparse weights, and why?

1.9The same weight with Adam

We now train \(w_{\text{veg},2}\) (back at 0.5, no regularization) with Adam: \(\eta = 0.01\), \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\), both moment estimates starting at 0, with bias correction. Its first gradient is \(g_1 = -0.5836\) (part 1.5).

  1. Compute the first and second moment estimates, their bias-corrected values, the update and the new weight after step \(t = 1\).
  2. Show that, whatever the non-zero first gradient, the first Adam step has size almost exactly \(\eta\). How large would the first step be without bias correction?
  3. The next mini-batch gives \(g_2 = -0.05\). Compute step \(t = 2\). Show that the update is about \(+0.0073\), and give the new weight.
  4. Plain gradient descent with the same \(\eta = 0.01\) would move this weight by how much over these two steps? Comment on what Adam does differently.
Problem 2 · 5 points · about 20 minutes

Designing a photo classifier

A recycling plant puts a camera above its conveyor belt to sort household waste into 5 classes: plastic, paper, glass, metal and organic. Each photo is resized to 96 × 96 pixels in colour (RGB). The proposed CNN:

LayerSpecification
Input96 × 96 × 3
Conv116 filters 5 × 5, stride 1, zero-padding 2, ReLU
Pool1max-pooling 2 × 2, stride 2
Conv232 filters 3 × 3, stride 1, no padding, ReLU
Pool2max-pooling 2 × 2, stride 2
Conv364 filters 3 × 3, stride 2, no padding, ReLU
Pool3max-pooling 2 × 2, stride 2
Flatten–
FC1128 neurons, ReLU, dropout with rate 0.2
FC25 neurons, softmax

2.1Output volumes

  1. Give the output volume (height × width × depth) of every layer from Conv1 to Flatten. Show that the flattened vector has 1,600 components.
  2. Pool3 receives an 11 × 11 map. What happens to its last row and last column? How could the padding of Conv3 be changed so that nothing is lost, and what would that cost?

2.2Parameter count

  1. Count the trainable parameters of every layer and show that the network has 229,925 in total.
  2. Which layer holds most of them, and what fraction? Why do pooling, ReLU, flatten and dropout add none?

2.3Why convolution?

  1. Suppose Conv1 were replaced by a fully connected layer producing the same 96 × 96 × 16 output from the same input. How many weights would it have (biases excluded)? Compare with Conv1.
  2. How many positions is each Conv1 filter applied at, so how many times is each of its weights reused per photo?
  3. Name the two properties of images that justify replacing the dense layer by a convolution.

2.4How much data?

  1. Using the course’s rule of thumb that links training-set size, number of free parameters and tolerated test error, how many labelled photos would this network need for a 10 % test error?
  2. The plant has 15,000 labelled photos. Conclude, and propose two remedies from the course: one acting on the data, one on the model.

2.5“It has no parameters, so it doesn’t matter”

An intern argues: “Pool1 has no parameters, so removing it cannot change the size of the model.”

  1. Recompute the volumes from Conv2 to Flatten without Pool1.
  2. Give the new total number of parameters. Was the intern right? Explain which layer changes and why.

2.6Reusing the trained network

The network (with Pool1) is trained. Two new projects want to reuse it. A: another plant wants to separate three kinds of plastic from the same kind of camera photos, with 400 labelled photos. B: a hospital wants to classify grayscale chest X-rays as normal or pneumonia, with 300 labelled scans.

  1. For project A, what do you replace, and how many parameters are trainable if everything else is frozen?
  2. For project B, give two reasons why reusing this network is much harder, and say what you would do.

2.7Dropout at test time

Use the slides’ version of dropout, with no rescaling during training. For some photo, all 128 FC1 outputs equal 1.5, and every weight from FC1 to the output neuron “glass” equals 0.05 (ignore its bias).

  1. What is the expected weighted input of “glass” during training? What is it at test time without any correction?
  2. What correction makes the two match? What would you get by multiplying by the dropout rate instead? What does inverted dropout do?
Problem 3 · 6 points · about 30 minutes

Reading sequences

Part A · A vanilla RNN for sentiment

A sentiment classifier reads short product reviews word by word. To keep the arithmetic small, each word is a single number (a 1-dimensional embedding). The review “great but slow” gives \(x = (x_1, x_2, x_3) = (1,\ 0.5,\ -1)\). Its label is negative: \(t = 0\).

The RNN has hidden size 1, activation tanh, initial state \(a_0 = 0\), recurrent weight \(w_{aa} = 0.6\), input weight \(w_{ax} = 1\) and bias \(b_a = 0\). Only the last step has an output: one sigmoid neuron with weight \(w_{ya} = 1.5\) and bias \(b_y = 0.8\) on \(a_3\), giving \(\hat y\), the probability that the review is positive. The loss \(J\) is the binary cross-entropy.

3.1Forward pass

  1. Which type of RNN is this, in the course’s classification? How many trainable parameters does it have, and how would that number change for a 40-word review?
  2. Compute \(a_1, a_2, a_3\). Show that \(a_3 \approx -0.5037\).
  3. Compute \(\hat y\) and the loss \(J\). Is the review classified correctly (threshold 0.5)?

3.2Backpropagation through time

  1. Prove that the derivative of \(J\) with respect to the output neuron’s weighted input equals \(\hat y - t\), for either value of \(t\). Deduce \(\partial J/\partial w_{ya}\) and \(\partial J/\partial b_y\).
  2. Show that \(\partial J/\partial a_3 \approx 0.7667\).
  3. Let \(d_t\) be the derivative of \(J\) with respect to the weighted input of \(a_t\) (the argument of tanh). Compute \(d_3\), \(d_2\) and \(d_1\).
  4. Compute \(\partial J/\partial w_{aa}\), \(\partial J/\partial w_{ax}\) and \(\partial J/\partial b_a\). Explain why each is a sum over the time steps.

3.3Vanishing gradients

  1. Compute \(\partial a_3/\partial a_1\).
  2. Show that, for this network and whatever the inputs, \(|\partial a_T/\partial a_1| \le 0.6^{\,T-1}\). Evaluate it for a 30-word review and explain the consequence, for instance when the first word is “not”.
  3. What could happen instead if \(w_{aa} = 3\)?

Part B · An LSTM that must remember, then forget

A language model has to remember that the subject of the sentence is plural, to write “are” rather than “is” several words later. This fact is stored in the cell state of an LSTM cell with hidden size 1, which starts with \(h_0 = 0\) and \(C_0 = 1.5\). It reads \(x_1 = 0.5\) (another word of the sentence), then \(x_2 = -1\) (a full stop, which ends the sentence). Each gate looks at \(h_{t-1}\) and \(x_t\):

GateActivationWeight on \(h_{t-1}\)Weight on \(x_t\)Bias
forget \(f\)σ0.520.5
input \(i\)σ11−1
candidate \(\tilde C\)tanh−110
output \(o\)σ10.50.5

3.4Two steps of an LSTM

  1. Step 1: compute the four gate values, then \(C_1\) and \(h_1\). Show that \(C_1 \approx 1.4008\) and \(h_1 \approx 0.6014\).
  2. Step 2: compute \(C_2\) and \(h_2\). What happened to the stored fact, which gate did it, and is that what we want here?
  3. Compute the derivative of \(C_2\) with respect to \(C_0\) along the cell-state path only, and compare it with \(\partial a_3/\partial a_1\) from 3.3. The LSTM is barely better on these two steps. Explain why, and what still makes its memory fundamentally different from the RNN’s.

3.5Real-size recurrent layers

  1. A real model uses word embeddings of dimension 50 and hidden size 100. Show that an LSTM layer has 60,400 parameters (biases included) and that this is exactly four times a vanilla RNN layer of the same size.
  2. How many would a GRU layer have, biases included?

Part C · Attention in translation

A translation model reads the French sentence “le chat dort”. Its encoder produced \(h_1 = (1, 1, 0)\) for “le”, \(h_2 = (0, 2, 1)\) for “chat” and \(h_3 = (1, 0, 2)\) for “dort”. The decoder has already written “the”, and its current state is \(s = (0, 1, 0.5)\). The score is the dot product.

3.6Attention weights and context

  1. Compute the three scores, the attention weights and the context vector. Show that \(\alpha_2 \approx 0.6914\). Which French word is the decoder looking at, and which English word is it probably about to write?
  2. Recompute the weights with the scores divided by \(\sqrt 3\). What changes, and why do Transformers do this?
  3. Why does attention help with long sentences, compared with a plain encoder–decoder?
End of the paper

Hand in

When you submit, the timer stops and the solutions to every part appear. Then you mark each part yourself. This attempt cannot be reopened.