MIAI504 · Neural Networks & Deep Learning

Worked examples, by hand

Every computation the exam can ask you to do on paper, written out step by step: one backprop pass through a 2-2-2 network, convolution and shape tracing for CNNs, and an RNN, LSTM, GRU and attention step.

PART 1 · LECTURES 03–04

Backpropagation by hand

Start with one neuron to see the chain rule, then do the full Lecture 03 example (slides 47–58).

1.1 Warm-up: one sigmoid neuron

Given

Input \(x = 1.5\), weight \(w = 0.8\), bias \(b = -0.2\), target \(t = 1\), learning rate \(\eta = 0.5\). Sigmoid activation, error \(E = \tfrac12 (t - o)^2\).

The whole idea of backprop is in one line. The error depends on \(w\) only through a chain \(w \to z \to o \to E\), so you multiply the three local derivatives:

\[\frac{\partial E}{\partial w} = \underbrace{\frac{\partial E}{\partial o}}_{(o-t)} \cdot \underbrace{\frac{\partial o}{\partial z}}_{o(1-o)} \cdot \underbrace{\frac{\partial z}{\partial w}}_{x}\]
forward\(z = 0.8 \cdot 1.5 - 0.2 = 1.0\),  \(o = \sigma(1.0) = 0.731058579\)
error\(E = \tfrac12(1 - 0.731058579)^2 = 0.036164744\)
∂E/∂o\(o - t = -0.268941421\)
∂o/∂z\(o(1-o) = 0.731058579 \cdot 0.268941421 = 0.196611933\)
δ = ∂E/∂z\(-0.268941421 \cdot 0.196611933 = -0.052877093\)
∂E/∂w\(\delta \cdot x = -0.052877093 \cdot 1.5 = -0.079315639\)
∂E/∂b\(\delta \cdot 1 = -0.052877093\)
update\(w^+ = 0.8 - 0.5(-0.079315639) = 0.839657820\),  \(b^+ = -0.2 - 0.5(-0.052877093) = -0.173561454\)

The gradient is negative, so the weight goes up. That makes sense: the output is too low (0.73 vs 1), and a bigger weight pushes it higher.

The quantity \(\delta\), the error at a neuron's input, is the thing you will carry backwards through a network. Once you have a neuron's \(\delta\), the gradient for any weight coming into it is just \(\delta \times\) whatever that weight multiplies.

1.2 The course example: a 2-2-2 network

This is Example 3 in Lecture 03 (slides 47–58). The slides show the working as images, so here it is in full.

Given

Inputs \(i_1 = 0.05,\ i_2 = 0.10\). Targets \(t_1 = 0.01,\ t_2 = 0.99\). Weights as in the figure. Biases \(b_1 = 0.35\) (shared by both hidden neurons) and \(b_2 = 0.60\) (shared by both outputs). Logistic sigmoid everywhere, \(\eta = 0.5\), \(E = \sum \tfrac12(t - o)^2\). Biases are not updated in this example.

Step 1 · Forward pass

For each neuron: weighted sum of what comes in, then sigmoid. Hidden layer first, then use the hidden outputs as inputs to the output layer.

net h1\(w_1 i_1 + w_2 i_2 + b_1 = 0.15(0.05) + 0.20(0.10) + 0.35 = 0.3775\)
out h1\(\sigma(0.3775) = 1/(1+e^{-0.3775}) = 0.593269992\)
net h2\(0.25(0.05) + 0.30(0.10) + 0.35 = 0.3925\)
out h2\(\sigma(0.3925) = 0.596884378\)
net o1\(w_5 h_1 + w_6 h_2 + b_2 = 0.40(0.593269992) + 0.45(0.596884378) + 0.60 = 1.105905967\)
out o1\(\sigma(1.105905967) = 0.751365070\)
net o2\(0.50(0.593269992) + 0.55(0.596884378) + 0.60 = 1.224921404\)
out o2\(\sigma(1.224921404) = 0.772928465\)

Step 2 · Total error

E o1\(\tfrac12(0.01 - 0.751365070)^2 = 0.274811083\)
E o2\(\tfrac12(0.99 - 0.772928465)^2 = 0.023560026\)
E total\(0.274811083 + 0.023560026 = 0.298371109\)

Step 3 · Output layer: w5 in full, then the rest

\(w_5\) connects \(h_1\) to \(o_1\). It only affects the error through \(o_1\), so:

\[\frac{\partial E}{\partial w_5} = \frac{\partial E}{\partial o_1}\cdot\frac{\partial o_1}{\partial net_{o1}}\cdot\frac{\partial net_{o1}}{\partial w_5} = (o_1 - t_1)\cdot o_1(1-o_1)\cdot h_1 = \delta_{o1}\cdot h_1\]
∂E/∂o1\(o_1 - t_1 = 0.751365070 - 0.01 = 0.741365070\)
∂o1/∂net\(o_1(1 - o_1) = 0.751365070 \cdot 0.248634930 = 0.186815602\)
δ o1\(0.741365070 \cdot 0.186815602 = 0.138498562\)
∂E/∂w5\(\delta_{o1} \cdot h_1 = 0.138498562 \cdot 0.593269992 = 0.082167041\)
w5 new\(0.40 - 0.5 \cdot 0.082167041 = 0.358916480\)

Do the same for \(o_2\): \(\delta_{o2} = (0.772928465 - 0.99)\cdot 0.175510053 = -0.038098237\). It is negative because \(o_2\) is below its target, so the weights into \(o_2\) will go up.

WeightGradient = δ × input∂E/∂wOldNew = old − 0.5·grad
w5 (h1→o1)δo1 · h10.0821670410.400.358916480
w6 (h2→o1)δo1 · h20.0826676280.450.408666186
w7 (h1→o2)δo2 · h1−0.0226025400.500.511301270
w8 (h2→o2)δo2 · h2−0.0227402420.550.561370121

Write the new weights down but keep using the old \(w_5 \dots w_8\) in step 4. The whole backward pass uses the weights from the forward pass. Updating them all at the end is what the course and most textbooks do.

Step 4 · Hidden layer: w1 in full, then the rest

\(w_1\) connects \(i_1\) to \(h_1\). The difference from the output layer: \(h_1\) feeds both outputs, so its error is the sum of what flows back from each one.

\[\frac{\partial E}{\partial h_1} = \delta_{o1} w_5 + \delta_{o2} w_7 \qquad \delta_{h1} = \frac{\partial E}{\partial h_1}\cdot h_1(1-h_1) \qquad \frac{\partial E}{\partial w_1} = \delta_{h1}\cdot i_1\]
∂E/∂h1\(0.138498562(0.40) + (-0.038098237)(0.50) = 0.055399425 - 0.019049118 = 0.036350306\)
h1(1−h1)\(0.593269992 \cdot 0.406730008 = 0.241300709\)
δ h1\(0.036350306 \cdot 0.241300709 = 0.008771355\)
∂E/∂w1\(\delta_{h1}\cdot i_1 = 0.008771355 \cdot 0.05 = 0.000438568\)
w1 new\(0.15 - 0.5 \cdot 0.000438568 = 0.149780716\)

Same for \(h_2\), which uses \(w_6\) and \(w_8\) (the weights leaving \(h_2\)): \(\partial E/\partial h_2 = 0.138498562(0.45) + (-0.038098237)(0.55) = 0.041370323\), and \(\delta_{h2} = 0.041370323 \cdot 0.240613417 = 0.009954255\).

WeightGradient = δ × input∂E/∂wOldNew
w1 (i1→h1)δh1 · i10.0004385680.150.149780716
w2 (i2→h1)δh1 · i20.0008771350.200.199561432
w3 (i1→h2)δh2 · i10.0004977130.250.249751144
w4 (i2→h2)δh2 · i20.0009954250.300.299502287

Step 5 · Check

A second forward pass with all 8 new weights gives \(o_1 = 0.742088\) and \(o_2 = 0.775285\): both moved toward their targets. The total error drops from 0.298371109 to 0.291027774.

The slides say 0.291027924. That small gap comes from rounding in the original source, not from a different method. If you get 0.29102777, you did it right.

1.3 Variations the exam can throw at you

The method above never changes. Only one of the three chain-rule factors changes.

If the question says…What changes
Update the biases tooA bias is a weight whose input is always 1. So \(\partial E/\partial b = \delta\), summed over every neuron that shares it. In the example: \(\partial E/\partial b_2 = \delta_{o1} + \delta_{o2} = 0.100400325\), and \(\partial E/\partial b_1 = \delta_{h1} + \delta_{h2} = 0.018725609\).
Linear output (regression; Lecture 03, Example 1)\(f'(net) = 1\), so \(\delta_o = o - t\).
Sigmoid with slope a: \(f = 1/(1+e^{-a x})\)\(f' = a\, f(1-f)\). Multiply every \(o(1-o)\) by \(a\).
tanh\(f' = 1 - f^2\) (use the output value, not the net value).
ReLU\(f' = 1\) if net > 0, else 0. A neuron with net ≤ 0 passes no error back.
Error written as \(\tfrac12(t-o)^2\) with \(\Delta w = +\eta\,\delta\,x\)The same rule with the sign flipped: here \(\delta = (t - o) f'\). Pick one convention and keep it through the whole problem.
Mean error over N samples: \(\tfrac{1}{2N}\sum\)Batch learning: compute the gradient for each sample, average them, then do one update.

Softmax with cross-entropy

Scores \(z = (2.0,\ 1.0,\ 0.1)\), true class is the first one, so \(t = (1, 0, 0)\).

exponentials\(e^{2} = 7.389056,\ e^{1} = 2.718282,\ e^{0.1} = 1.105171\), sum \(= 11.212509\)
softmax\(p = (0.659001,\ 0.242433,\ 0.098566)\), which sums to 1
loss\(-\ln p_{true} = -\ln 0.659001 = 0.417030\)
δ = p − t\((-0.340999,\ 0.242433,\ 0.098566)\)

Remember this result: with softmax and cross-entropy together, the output \(\delta\) is just \(p - t\). There is no extra derivative factor, so this pair does not suffer from the slow learning that sigmoid with squared error has when the output saturates.

Momentum, two steps

\(w_0 = 1.0\), \(\eta = 0.1\), \(\beta = 0.9\), \(v_0 = 0\). Gradients at steps 1 and 2: 0.4 and 0.3.

rule\(v_t = \beta v_{t-1} - \eta\, g_t, \quad w_t = w_{t-1} + v_t\)
step 1\(v_1 = 0.9(0) - 0.1(0.4) = -0.04, \quad w_1 = 0.96\)
step 2\(v_2 = 0.9(-0.04) - 0.1(0.3) = -0.066, \quad w_2 = 0.894\)

Plain gradient descent would reach 0.93 after these two steps. Momentum gets further (0.894) because the first step's velocity carries over.

Where marks are lost

  • Using the updated \(w_5\dots w_8\) when computing hidden deltas. Always use the old weights.
  • For a hidden neuron, forgetting to sum the error coming back from every output it feeds.
  • Computing \(\sigma'(net)\) as \(net(1-net)\). It is \(out(1-out)\): use the neuron's output.
  • Multiplying \(\delta\) by the wrong input. For \(w_5\) the input is \(h_1\) (the neuron the weight comes from), not \(i_1\).
  • Rounding to 2–3 decimals early. Hidden-layer gradients here are around 0.0004, so keep at least 6 significant figures.
Practice · timed, 25 minutes

Same network shape, new numbers. \(i_1 = 0.2,\ i_2 = 0.6\). \(w_1 = 0.1,\ w_2 = 0.3,\ w_3 = 0.2,\ w_4 = 0.4\). \(w_5 = 0.5,\ w_6 = -0.3,\ w_7 = 0.6,\ w_8 = 0.2\). \(b_1 = 0.25,\ b_2 = 0.1\) (not updated). Targets \(t_1 = 1,\ t_2 = 0\). Sigmoid, \(\eta = 0.5\).

  1. Do the forward pass and compute the total error.
  2. Compute \(\delta_{o1}, \delta_{o2}\) and the new \(w_5 \dots w_8\).
  3. Compute \(\delta_{h1}, \delta_{h2}\) and the new \(w_1 \dots w_4\).
Answers
net h1, h10.45 → 0.610639234
net h2, h20.53 → 0.629483112
net o1, o10.216474683 → 0.553908317
net o2, o20.592280163 → 0.643888150
E total0.099498895 + 0.207295975 = 0.306794869
δo1, δo2−0.110226531, 0.147641106
new w5, w6, w7, w80.533654322, −0.265307130, 0.554922274, 0.153531209
∂E/∂h1, ∂E/∂h20.033471398, 0.062596180
δh1, δh20.007958125, 0.014599565
new w1, w2, w3, w40.099204188, 0.297612563, 0.198540043, 0.395620130
E after one update0.293781085

Watch \(w_6\): it starts negative. \(\partial E/\partial h_2 = \delta_{o1}w_6 + \delta_{o2}w_8 = (-0.110226531)(-0.3) + (0.147641106)(0.2)\). The two negatives multiply to a positive.

PART 2 · LECTURES 05–06

CNN mechanics by hand

Convolution, stride, padding, pooling, multiple channels, then tracing a whole network's shapes and parameter counts.

2.1 One convolution, stride 1

A convolution layer slides a small kernel over the input. At each position it multiplies the overlapping numbers element by element and adds them up, giving one output number. (Strictly this is cross-correlation, since the kernel isn't flipped. Deep learning calls it convolution anyway.)

Input 5×5
12013
01231
10120
21011
02102
∗
Kernel 3×3
10−1
10−1
10−1
=
Output 3×3
−1−3−1
0−41
10−1
top-left\(1(1) + 2(0) + 0(-1) + 0(1) + 1(0) + 2(-1) + 1(1) + 0(0) + 1(-1) = 1 - 2 + 1 - 1 = -1\)
next →window is columns 2–4 of rows 1–3: \((2+1+0) - (1+3+2) = -3\)
shortcutthis kernel computes (left column sum) − (right column sum) of each window. It is a vertical-edge detector.
size\((5 - 3)/1 + 1 = 3\), so 3×3

Then add the bias (here 0) and apply the activation. Each output cell is literally one neuron that only looks at a 3×3 patch, and every one of those neurons shares the same 9 weights.

2.2 Stride 2

Stride 2 means the window jumps 2 cells at a time. Windows start at rows/columns 0 and 2 only.

size\(\lfloor (5 - 3)/2 \rfloor + 1 = 2\), so 2×2
output\(\begin{bmatrix}-1 & -1\\ 1 & -1\end{bmatrix}\) (the 4 corners of the stride-1 output)

2.3 Padding 1, then ReLU, then 2×2 max pooling

Padding adds a border of zeros so the kernel can be centred on edge pixels. With \(P = 1\) the 5×5 input becomes 7×7, and the output stays 5×5 ("same" padding).

Padded input 7×7, first window
0000000
0120130
0012310
0101200
0210110
0021020
0000000
Conv output 5×5
−3−1−1−24
−3−1−3−16
−20−416
−310−13
−312−21

Top-left: the window covers \(\begin{bmatrix}0&0&0\\0&1&2\\0&0&1\end{bmatrix}\), giving (left column 0) − (right column 0 + 2 + 1) = −3.

Now ReLU sets the negatives to 0, and 2×2 max pooling with stride 2 takes the largest value in each 2×2 block:

After ReLU
00004
00006
00016
01003
01201
→
Max pool 2×2
00
11

The edge case: \(\lfloor (5-2)/2 \rfloor + 1 = 2\), not 2.5. The window can't fit a third time, so the last row and column (crossed out) are simply dropped. Here that throws away the strongest activations in the map, the 6s. That is why architectures choose sizes that divide evenly, or pad before pooling.

2.4 The output-size formula and its edge cases

\[O = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1\]

\(W\) = input width (or height), \(K\) = kernel size, \(P\) = padding on each side, \(S\) = stride. Apply it to height and width separately if they differ. Pooling uses the same formula.

CaseRuleExample
Division not wholeRound down (floor). The leftover cells are ignored.W=28, K=3, S=2: ⌊25/2⌋+1 = 13
"Same" paddingWith S = 1, \(P = (K-1)/2\) keeps the size.K=3 → P=1; K=5 → P=2
"Valid" paddingP = 0. The output shrinks by K − 1.32, K=5 → 28
PoolingUsually K = S and P = 0. It halves the size when K = S = 2. No parameters.28 → 14
1×1 convolutionKeeps width and height. Only changes the number of channels (mixes them).14×14×64 → 14×14×16
Output depthAlways equals the number of filters, whatever the input depth.any input, 32 filters → depth 32

2.5 Several input channels

With a colour image (3 channels) or a previous layer with many feature maps, each filter has the same depth as its input. You convolve each channel with its own slice of the filter, add up the slices, and add one bias. One filter therefore gives one output map, however many channels go in.

Channel A
102
310
021
Kernel slice A
1−1
02
Channel B
011
201
110
Kernel slice B
20
1−1
slice A\(1(1) + 0(-1) + 3(0) + 1(2) = 3\)
slice B\(0(2) + 1(0) + 2(1) + 0(-1) = 2\)
top-left\(3 + 2 + \text{bias } 1 = 6\)
full outputA gives \(\begin{bmatrix}3&-2\\6&3\end{bmatrix}\), B gives \(\begin{bmatrix}2&1\\4&1\end{bmatrix}\); sum + 1 = \(\begin{bmatrix}6&0\\11&5\end{bmatrix}\), one 2×2 map
\[\text{params per conv layer} = (K \cdot K \cdot C_{in} + 1)\cdot C_{out}\]

Here: \((2\cdot2\cdot2 + 1)\cdot 1 = 9\) parameters. The "+1" is the one bias per filter.

2.6 Tracing a whole network

This is the classic exam question: given an architecture, fill in the shape after each layer and the number of parameters. Example: a LeNet-5 style network on a 32×32 greyscale image.

LayerSize calculationOutputParameters
Input32×32×10
Conv 5×5, 6 filters32−5+1 = 2828×28×6(5·5·1+1)·6 = 156
Max pool 2×2, S=2⌊26/2⌋+1 = 1414×14×60
Conv 5×5, 16 filters14−5+1 = 1010×10×16(5·5·6+1)·16 = 2,416
Max pool 2×2, S=210/2 = 55×5×160
Flatten5·5·164000
FC 120120400·120+120 = 48,120
FC 8484120·84+84 = 10,164
FC 10 (softmax)1084·10+10 = 850
Total61,706

Notice where the parameters are: the two conv layers hold 2,572 of them, and the first fully connected layer alone holds 48,120. Convolutions are cheap because of weight sharing (one small kernel reused everywhere) and local connectivity (each output only sees a patch).

Second trace: colour input, "same" padding and a stride that doesn't divide

LayerSize calculationOutputParameters
Input RGB64×64×30
Conv 3×3, 32, S=1, P=1(64−3+2)/1+1 = 6464×64×32(3·3·3+1)·32 = 896
Max pool 2×2, S=23232×32×320
Conv 3×3, 64, S=2, P=0⌊(32−3)/2⌋+1 = ⌊14.5⌋+1 = 1515×15×64(3·3·32+1)·64 = 18,496
Conv 3×3, 128, S=1, P=015−3+1 = 1313×13×128(3·3·64+1)·128 = 73,856
Max pool 2×2, S=2⌊11/2⌋+1 = 6 (last row/col dropped)6×6×1280
Flatten → FC 106·6·128 = 4,608104,608·10+10 = 46,090

The "parameter trap" comparison (video 6): a single fully connected layer of just 100 neurons on the raw 64×64×3 image would need \(12{,}288 \cdot 100 + 100 = 1{,}228{,}900\) parameters. The first conv layer above does useful work with 896.

2.7 How a CNN is trained

  • Same backprop as Part 1. A conv layer is a layer of neurons with shared weights and local connections.
  • Shared weights: a kernel weight is used at every position, so its gradient is the sum of the gradients from all positions.
  • Max pooling: the gradient goes only to the cell that was the maximum in each window. The other cells get 0. Average pooling splits it equally.
  • Transfer learning (Lecture 05): take a network pretrained on a big dataset. With a small dataset similar to the original, freeze the conv layers and retrain only the new classifier head. With more data or a different domain, also fine-tune the top layers with a small learning rate.
Practice
  1. Input 227×227×3. Conv 11×11, 96 filters, stride 4, no padding. Output size and parameters?
  2. Then max pool 3×3 with stride 2. Output size?
  3. Input 7×7, kernel 3×3, stride 2, padding 1. Output size?
  4. Conv 3×3 from 16 channels to 32 channels. Parameters?
  5. A 1×1 conv from 64 channels to 16. Parameters, and what happens to width and height?
Answers
  1. \((227-11)/4 + 1 = 55\), so 55×55×96. Parameters \((11\cdot11\cdot3+1)\cdot 96 = 34{,}944\). (This is AlexNet's first layer.)
  2. \(\lfloor(55-3)/2\rfloor+1 = 27\), so 27×27×96, with 0 parameters.
  3. \(\lfloor(7-3+2)/2\rfloor + 1 = 4\), so 4×4.
  4. \((3\cdot3\cdot16+1)\cdot32 = 4{,}640\).
  5. \((1\cdot1\cdot64+1)\cdot16 = 1{,}040\). Width and height don't change; only the depth goes 64 → 16.
PART 3 · LECTURE 07

RNNs by hand, from zero

An RNN is one small network applied at every time step, passing a memory \(h\) from each step to the next. Everything below uses scalars so you can do it on paper.

3.1 The idea and the equations

A feed-forward network sees one input and forgets it. For sequences (text, speech, time series) the meaning depends on order and context. An RNN keeps a hidden state \(h_t\) that summarises everything seen so far, and updates it at each step:

\[h_t = \tanh(W_{xh}\, x_t + W_{hh}\, h_{t-1} + b) \qquad y_t = W_{hy}\, h_t\]
  • The same weights \(W_{xh}, W_{hh}, W_{hy}\) are used at every time step. That is what lets it handle sequences of any length.
  • \(h_0\) is usually 0.
  • "Unrolling" means drawing one copy of the network per time step, linked by \(h\). An unrolled RNN is just a deep feed-forward network with shared weights.
TypeInput → outputExample
One-to-one1 → 1Ordinary classification (not really recurrent)
One-to-many1 → sequenceImage captioning: one image in, a sentence out
Many-to-onesequence → 1Sentiment of a review; output only at the last step
Many-to-many (same length)sequence → sequence, alignedTagging each word (part of speech, named entities)
Many-to-many (encoder-decoder)sequence → different-length sequenceMachine translation

3.2 Forward pass over 3 time steps

Given · many-to-one

\(w_x = 0.6,\ w_h = 0.8,\ w_y = 1.5,\ b = 0,\ h_0 = 0\). Input sequence \(x = (1.0,\ 0.5,\ -1.0)\). Output only at the last step, target \(t = 0.5\), \(E = \tfrac12(y_3 - t)^2\).

t = 1\(net_1 = 0.6(1.0) + 0.8(0) = 0.6, \quad h_1 = \tanh(0.6) = 0.537050\)
t = 2\(net_2 = 0.6(0.5) + 0.8(0.537050) = 0.729640, \quad h_2 = \tanh(0.729640) = 0.622845\)
t = 3\(net_3 = 0.6(-1.0) + 0.8(0.622845) = -0.101724, \quad h_3 = \tanh(-0.101724) = -0.101375\)
output\(y_3 = 1.5 \cdot (-0.101375) = -0.152062\)
error\(E = \tfrac12(-0.152062 - 0.5)^2 = 0.212592\)

Notice how \(h_2\) carries the first two inputs forward: without it, step 3 would only see \(x_3 = -1\).

3.3 Backpropagation through time (BPTT)

Unroll the network, then backprop exactly as in Part 1, from step 3 back to step 1. Two new things:

  • The error reaches \(h_{k-1}\) through \(h_k\): \(\ \partial E/\partial h_{k-1} = \delta_k \cdot w_h\), where \(\delta_k = \partial E/\partial h_k \cdot (1 - h_k^2)\) (tanh derivative).
  • Because \(w_h\) is used at every step, its gradient is the sum over the steps: \(\ \partial E/\partial w_h = \sum_k \delta_k\, h_{k-1}\). Same for \(w_x\): \(\sum_k \delta_k\, x_k\).
start\(\partial E/\partial y_3 = y_3 - t = -0.652062\);  \(\partial E/\partial w_y = -0.652062 \cdot h_3 = 0.066103\);  \(\partial E/\partial h_3 = -0.652062 \cdot 1.5 = -0.978093\)
Step k∂E/∂h_k1 − h_k²δ_kδ_k · h_{k−1} (for w_h)δ_k · x_k (for w_x)
3−0.9780930.989723−0.968041−0.6029400.968041
2−0.968041·0.8 = −0.7744330.612064−0.474003−0.254563−0.237001
1−0.474003·0.8 = −0.3792020.711578−0.2698320 (h₀ = 0)−0.269832
Sum = gradient∂E/∂w_h = −0.857503∂E/∂w_x = 0.461208

Update as usual, \(w \leftarrow w - \eta\,\partial E/\partial w\). (These values were checked against a numerical gradient.)

Why RNNs forget: vanishing and exploding gradients

Look at the \(\partial E/\partial h_k\) column: each step back multiplies by \(w_h (1 - h_k^2)\). Here those factors are 0.79, 0.49 and 0.57, so the signal from step 3 shrinks fast. Over 20 steps with a factor around 0.4: \(0.4^{20} \approx 10^{-8}\). The early inputs get almost no gradient, so the network cannot learn long-range dependencies. If the factor is above 1 (say 1.5), it explodes instead: \(1.5^{20} \approx 3{,}325\).

  • Exploding: fixed with gradient clipping (cap the gradient's size).
  • Vanishing: fixed by changing the cell: LSTM and GRU.

3.4 One LSTM step

An LSTM adds a second memory, the cell state \(c_t\), and three gates. Each gate is a sigmoid between 0 (block) and 1 (let through).

\[\begin{aligned} f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) && \text{forget: how much old memory to keep}\\ i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) && \text{input: how much new info to write}\\ \tilde c_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) && \text{candidate: the new info itself}\\ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) && \text{output: how much memory to show}\\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde c_t \qquad h_t = o_t \odot \tanh(c_t) \end{aligned}\]
Given

\(x_t = 1,\ h_{t-1} = 0.5,\ c_{t-1} = 0.8\). Weights \((W, U, b)\): forget \((0.5, 0.5, 0)\), input \((1, -1, 0)\), candidate \((0.8, 0.4, 0)\), output \((0.5, 1, -0.5)\).

forget\(f = \sigma(0.5 + 0.25) = \sigma(0.75) = 0.679179\)
input\(i = \sigma(1 - 0.5) = \sigma(0.5) = 0.622459\)
candidate\(\tilde c = \tanh(0.8 + 0.2) = \tanh(1.0) = 0.761594\)
output gate\(o = \sigma(0.5 + 0.5 - 0.5) = \sigma(0.5) = 0.622459\)
cell\(c = 0.679179(0.8) + 0.622459(0.761594) = 0.543343 + 0.474061 = 1.017404\)
hidden\(h = 0.622459 \cdot \tanh(1.017404) = 0.622459 \cdot 0.768807 = 0.478551\)

Why this fixes vanishing gradients: the cell state is updated by addition (\(f \cdot c_{t-1} + \dots\)), not by squashing it through a weight matrix and a tanh at every step. Going back one step, the gradient along the cell path is just multiplied by \(f\). When the forget gate stays near 1, the gradient passes through many steps almost unchanged.

3.5 One GRU step

A GRU merges the cell and hidden state and uses only two gates, so it has fewer parameters and is faster. It usually performs about as well as an LSTM.

\[\begin{aligned} z_t &= \sigma(W_z x_t + U_z h_{t-1}) && \text{update gate}\\ r_t &= \sigma(W_r x_t + U_r h_{t-1}) && \text{reset gate}\\ \tilde h_t &= \tanh(W x_t + U (r_t \odot h_{t-1})) && \text{candidate}\\ h_t &= (1 - z_t)\odot h_{t-1} + z_t \odot \tilde h_t \end{aligned}\]

Same \(x_t = 1,\ h_{t-1} = 0.5\). Update gate weights \((0.5, 0.5)\), reset \((1, -1)\), candidate \((0.8, 0.4)\).

update\(z = \sigma(0.75) = 0.679179\)
reset\(r = \sigma(0.5) = 0.622459\),  \(r \cdot h_{t-1} = 0.311230\)
candidate\(\tilde h = \tanh(0.8 + 0.4 \cdot 0.311230) = \tanh(0.924492) = 0.728015\)
hidden\(h = (1 - 0.679179)(0.5) + 0.679179(0.728015) = 0.654863\)

Some books swap the roles: \(h_t = z_t \odot h_{t-1} + (1-z_t)\odot\tilde h_t\). Both are correct; use the one in your exam statement.

3.6 Parameter counts and the variants

For input size \(x\) and hidden size \(h\), one vanilla RNN cell has \(h\cdot x\) (input weights) \(+\ h\cdot h\) (recurrent weights) \(+\ h\) (bias). An LSTM has 4 such blocks (3 gates + candidate); a GRU has 3.

CellFormulax = 10, h = 20
Vanilla RNNh(x + h) + h200 + 400 + 20 = 620
LSTM4 · [h(x + h) + h]2,480
GRU3 · [h(x + h) + h]1,860
Bidirectional (any cell)2 × the celle.g. BiLSTM 4,960
  • Bidirectional RNN / BiLSTM: one RNN reads left to right, another right to left, and their hidden states are concatenated. Each position then sees past and future context. It needs the whole sequence up front, so it doesn't work for real-time generation.
  • Encoder-decoder: the encoder reads the whole input and its final hidden state becomes a fixed-size context vector. The decoder generates the output from it. The weakness: one fixed vector must hold an entire long sentence (the bottleneck).

3.7 Attention, one step by hand

Attention removes the bottleneck. At each output step, the decoder looks back at all encoder states and takes a weighted average, with weights chosen by relevance.

Given

Encoder states \(h_1 = (1, 0),\ h_2 = (0, 1),\ h_3 = (1, 1)\). Current decoder state (the query) \(s = (1, 2)\). Dot-product scores.

1 · scores\(s\cdot h_1 = 1,\quad s\cdot h_2 = 2,\quad s\cdot h_3 = 3\)
2 · softmax\(e^1, e^2, e^3 = 2.718, 7.389, 20.086\), sum \(30.193\), so \(\alpha = (0.090,\ 0.245,\ 0.665)\)
3 · context\(c = 0.090(1,0) + 0.245(0,1) + 0.665(1,1) = (0.755,\ 0.910)\)

The decoder pays most attention to \(h_3\), the state most aligned with its query. The context \(c\) is then fed into the decoder along with \(s\).

Self-attention (the core of Transformers) does the same thing within one sequence: every position builds a query \(Q\), a key \(K\) and a value \(V\) from its own vector, and attends to every other position:

\[\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V\]

Dividing by \(\sqrt{d_k}\) keeps the scores from growing with the vector size, which would make the softmax too peaked. Scaled, the example above becomes scores \((0.707, 1.414, 2.121)\), weights \((0.140, 0.284, 0.576)\): a softer spread. Unlike an RNN there's no step-by-step recurrence, so all positions are computed in parallel.

Where marks are lost

  • Forgetting that the RNN weights are shared, so the gradient is a sum over time steps.
  • Using \(\sigma'\) instead of \(\tanh' = 1 - h^2\) for the hidden state.
  • Counting LSTM parameters with 3 blocks instead of 4. The candidate \(\tilde c\) has weights too.
  • Saying LSTMs fix exploding gradients. They mainly fix vanishing ones; exploding is handled by clipping.
Practice
  1. Scalar RNN with \(w_x = 1,\ w_h = 0.5,\ b = 0,\ h_0 = 0\), input \(x = (0.5, -0.5)\). Compute \(h_1, h_2\).
  2. LSTM with input size 50 and hidden size 100. Parameters? And for a GRU?
  3. Which RNN type for: (a) generating music from a genre label, (b) spam detection on an email, (c) translating French to English?
Answers
  1. \(h_1 = \tanh(0.5) = 0.462117\); \(\ h_2 = \tanh(-0.5 + 0.5 \cdot 0.462117) = \tanh(-0.268941) = -0.262640\).
  2. One block: \(100\cdot50 + 100\cdot100 + 100 = 15{,}100\). LSTM: \(4 \times 15{,}100 = 60{,}400\). GRU: \(3 \times 15{,}100 = 45{,}300\).
  3. (a) one-to-many, (b) many-to-one, (c) many-to-many encoder-decoder (ideally with attention).
Σ · ONE PAGE

Formula sheet

Copy this onto one sheet by hand the night before. Writing it out is part of learning it.

TopicFormula
Sigmoid\(\sigma(z) = \frac{1}{1+e^{-z}},\ \ \sigma' = \sigma(1-\sigma)\) (with slope \(a\): \(a\,\sigma(1-\sigma)\))
tanh / ReLU\(\tanh' = 1 - \tanh^2\);  ReLU\('\) = 1 if \(z>0\) else 0
Output delta\(\delta_o = (o - t)\, f'(net_o)\); with softmax and cross-entropy, \(\delta = p - t\)
Hidden delta\(\delta_h = \big(\sum_k \delta_k\, w_{hk}\big)\, f'(net_h)\), using the old weights
Weight gradient\(\partial E/\partial w_{ij} = \delta_j \cdot (\text{input on that weight})\); bias: \(\delta_j\)
Update\(w \leftarrow w - \eta\, \partial E/\partial w\)
Momentum\(v \leftarrow \beta v - \eta g,\ \ w \leftarrow w + v\)
Softmax / CE\(p_i = e^{z_i}/\sum_j e^{z_j}\);  \(L = -\sum t_i \ln p_i\)
Conv output size\(O = \lfloor (W - K + 2P)/S \rfloor + 1\); same padding \(P = (K-1)/2\)
Conv parameters\((K\cdot K\cdot C_{in} + 1)\cdot C_{out}\); pooling has 0
FC parameters\(n_{in}\cdot n_{out} + n_{out}\)
RNN\(h_t = \tanh(W_{xh}x_t + W_{hh}h_{t-1} + b)\), \(\ y_t = W_{hy}h_t\)
BPTT\(\partial E/\partial W = \sum_t (\dots)\); each step back multiplies by \(W_{hh}\cdot\tanh'\)
LSTM\(c_t = f\odot c_{t-1} + i\odot\tilde c\), \(\ h_t = o\odot\tanh(c_t)\); params \(4[h(x+h)+h]\)
GRU\(h_t = (1-z)\odot h_{t-1} + z\odot\tilde h\); params \(3[h(x+h)+h]\)
Attentionscores → softmax → weighted sum; \(\ \text{softmax}(QK^\top/\sqrt{d_k})\,V\)