Worked examples, by hand
Every computation the exam can ask you to do on paper, written out step by step: one backprop pass through a 2-2-2 network, convolution and shape tracing for CNNs, and an RNN, LSTM, GRU and attention step.
- How to use it: read the givens, cover what follows, do the step on paper, then uncover and compare. Only read ahead once your number matches.
- Practice problems (green boxes) use fresh numbers. Their answers are folded away. Use them for a timed redo.
- Every number here was computed by a script, so if yours differs past the 4th decimal, the mistake is in your working.
Backpropagation by hand
Start with one neuron to see the chain rule, then do the full Lecture 03 example (slides 47–58).
1.1 Warm-up: one sigmoid neuron
Input \(x = 1.5\), weight \(w = 0.8\), bias \(b = -0.2\), target \(t = 1\), learning rate \(\eta = 0.5\). Sigmoid activation, error \(E = \tfrac12 (t - o)^2\).
The whole idea of backprop is in one line. The error depends on \(w\) only through a chain \(w \to z \to o \to E\), so you multiply the three local derivatives:
The gradient is negative, so the weight goes up. That makes sense: the output is too low (0.73 vs 1), and a bigger weight pushes it higher.
The quantity \(\delta\), the error at a neuron's input, is the thing you will carry backwards through a network. Once you have a neuron's \(\delta\), the gradient for any weight coming into it is just \(\delta \times\) whatever that weight multiplies.
1.2 The course example: a 2-2-2 network
This is Example 3 in Lecture 03 (slides 47–58). The slides show the working as images, so here it is in full.
Inputs \(i_1 = 0.05,\ i_2 = 0.10\). Targets \(t_1 = 0.01,\ t_2 = 0.99\). Weights as in the figure. Biases \(b_1 = 0.35\) (shared by both hidden neurons) and \(b_2 = 0.60\) (shared by both outputs). Logistic sigmoid everywhere, \(\eta = 0.5\), \(E = \sum \tfrac12(t - o)^2\). Biases are not updated in this example.
Step 1 · Forward pass
For each neuron: weighted sum of what comes in, then sigmoid. Hidden layer first, then use the hidden outputs as inputs to the output layer.
Step 2 · Total error
Step 3 · Output layer: w5 in full, then the rest
\(w_5\) connects \(h_1\) to \(o_1\). It only affects the error through \(o_1\), so:
Do the same for \(o_2\): \(\delta_{o2} = (0.772928465 - 0.99)\cdot 0.175510053 = -0.038098237\). It is negative because \(o_2\) is below its target, so the weights into \(o_2\) will go up.
| Weight | Gradient = δ × input | ∂E/∂w | Old | New = old − 0.5·grad |
|---|---|---|---|---|
| w5 (h1→o1) | δo1 · h1 | 0.082167041 | 0.40 | 0.358916480 |
| w6 (h2→o1) | δo1 · h2 | 0.082667628 | 0.45 | 0.408666186 |
| w7 (h1→o2) | δo2 · h1 | −0.022602540 | 0.50 | 0.511301270 |
| w8 (h2→o2) | δo2 · h2 | −0.022740242 | 0.55 | 0.561370121 |
Write the new weights down but keep using the old \(w_5 \dots w_8\) in step 4. The whole backward pass uses the weights from the forward pass. Updating them all at the end is what the course and most textbooks do.
Step 4 · Hidden layer: w1 in full, then the rest
\(w_1\) connects \(i_1\) to \(h_1\). The difference from the output layer: \(h_1\) feeds both outputs, so its error is the sum of what flows back from each one.
Same for \(h_2\), which uses \(w_6\) and \(w_8\) (the weights leaving \(h_2\)): \(\partial E/\partial h_2 = 0.138498562(0.45) + (-0.038098237)(0.55) = 0.041370323\), and \(\delta_{h2} = 0.041370323 \cdot 0.240613417 = 0.009954255\).
| Weight | Gradient = δ × input | ∂E/∂w | Old | New |
|---|---|---|---|---|
| w1 (i1→h1) | δh1 · i1 | 0.000438568 | 0.15 | 0.149780716 |
| w2 (i2→h1) | δh1 · i2 | 0.000877135 | 0.20 | 0.199561432 |
| w3 (i1→h2) | δh2 · i1 | 0.000497713 | 0.25 | 0.249751144 |
| w4 (i2→h2) | δh2 · i2 | 0.000995425 | 0.30 | 0.299502287 |
Step 5 · Check
A second forward pass with all 8 new weights gives \(o_1 = 0.742088\) and \(o_2 = 0.775285\): both moved toward their targets. The total error drops from 0.298371109 to 0.291027774.
The slides say 0.291027924. That small gap comes from rounding in the original source, not from a different method. If you get 0.29102777, you did it right.
1.3 Variations the exam can throw at you
The method above never changes. Only one of the three chain-rule factors changes.
| If the question says… | What changes |
|---|---|
| Update the biases too | A bias is a weight whose input is always 1. So \(\partial E/\partial b = \delta\), summed over every neuron that shares it. In the example: \(\partial E/\partial b_2 = \delta_{o1} + \delta_{o2} = 0.100400325\), and \(\partial E/\partial b_1 = \delta_{h1} + \delta_{h2} = 0.018725609\). |
| Linear output (regression; Lecture 03, Example 1) | \(f'(net) = 1\), so \(\delta_o = o - t\). |
| Sigmoid with slope a: \(f = 1/(1+e^{-a x})\) | \(f' = a\, f(1-f)\). Multiply every \(o(1-o)\) by \(a\). |
| tanh | \(f' = 1 - f^2\) (use the output value, not the net value). |
| ReLU | \(f' = 1\) if net > 0, else 0. A neuron with net ≤ 0 passes no error back. |
| Error written as \(\tfrac12(t-o)^2\) with \(\Delta w = +\eta\,\delta\,x\) | The same rule with the sign flipped: here \(\delta = (t - o) f'\). Pick one convention and keep it through the whole problem. |
| Mean error over N samples: \(\tfrac{1}{2N}\sum\) | Batch learning: compute the gradient for each sample, average them, then do one update. |
Softmax with cross-entropy
Scores \(z = (2.0,\ 1.0,\ 0.1)\), true class is the first one, so \(t = (1, 0, 0)\).
Remember this result: with softmax and cross-entropy together, the output \(\delta\) is just \(p - t\). There is no extra derivative factor, so this pair does not suffer from the slow learning that sigmoid with squared error has when the output saturates.
Momentum, two steps
\(w_0 = 1.0\), \(\eta = 0.1\), \(\beta = 0.9\), \(v_0 = 0\). Gradients at steps 1 and 2: 0.4 and 0.3.
Plain gradient descent would reach 0.93 after these two steps. Momentum gets further (0.894) because the first step's velocity carries over.
Where marks are lost
- Using the updated \(w_5\dots w_8\) when computing hidden deltas. Always use the old weights.
- For a hidden neuron, forgetting to sum the error coming back from every output it feeds.
- Computing \(\sigma'(net)\) as \(net(1-net)\). It is \(out(1-out)\): use the neuron's output.
- Multiplying \(\delta\) by the wrong input. For \(w_5\) the input is \(h_1\) (the neuron the weight comes from), not \(i_1\).
- Rounding to 2–3 decimals early. Hidden-layer gradients here are around 0.0004, so keep at least 6 significant figures.
Same network shape, new numbers. \(i_1 = 0.2,\ i_2 = 0.6\). \(w_1 = 0.1,\ w_2 = 0.3,\ w_3 = 0.2,\ w_4 = 0.4\). \(w_5 = 0.5,\ w_6 = -0.3,\ w_7 = 0.6,\ w_8 = 0.2\). \(b_1 = 0.25,\ b_2 = 0.1\) (not updated). Targets \(t_1 = 1,\ t_2 = 0\). Sigmoid, \(\eta = 0.5\).
- Do the forward pass and compute the total error.
- Compute \(\delta_{o1}, \delta_{o2}\) and the new \(w_5 \dots w_8\).
- Compute \(\delta_{h1}, \delta_{h2}\) and the new \(w_1 \dots w_4\).
Answers
| net h1, h1 | 0.45 → 0.610639234 |
| net h2, h2 | 0.53 → 0.629483112 |
| net o1, o1 | 0.216474683 → 0.553908317 |
| net o2, o2 | 0.592280163 → 0.643888150 |
| E total | 0.099498895 + 0.207295975 = 0.306794869 |
| δo1, δo2 | −0.110226531, 0.147641106 |
| new w5, w6, w7, w8 | 0.533654322, −0.265307130, 0.554922274, 0.153531209 |
| ∂E/∂h1, ∂E/∂h2 | 0.033471398, 0.062596180 |
| δh1, δh2 | 0.007958125, 0.014599565 |
| new w1, w2, w3, w4 | 0.099204188, 0.297612563, 0.198540043, 0.395620130 |
| E after one update | 0.293781085 |
Watch \(w_6\): it starts negative. \(\partial E/\partial h_2 = \delta_{o1}w_6 + \delta_{o2}w_8 = (-0.110226531)(-0.3) + (0.147641106)(0.2)\). The two negatives multiply to a positive.
CNN mechanics by hand
Convolution, stride, padding, pooling, multiple channels, then tracing a whole network's shapes and parameter counts.
2.1 One convolution, stride 1
A convolution layer slides a small kernel over the input. At each position it multiplies the overlapping numbers element by element and adds them up, giving one output number. (Strictly this is cross-correlation, since the kernel isn't flipped. Deep learning calls it convolution anyway.)
| 1 | 2 | 0 | 1 | 3 |
| 0 | 1 | 2 | 3 | 1 |
| 1 | 0 | 1 | 2 | 0 |
| 2 | 1 | 0 | 1 | 1 |
| 0 | 2 | 1 | 0 | 2 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| −1 | −3 | −1 |
| 0 | −4 | 1 |
| 1 | 0 | −1 |
Then add the bias (here 0) and apply the activation. Each output cell is literally one neuron that only looks at a 3×3 patch, and every one of those neurons shares the same 9 weights.
2.2 Stride 2
Stride 2 means the window jumps 2 cells at a time. Windows start at rows/columns 0 and 2 only.
2.3 Padding 1, then ReLU, then 2×2 max pooling
Padding adds a border of zeros so the kernel can be centred on edge pixels. With \(P = 1\) the 5×5 input becomes 7×7, and the output stays 5×5 ("same" padding).
| 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 2 | 0 | 1 | 3 | 0 |
| 0 | 0 | 1 | 2 | 3 | 1 | 0 |
| 0 | 1 | 0 | 1 | 2 | 0 | 0 |
| 0 | 2 | 1 | 0 | 1 | 1 | 0 |
| 0 | 0 | 2 | 1 | 0 | 2 | 0 |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| −3 | −1 | −1 | −2 | 4 |
| −3 | −1 | −3 | −1 | 6 |
| −2 | 0 | −4 | 1 | 6 |
| −3 | 1 | 0 | −1 | 3 |
| −3 | 1 | 2 | −2 | 1 |
Top-left: the window covers \(\begin{bmatrix}0&0&0\\0&1&2\\0&0&1\end{bmatrix}\), giving (left column 0) − (right column 0 + 2 + 1) = −3.
Now ReLU sets the negatives to 0, and 2×2 max pooling with stride 2 takes the largest value in each 2×2 block:
| 0 | 0 | 0 | 0 | 4 |
| 0 | 0 | 0 | 0 | 6 |
| 0 | 0 | 0 | 1 | 6 |
| 0 | 1 | 0 | 0 | 3 |
| 0 | 1 | 2 | 0 | 1 |
| 0 | 0 |
| 1 | 1 |
The edge case: \(\lfloor (5-2)/2 \rfloor + 1 = 2\), not 2.5. The window can't fit a third time, so the last row and column (crossed out) are simply dropped. Here that throws away the strongest activations in the map, the 6s. That is why architectures choose sizes that divide evenly, or pad before pooling.
2.4 The output-size formula and its edge cases
\(W\) = input width (or height), \(K\) = kernel size, \(P\) = padding on each side, \(S\) = stride. Apply it to height and width separately if they differ. Pooling uses the same formula.
| Case | Rule | Example |
|---|---|---|
| Division not whole | Round down (floor). The leftover cells are ignored. | W=28, K=3, S=2: ⌊25/2⌋+1 = 13 |
| "Same" padding | With S = 1, \(P = (K-1)/2\) keeps the size. | K=3 → P=1; K=5 → P=2 |
| "Valid" padding | P = 0. The output shrinks by K − 1. | 32, K=5 → 28 |
| Pooling | Usually K = S and P = 0. It halves the size when K = S = 2. No parameters. | 28 → 14 |
| 1×1 convolution | Keeps width and height. Only changes the number of channels (mixes them). | 14×14×64 → 14×14×16 |
| Output depth | Always equals the number of filters, whatever the input depth. | any input, 32 filters → depth 32 |
2.5 Several input channels
With a colour image (3 channels) or a previous layer with many feature maps, each filter has the same depth as its input. You convolve each channel with its own slice of the filter, add up the slices, and add one bias. One filter therefore gives one output map, however many channels go in.
| 1 | 0 | 2 |
| 3 | 1 | 0 |
| 0 | 2 | 1 |
| 1 | −1 |
| 0 | 2 |
| 0 | 1 | 1 |
| 2 | 0 | 1 |
| 1 | 1 | 0 |
| 2 | 0 |
| 1 | −1 |
Here: \((2\cdot2\cdot2 + 1)\cdot 1 = 9\) parameters. The "+1" is the one bias per filter.
2.6 Tracing a whole network
This is the classic exam question: given an architecture, fill in the shape after each layer and the number of parameters. Example: a LeNet-5 style network on a 32×32 greyscale image.
| Layer | Size calculation | Output | Parameters |
|---|---|---|---|
| Input | 32×32×1 | 0 | |
| Conv 5×5, 6 filters | 32−5+1 = 28 | 28×28×6 | (5·5·1+1)·6 = 156 |
| Max pool 2×2, S=2 | ⌊26/2⌋+1 = 14 | 14×14×6 | 0 |
| Conv 5×5, 16 filters | 14−5+1 = 10 | 10×10×16 | (5·5·6+1)·16 = 2,416 |
| Max pool 2×2, S=2 | 10/2 = 5 | 5×5×16 | 0 |
| Flatten | 5·5·16 | 400 | 0 |
| FC 120 | 120 | 400·120+120 = 48,120 | |
| FC 84 | 84 | 120·84+84 = 10,164 | |
| FC 10 (softmax) | 10 | 84·10+10 = 850 | |
| Total | 61,706 | ||
Notice where the parameters are: the two conv layers hold 2,572 of them, and the first fully connected layer alone holds 48,120. Convolutions are cheap because of weight sharing (one small kernel reused everywhere) and local connectivity (each output only sees a patch).
Second trace: colour input, "same" padding and a stride that doesn't divide
| Layer | Size calculation | Output | Parameters |
|---|---|---|---|
| Input RGB | 64×64×3 | 0 | |
| Conv 3×3, 32, S=1, P=1 | (64−3+2)/1+1 = 64 | 64×64×32 | (3·3·3+1)·32 = 896 |
| Max pool 2×2, S=2 | 32 | 32×32×32 | 0 |
| Conv 3×3, 64, S=2, P=0 | ⌊(32−3)/2⌋+1 = ⌊14.5⌋+1 = 15 | 15×15×64 | (3·3·32+1)·64 = 18,496 |
| Conv 3×3, 128, S=1, P=0 | 15−3+1 = 13 | 13×13×128 | (3·3·64+1)·128 = 73,856 |
| Max pool 2×2, S=2 | ⌊11/2⌋+1 = 6 (last row/col dropped) | 6×6×128 | 0 |
| Flatten → FC 10 | 6·6·128 = 4,608 | 10 | 4,608·10+10 = 46,090 |
The "parameter trap" comparison (video 6): a single fully connected layer of just 100 neurons on the raw 64×64×3 image would need \(12{,}288 \cdot 100 + 100 = 1{,}228{,}900\) parameters. The first conv layer above does useful work with 896.
2.7 How a CNN is trained
- Same backprop as Part 1. A conv layer is a layer of neurons with shared weights and local connections.
- Shared weights: a kernel weight is used at every position, so its gradient is the sum of the gradients from all positions.
- Max pooling: the gradient goes only to the cell that was the maximum in each window. The other cells get 0. Average pooling splits it equally.
- Transfer learning (Lecture 05): take a network pretrained on a big dataset. With a small dataset similar to the original, freeze the conv layers and retrain only the new classifier head. With more data or a different domain, also fine-tune the top layers with a small learning rate.
- Input 227×227×3. Conv 11×11, 96 filters, stride 4, no padding. Output size and parameters?
- Then max pool 3×3 with stride 2. Output size?
- Input 7×7, kernel 3×3, stride 2, padding 1. Output size?
- Conv 3×3 from 16 channels to 32 channels. Parameters?
- A 1×1 conv from 64 channels to 16. Parameters, and what happens to width and height?
Answers
- \((227-11)/4 + 1 = 55\), so 55×55×96. Parameters \((11\cdot11\cdot3+1)\cdot 96 = 34{,}944\). (This is AlexNet's first layer.)
- \(\lfloor(55-3)/2\rfloor+1 = 27\), so 27×27×96, with 0 parameters.
- \(\lfloor(7-3+2)/2\rfloor + 1 = 4\), so 4×4.
- \((3\cdot3\cdot16+1)\cdot32 = 4{,}640\).
- \((1\cdot1\cdot64+1)\cdot16 = 1{,}040\). Width and height don't change; only the depth goes 64 → 16.
RNNs by hand, from zero
An RNN is one small network applied at every time step, passing a memory \(h\) from each step to the next. Everything below uses scalars so you can do it on paper.
3.1 The idea and the equations
A feed-forward network sees one input and forgets it. For sequences (text, speech, time series) the meaning depends on order and context. An RNN keeps a hidden state \(h_t\) that summarises everything seen so far, and updates it at each step:
- The same weights \(W_{xh}, W_{hh}, W_{hy}\) are used at every time step. That is what lets it handle sequences of any length.
- \(h_0\) is usually 0.
- "Unrolling" means drawing one copy of the network per time step, linked by \(h\). An unrolled RNN is just a deep feed-forward network with shared weights.
| Type | Input → output | Example |
|---|---|---|
| One-to-one | 1 → 1 | Ordinary classification (not really recurrent) |
| One-to-many | 1 → sequence | Image captioning: one image in, a sentence out |
| Many-to-one | sequence → 1 | Sentiment of a review; output only at the last step |
| Many-to-many (same length) | sequence → sequence, aligned | Tagging each word (part of speech, named entities) |
| Many-to-many (encoder-decoder) | sequence → different-length sequence | Machine translation |
3.2 Forward pass over 3 time steps
\(w_x = 0.6,\ w_h = 0.8,\ w_y = 1.5,\ b = 0,\ h_0 = 0\). Input sequence \(x = (1.0,\ 0.5,\ -1.0)\). Output only at the last step, target \(t = 0.5\), \(E = \tfrac12(y_3 - t)^2\).
Notice how \(h_2\) carries the first two inputs forward: without it, step 3 would only see \(x_3 = -1\).
3.3 Backpropagation through time (BPTT)
Unroll the network, then backprop exactly as in Part 1, from step 3 back to step 1. Two new things:
- The error reaches \(h_{k-1}\) through \(h_k\): \(\ \partial E/\partial h_{k-1} = \delta_k \cdot w_h\), where \(\delta_k = \partial E/\partial h_k \cdot (1 - h_k^2)\) (tanh derivative).
- Because \(w_h\) is used at every step, its gradient is the sum over the steps: \(\ \partial E/\partial w_h = \sum_k \delta_k\, h_{k-1}\). Same for \(w_x\): \(\sum_k \delta_k\, x_k\).
| Step k | ∂E/∂h_k | 1 − h_k² | δ_k | δ_k · h_{k−1} (for w_h) | δ_k · x_k (for w_x) |
|---|---|---|---|---|---|
| 3 | −0.978093 | 0.989723 | −0.968041 | −0.602940 | 0.968041 |
| 2 | −0.968041·0.8 = −0.774433 | 0.612064 | −0.474003 | −0.254563 | −0.237001 |
| 1 | −0.474003·0.8 = −0.379202 | 0.711578 | −0.269832 | 0 (h₀ = 0) | −0.269832 |
| Sum = gradient | ∂E/∂w_h = −0.857503 | ∂E/∂w_x = 0.461208 | |||
Update as usual, \(w \leftarrow w - \eta\,\partial E/\partial w\). (These values were checked against a numerical gradient.)
Why RNNs forget: vanishing and exploding gradients
Look at the \(\partial E/\partial h_k\) column: each step back multiplies by \(w_h (1 - h_k^2)\). Here those factors are 0.79, 0.49 and 0.57, so the signal from step 3 shrinks fast. Over 20 steps with a factor around 0.4: \(0.4^{20} \approx 10^{-8}\). The early inputs get almost no gradient, so the network cannot learn long-range dependencies. If the factor is above 1 (say 1.5), it explodes instead: \(1.5^{20} \approx 3{,}325\).
- Exploding: fixed with gradient clipping (cap the gradient's size).
- Vanishing: fixed by changing the cell: LSTM and GRU.
3.4 One LSTM step
An LSTM adds a second memory, the cell state \(c_t\), and three gates. Each gate is a sigmoid between 0 (block) and 1 (let through).
\(x_t = 1,\ h_{t-1} = 0.5,\ c_{t-1} = 0.8\). Weights \((W, U, b)\): forget \((0.5, 0.5, 0)\), input \((1, -1, 0)\), candidate \((0.8, 0.4, 0)\), output \((0.5, 1, -0.5)\).
Why this fixes vanishing gradients: the cell state is updated by addition (\(f \cdot c_{t-1} + \dots\)), not by squashing it through a weight matrix and a tanh at every step. Going back one step, the gradient along the cell path is just multiplied by \(f\). When the forget gate stays near 1, the gradient passes through many steps almost unchanged.
3.5 One GRU step
A GRU merges the cell and hidden state and uses only two gates, so it has fewer parameters and is faster. It usually performs about as well as an LSTM.
Same \(x_t = 1,\ h_{t-1} = 0.5\). Update gate weights \((0.5, 0.5)\), reset \((1, -1)\), candidate \((0.8, 0.4)\).
Some books swap the roles: \(h_t = z_t \odot h_{t-1} + (1-z_t)\odot\tilde h_t\). Both are correct; use the one in your exam statement.
3.6 Parameter counts and the variants
For input size \(x\) and hidden size \(h\), one vanilla RNN cell has \(h\cdot x\) (input weights) \(+\ h\cdot h\) (recurrent weights) \(+\ h\) (bias). An LSTM has 4 such blocks (3 gates + candidate); a GRU has 3.
| Cell | Formula | x = 10, h = 20 |
|---|---|---|
| Vanilla RNN | h(x + h) + h | 200 + 400 + 20 = 620 |
| LSTM | 4 · [h(x + h) + h] | 2,480 |
| GRU | 3 · [h(x + h) + h] | 1,860 |
| Bidirectional (any cell) | 2 × the cell | e.g. BiLSTM 4,960 |
- Bidirectional RNN / BiLSTM: one RNN reads left to right, another right to left, and their hidden states are concatenated. Each position then sees past and future context. It needs the whole sequence up front, so it doesn't work for real-time generation.
- Encoder-decoder: the encoder reads the whole input and its final hidden state becomes a fixed-size context vector. The decoder generates the output from it. The weakness: one fixed vector must hold an entire long sentence (the bottleneck).
3.7 Attention, one step by hand
Attention removes the bottleneck. At each output step, the decoder looks back at all encoder states and takes a weighted average, with weights chosen by relevance.
Encoder states \(h_1 = (1, 0),\ h_2 = (0, 1),\ h_3 = (1, 1)\). Current decoder state (the query) \(s = (1, 2)\). Dot-product scores.
The decoder pays most attention to \(h_3\), the state most aligned with its query. The context \(c\) is then fed into the decoder along with \(s\).
Self-attention (the core of Transformers) does the same thing within one sequence: every position builds a query \(Q\), a key \(K\) and a value \(V\) from its own vector, and attends to every other position:
Dividing by \(\sqrt{d_k}\) keeps the scores from growing with the vector size, which would make the softmax too peaked. Scaled, the example above becomes scores \((0.707, 1.414, 2.121)\), weights \((0.140, 0.284, 0.576)\): a softer spread. Unlike an RNN there's no step-by-step recurrence, so all positions are computed in parallel.
Where marks are lost
- Forgetting that the RNN weights are shared, so the gradient is a sum over time steps.
- Using \(\sigma'\) instead of \(\tanh' = 1 - h^2\) for the hidden state.
- Counting LSTM parameters with 3 blocks instead of 4. The candidate \(\tilde c\) has weights too.
- Saying LSTMs fix exploding gradients. They mainly fix vanishing ones; exploding is handled by clipping.
- Scalar RNN with \(w_x = 1,\ w_h = 0.5,\ b = 0,\ h_0 = 0\), input \(x = (0.5, -0.5)\). Compute \(h_1, h_2\).
- LSTM with input size 50 and hidden size 100. Parameters? And for a GRU?
- Which RNN type for: (a) generating music from a genre label, (b) spam detection on an email, (c) translating French to English?
Answers
- \(h_1 = \tanh(0.5) = 0.462117\); \(\ h_2 = \tanh(-0.5 + 0.5 \cdot 0.462117) = \tanh(-0.268941) = -0.262640\).
- One block: \(100\cdot50 + 100\cdot100 + 100 = 15{,}100\). LSTM: \(4 \times 15{,}100 = 60{,}400\). GRU: \(3 \times 15{,}100 = 45{,}300\).
- (a) one-to-many, (b) many-to-one, (c) many-to-many encoder-decoder (ideally with attention).
Formula sheet
Copy this onto one sheet by hand the night before. Writing it out is part of learning it.
| Topic | Formula |
|---|---|
| Sigmoid | \(\sigma(z) = \frac{1}{1+e^{-z}},\ \ \sigma' = \sigma(1-\sigma)\) (with slope \(a\): \(a\,\sigma(1-\sigma)\)) |
| tanh / ReLU | \(\tanh' = 1 - \tanh^2\); ReLU\('\) = 1 if \(z>0\) else 0 |
| Output delta | \(\delta_o = (o - t)\, f'(net_o)\); with softmax and cross-entropy, \(\delta = p - t\) |
| Hidden delta | \(\delta_h = \big(\sum_k \delta_k\, w_{hk}\big)\, f'(net_h)\), using the old weights |
| Weight gradient | \(\partial E/\partial w_{ij} = \delta_j \cdot (\text{input on that weight})\); bias: \(\delta_j\) |
| Update | \(w \leftarrow w - \eta\, \partial E/\partial w\) |
| Momentum | \(v \leftarrow \beta v - \eta g,\ \ w \leftarrow w + v\) |
| Softmax / CE | \(p_i = e^{z_i}/\sum_j e^{z_j}\); \(L = -\sum t_i \ln p_i\) |
| Conv output size | \(O = \lfloor (W - K + 2P)/S \rfloor + 1\); same padding \(P = (K-1)/2\) |
| Conv parameters | \((K\cdot K\cdot C_{in} + 1)\cdot C_{out}\); pooling has 0 |
| FC parameters | \(n_{in}\cdot n_{out} + n_{out}\) |
| RNN | \(h_t = \tanh(W_{xh}x_t + W_{hh}h_{t-1} + b)\), \(\ y_t = W_{hy}h_t\) |
| BPTT | \(\partial E/\partial W = \sum_t (\dots)\); each step back multiplies by \(W_{hh}\cdot\tanh'\) |
| LSTM | \(c_t = f\odot c_{t-1} + i\odot\tilde c\), \(\ h_t = o\odot\tanh(c_t)\); params \(4[h(x+h)+h]\) |
| GRU | \(h_t = (1-z)\odot h_{t-1} + z\odot\tilde h\); params \(3[h(x+h)+h]\) |
| Attention | scores → softmax → weighted sum; \(\ \text{softmax}(QK^\top/\sqrt{d_k})\,V\) |