Neural networks, from one neuron to attention
A study guide for the whole course. Each lecture opens with a short version, then works through the slides topic by topic, with every worked example recomputed so you can check the numbers.
The network sees only the two coordinates of each point. Each hidden neuron draws one simple boundary, and the output neuron combines them into a curved one. The background shows the prediction at every location: blue means class 1, rose means class 0.
Each square is one neuron, painted with its own output over the whole input plane. First-layer neurons can only draw straight boundaries; later layers combine them. Lines are weights: blue positive, rose negative, thicker means larger. Lecture 3 calls the hidden layers feature detectors; this is what that looks like.
Try it: on XOR, set one hidden layer with 1 neuron. A single neuron draws one straight line, so accuracy sticks near 75% (Lecture 2). More neurons let the output combine several lines (Lecture 3): two reach about 90%, two layers of four reach 100%. The spiral is harder: switch to ReLU with 3 layers of 8 neurons and \(\eta = 0.01\). Then press New weights at \(\eta = 0.3\): the steps are too large, ReLU neurons switch off, and the network sticks at 50% (Lecture 4).
The course in seven steps
The lectures build on one another. Read the list as a story: each step fixes a limit of the one before it.
- 01
Introduction to machine learning
Learning from labelled data: types of learning, the seven-step pipeline, features, distances, normalization and overfitting. 74 slides.
- 02
Neurons and the perceptron
A neuron is a weighted sum plus a bias, passed through an activation. The perceptron learns its weights, but only draws a straight line. 28 + 31 slides.
- 03
Multilayer perceptron and backpropagation
Hidden layers bend the boundary (XOR becomes solvable). Backpropagation sends the error backward to train every weight. 58 slides.
- 04
Cost functions and optimization
Choosing the loss, the batch size, the optimizer, the activation and the initial weights, and fighting overfitting with L1, L2 and dropout. 52 slides.
- 05
Transfer learning
Training is expensive, so reuse a network that already learned edges and shapes, and retrain only its last layers. 16 slides.
- 06
Convolutional neural networks
Dense layers waste parameters on images. Small shared filters slide over the image instead, followed by ReLU and pooling. 68 slides.
- 07
Recurrent networks and attention
For sequences, a network keeps a memory. LSTM and GRU gates protect it, and attention lets the model look back at any position. 63 slides.
Introduction to machine learning
What it means for a computer to learn, the kinds of learning, and the pipeline every project follows: collect data, describe it with features, pick a model, train, evaluate, tune and deploy.
- Machine learning is the part of AI where computers learn from past data to make future decisions, without being explicitly programmed. Deep learning is the part of ML that uses neural networks with many layers.
- Supervised learning uses labelled data (classification or regression). Unsupervised learning finds structure in unlabelled data (clustering). Semi-supervised mixes both; reinforcement learning learns from rewards.
- Each sample becomes a feature vector, a point in feature space. A good representation has small intra-class variation, large inter-class separation and a simple decision boundary.
- Normalize features so that no single feature dominates a distance. Compare patterns with metric distances (L1, L2, L∞, Mahalanobis) or the edit distance for strings.
- More features need more data (curse of dimensionality). A model that fits the training set too closely overfits; keep separate validation and test sets.
What machine learning is
The slides give two definitions. Machine learning is the ability of computers to learn from past data in order to make decisions in the future without being explicitly programmed. Put more simply, it is about learning to do better in the future based on what was experienced in the past, by designing efficient and accurate prediction algorithms.
The standard example is a spam filter. Instead of writing rules by hand, you give the algorithm many emails already labelled spam or not spam, and it learns a rule that classifies new emails.
The three terms nest: artificial intelligence contains machine learning, and machine learning contains deep learning. Deep learning uses neural networks with many layers to model complex relationships in data.
Types of learning
| Type | Training data | What it does | Examples |
|---|---|---|---|
| Supervised | Labelled | Learns a mapping from input to label, then predicts labels for new data. Two kinds: classification (a category) and regression (a number). | Linear and logistic regression, decision trees, random forests, SVM, neural networks |
| Unsupervised | Unlabelled | Finds patterns on its own. Clustering and dimensionality reduction are the common tasks. Useful when labels cannot be obtained. | K-Means, hierarchical, DBSCAN, spectral, fuzzy clustering |
| Semi-supervised | Partly labelled | Learns from a few labels plus many unlabelled samples. Useful when labels are scarce or expensive. | Label propagation, self-training |
| Reinforcement | Rewards and punishments | An agent acts in an environment and learns to make better decisions over time. | Game playing, robotics, autonomous vehicles |
The slides add three types used heavily today:
- Deep learning: many-layer networks. CNNs for images and video, RNNs for sequences such as speech and text, LSTMs for long-term dependencies, GANs for generating data, autoencoders for feature learning and compression.
- Transfer learning: start from a model pre-trained on one task and fine-tune it for a new one. It needs less data and less time (Lecture 5).
- Online learning: update the model as new data arrives, for example in online advertising or e-commerce. Training once and then deploying is offline learning.
Applications shown in the lecture
- Vision: object detection, image classification, face recognition, self-driving cars, colouring black-and-white images, semantic segmentation.
- Handwriting: digit and character recognition, for example sorting letters at post offices. Writer recognition identifies who wrote a text.
- Licence plates: acquisition, enhancement, segmentation, then character recognition, all in real time.
- Diagnosis: computer-aided diagnosis highlights suspicious regions for breast, lung, colon and prostate cancer, bone metastases, coronary artery disease and congenital heart defects.
- Speech: recognizing what is said (dictating a document) and who is speaking.
The machine learning pipeline
Every project goes through the same seven steps. This list answers the common question “describe the steps of a machine learning system”.
- 1Data collectionDatabases, APIs, web scraping or sensors. Large amounts of data matter, but sensors are noisy and collecting data can be hard.
- 2Data preprocessingCleaning, filtering, normalization and feature engineering. Feature engineering covers feature extraction and feature selection.
- 3Model selectionChoose an algorithm suited to the problem, often by comparing several on the data.
- 4TrainingFeed the data in and adjust the parameters to minimize the error between predictions and true labels.
- 5EvaluationMeasure performance on a test set of unseen data: accuracy, precision, recall, F1-score.
- 6Hyperparameter tuningGrid search, random search or other methods to find the best settings.
- 7DeploymentPut the model into production, for example behind an API or inside an existing system.
The lecture follows one example through these steps: a camera on a conveyor belt must sort salmon from sea bass. Which features should be measured (length, width, height, skin lightness)? Are all of them useful? How should the classes be separated, and how accurate is the result? With length alone, the two histograms overlap a lot. Lightness alone overlaps less. Using both features gives a 2D space where a simple line separates the classes well.
Salmon or sea bass?Pick a feature, move the decision threshold, and count the misclassified fish.Open demoClose
200 synthetic fish (100 of each). The slide figures use the same idea: overlap between class histograms means unavoidable errors for that feature.
Features and representation
A feature is a measurable quantity that makes classes distinct, for example the mean intensity of an image region and its standard deviation. The features of one sample form a feature vector \(F = [f_1, f_2, \dots, f_l]\) of dimension \(l\), treated as a random vector. Each sample becomes a point in an \(l\)-dimensional feature space.
A good representation has three properties:
- Small intra-class variation: samples of the same class stay close together, even across fonts, poses, expressions or lighting.
- Large inter-class separation: different classes stay apart, even for identical twins or characters that look alike.
- A simple decision boundary: ideally a straight line or plane.
The slides also ask for invariance to rotation, translation and scale, robustness to noise and occlusion, and a low measurement cost so the system can run in real time.
The first feature vector is rarely the best one: it can contain useless or redundant features. Two fixes:
- Feature selection keeps the best subset of the original features.
- Feature extraction transforms the feature vector into a lower-dimensional space (PCA is the usual example).
Normalization
Features measured on different scales do not count equally in a distance. In the slide example, the first feature stays below 18 and the second goes up to 1121, so the distance is decided almost entirely by the second feature.
The slide’s fix is to divide each feature by its maximum: \(f_1\) by 18 and \(f_2\) by 1121. Both features then lie between 0 and 1. Lecture 4 adds min-max, z-score, robust and log scaling.
Which feature decides the distance?The five slide points before and after scaling, with each feature’s share of the squared distance.Open demoClose
Distances
Many classifiers compare a new pattern with known patterns or class models. A distance measures dissimilarity: lower means more similar. A distance is metric when it satisfies four properties:
The common metric distances all come from the Minkowski distance of order \(m\):
- Manhattan (L1), \(m = 1\): the sum of absolute differences.
- Euclidean (L2), \(m = 2\): the straight-line distance, \(\sqrt{\sum_k (x_k-y_k)^2}\).
- L∞: the largest single difference, \(\max_k |x_k-y_k|\).
- Mahalanobis: \(d(X,Y)^2=(X-Y)^\top\Sigma^{-1}(X-Y)\). It accounts for the covariance \(\Sigma\) of the features.
Slide example. Pattern \(p\) has \(F_1 = (4, 1, 3)\). Compare it with \(F = (2, 5, 1)\) and \(F' = (4, 6, 2)\):
\(p\) is closer to \(F\) than to \(F'\).
Weighted distance. Weights give some features more importance:
With \(w = (0.3, 0.6, 0.1)\), feature 2 matters most, then feature 1, then feature 3. The slide computes \(\sqrt{0.3\cdot 4+0.6\cdot 16+0.1\cdot 4}=\sqrt{11.2}=3.35\).
The slide writes \(X = (4, 2, 3)\) but computes with \(x_2 = 1\), the same pattern as the previous example. With \(x_2 = 2\) the answer would be \(\sqrt{7.0} = 2.65\).
Distance calculatorChange the order \(m\) and the weights, and see which pattern is nearer.Open demoClose
Edit (Levenshtein) distance
For strings, the edit distance is the smallest number of operations (substitute, insert or delete one character) that turns \(s_1\) into \(s_2\). It is computed by recurrence, where \(\varepsilon\) is the empty string, starting from \(d(\varepsilon, \varepsilon) = 0\) and \(d(s, \varepsilon) = d(\varepsilon, s) = |s|\):
TRAIN → BRAIN needs 1 substitution. TRAIN → CRANE needs 3 (T→C, I→N, N→E).
Edit distance tableType two words and watch the dynamic-programming table and the cheapest sequence of edits.Open demoClose
Challenges, overfitting and data splits
- Representation: which features or model capture large intra-class variability (the letter T in many typefaces, a face under different poses and lighting)?
- Curse of dimensionality: as the number of input dimensions grows, the algorithm needs more data to generalize well.
- Choice of method, matching and evaluation. A simple boundary usually generalizes better than a complex one that bends around every training point. The slide also asks about the cost of each kind of misclassification.
- Testing: you need a test set that was not used for training, which reduces the data left to train on.
Overfitting is the classic trap: as model complexity grows, training error keeps falling but error on new data starts rising. The validation set detects the turning point; the test set gives the final, unbiased score.
| Split | Large data set | Small data set |
|---|---|---|
| Training | 50% | 60% |
| Validation | 25% | 20% |
| Testing | 25% | 20% |
Underfitting and overfittingFit polynomials of growing degree to 12 noisy points and watch training and validation error part ways.Open demoClose
Exam traps
- Classification predicts a category; regression predicts a continuous value. Both are supervised.
- Feature selection keeps a subset of the original features; feature extraction builds new, fewer features from them.
- The L2 distance squares the differences, so one large difference dominates; L∞ keeps only the largest. Normalize first when features have different scales.
- The edit distance counts substitutions, insertions and deletions, each with cost 1.
- The test set is never used to choose hyperparameters; that is the validation set’s job.
Neurons and the perceptron
An artificial neuron multiplies its inputs by weights, adds a bias and passes the sum through an activation function. The perceptron learns those weights from data, but it can only draw a straight line.
- A neuron computes \(v=\sum_j w_j x_j + b\) and outputs \(y=\varphi(v)\). Treating the bias as a weight on a constant input \(x_0 = +1\) gives \(v=\mathbf w^\top\mathbf x\).
- The McCulloch–Pitts neuron (1943) has binary inputs, a fixed threshold and no learning. It can build AND, OR and NOT gates, but not XOR.
- Rosenblatt’s perceptron (1958) learns: when a sample is misclassified, add \(\eta\,\mathbf x\) (should be positive) or subtract \(\eta\,\mathbf x\) (should be negative). It is guaranteed to converge only if the classes are linearly separable.
- Gradient descent minimizes any differentiable error by stepping against the gradient: \(w \leftarrow w - \eta\,\nabla E\). The learning rate \(\eta\) decides whether it crawls, converges or overshoots.
- Updates can be done per sample (stochastic), per small batch (mini-batch) or once per pass over the whole set (batch).
From the brain to an artificial neuron
Neural networks are inspired by the brain. The brain has about \(10^{11}\) neurons that work in parallel, and each connects to around \(10^4\) others through synapses. That parallelism and connectivity are why the brain recognizes a scene in roughly 100–200 ms.
An artificial neural network (ANN) is a massive set of parallel, distributed processing units that can store experiential knowledge. It resembles the brain in two ways:
- Knowledge is acquired from the environment through a learning phase.
- The connection strengths between neurons, the weights, store that knowledge.
The slide writes “\(10^{11}\) = 10 milliard”. \(10^{11}\) is 100 billion (100 milliard); 10 milliard would be \(10^{10}\). Quote \(10^{11}\).
The model of a neuron
Neuron \(k\) receives inputs \(x_1, \dots, x_m\) through weights \(w_{k1}, \dots, w_{km}\). It sums them, adds a bias \(b_k\), and applies an activation function \(\varphi\):
The bias can also be treated as one more weight, \(w_{k0}=b_k\), on a constant input \(x_0=+1\). The sum then starts from zero, \(v_k=\sum_{j=0}^{m}w_{kj}x_j\), and in vector form \(y=\varphi(\mathbf w^\top\mathbf x)\) with \(\mathbf x=[1,x_1,\dots,x_d]^\top\).
| Activation | \(\varphi(v)\) | Range | \(\varphi'(v)\) |
|---|---|---|---|
| Identity (linear) | \(v\) | \((-\infty, \infty)\) | \(1\) |
| Binary step (threshold) | \(1\) if \(v\ge 0\), else \(0\) | \(\{0, 1\}\) | \(0\), undefined at 0 |
| Sigmoid (logistic) | \(\dfrac{1}{1+e^{-v}}\) | \((0, 1)\) | \(\varphi(v)\,(1-\varphi(v))\) |
| Tanh | \(\dfrac{e^{v}-e^{-v}}{e^{v}+e^{-v}}\) | \((-1, 1)\) | \(1-\varphi(v)^2\) |
| ReLU | \(\max(0,v)\) | \([0, \infty)\) | \(1\) if \(v>0\), else \(0\) |
One-input example (slide 11). With weight \(w = 3\), input \(p = 2\) and bias \(b = -1.5\), the net input is \(n = 3\cdot 2 - 1.5 = 4.5\), so the output is \(a = f(4.5)\). With a sigmoid that is 0.989; with a step it is 1; with ReLU it is 4.5.
Multiple inputs. For a layer of neurons, the weights form a matrix \(W\) and the biases a vector \(\mathbf b\): \(\mathbf n = W\mathbf p + \mathbf b\), \(\mathbf a = f(W\mathbf p+\mathbf b)\).
The neuron in input space. Setting \(w_0 + w_1x_1 + w_2x_2 = 0\) gives a straight line (a plane or hyperplane in more dimensions). Points on one side give a positive sum, points on the other a negative one. Training a single neuron means placing that line so each class lands on its own side.
One neuron, liveMove inputs, weights and bias; see the weighted sum, the activation, and the line the neuron draws.Open demoClose
Bottom chart: the input plane. The shading is the neuron’s output everywhere; the line is where \(v = 0\). The dot is the current input.
Networks as directed graphs
A network is drawn as a directed graph, called its architectural graph. It is fully connected (dense) when every neuron in layer \(n\) connects to every neuron in layer \(n + 1\), and partially connected when each neuron connects only to some of them, as in CNNs and RNNs. A single-layer network has only an output layer; a multilayer network adds hidden layers in between.
| Architecture | Typical use |
|---|---|
| Feedforward (MLP) | Classification and regression on structured data |
| RNN | Sequences where order and context matter: NLP, time series, speech |
| CNN | Images and video: recognition, detection, segmentation |
| LSTM | Like RNNs but better at long sequences and long-term dependencies: translation, speech |
| Transformer | Attention captures global dependencies: translation, summarization, language models (BERT, GPT), vision transformers |
| GAN | A generator and a discriminator compete in a zero-sum game: image generation, style transfer, data augmentation |
| Autoencoder | Encoder compresses to a latent space, decoder reconstructs: dimensionality reduction, denoising, anomaly detection |
| GNN | Graph-structured data: social networks, molecules, recommendation |
| Capsule network | Models part–whole spatial hierarchies better than plain CNNs |
Designing a network: handwritten digits
For digit recognition, each pixel is one input feature, so the input dimension equals the number of pixels. The output layer has 10 neurons, one per digit. The training set must cover a large, representative variety of handwriting.
Prior knowledge can be built into the design in two ways, both used later by CNNs:
- Receptive fields: each neuron connects only to a small local region of the input (local connections).
- Weight sharing: several neurons use the same weights, which constrains the network and cuts the parameter count.
Building in invariance
A detector should still work when the object is rotated, shifted, scaled or recoloured. Three ways:
| Approach | How | Trade-off |
|---|---|---|
| By structure | Wire the weights so transformed versions of an input give the same output | The number of connections becomes prohibitive even for moderate images |
| By training | Show the network transformed copies: data augmentation (rotate, scale, translate) | Computational cost; risk of overfitting |
| By feature space | Feed features that are already invariant to the transformations | Fewer features, relaxed demands on the network, guaranteed invariance; but needs careful feature engineering |
Learning with a teacher
Learning is either with a teacher (supervised or reinforcement) or without one. With a teacher, samples are fed in, the error between prediction and true label is computed, and the weights are updated backward in proportion to their share of the error, until the error stops improving. Updates follow the gradient and come in three schedules:
- Batch: one update after processing the whole batch of samples.
- Stochastic: one update after each individual sample.
- Mini-batch: one update after each small subset.
A short history
- 1943 · McCulloch & PittsThe first mathematical neuron: binary inputs, a threshold, no learning.
- 1958 · Rosenblatt’s perceptronA neuron that learns its weights from labelled examples.
- 1969 · Minsky & PapertA single perceptron cannot learn XOR. Interest in neural networks collapses.
- 1986 · BackpropagationRumelhart, Hinton and Williams popularize training multilayer networks.
- 1998 · LeNetLeCun’s convolutional network reads handwritten digits and cheques.
- 2012 · AlexNetA deep CNN trained on GPUs wins ImageNet by a wide margin; deep learning takes off.
- 2017 · Transformer“Attention is all you need”: attention replaces recurrence, the basis of today’s language models.
The McCulloch–Pitts neuron and logic gates
Warren McCulloch (a neuroscientist) and Walter Pitts (a logician) proposed the first computational neuron in 1943. Inputs and output are Boolean, the activation is a threshold, and nothing is learned: the weights and threshold are set by hand. It is a theoretical model, but it already computes AND, OR and NOT.
The neuron fires when the weighted sum reaches the threshold \(T\), which is the slide’s “\(y=1\) if \(v\ge 0\)” with the bias \(b=-T\):
To design a gate, write one inequality per row of the truth table and solve. For AND, (1, 1) must fire and the other rows must not, which gives \(w_1 = w_2 = 1\), \(T = 2\).
| Gate | \(w_1\) | \(w_2\) | \(T\) | Check |
|---|---|---|---|---|
| AND | 1 | 1 | 2 | only \(1 + 1 = 2\) reaches 2 |
| OR | 1 | 1 | 1 | any single 1 reaches 1 |
| AND-NOT (\(x_1\) and not \(x_2\)) | 2 | −1 | 2 | only (1, 0) gives 2 |
| NOT (one input) | −1 | – | 0 | \(0 \ge 0\) fires; \(-1\) does not |
| NOR | −1 | −1 | 0 | only (0, 0) reaches 0 |
| NAND | −1 | −1 | −1 | fails only for (1, 1): \(-1 - 1 = -2\) |
| XOR | – | – | – | impossible with one neuron |
XOR is impossible because its two firing points, (0, 1) and (1, 0), sit on opposite corners. No single line separates them from (0, 0) and (1, 1). Solving it needs a hidden layer (Lecture 3).
Other weight choices also work: any line that puts the firing corners on one side is correct. The slide figures for NOT, NOR and NAND use a bias with a strict “fires if \(v > 0\)” rule, which is the same idea.
Gate designerChoose a gate, tune the weights and threshold, and check the truth table. Then try XOR.Open demoClose
McCulloch–Pitts versus Rosenblatt
| McCulloch–Pitts (1943) | Rosenblatt perceptron (1958) | |
|---|---|---|
| Purpose | Theoretical model of a biological neuron | Practical single artificial neuron |
| Inputs | Binary (0 or 1) | Real-valued |
| Weights | All identical | Different, positive or negative |
| Threshold | Single, fixed | Single, adjusted during learning |
| Activation | Step | Step |
| Learning rule | None | Perceptron learning rule |
| Used for | Theoretical foundation of ANNs | Simple pattern recognition, linear binary classification |
| Strength | Simple model that laid the groundwork | Introduced learning; enabled practical use |
| Weakness | Binary only, no learning, not for non-separable problems | Only linearly separable problems; can converge slowly |
The perceptron learning rule
If the two classes are linearly separable, there is a weight vector \(\mathbf w\) with \(\mathbf w^\top\mathbf x>0\) for every \(\mathbf x\) in class 1 and \(\mathbf w^\top\mathbf x\le 0\) for every \(\mathbf x\) in class 2 (the boundary case goes to class 2). The algorithm visits the samples one at a time:
- If sample \(\mathbf x(n)\) is classified correctly, keep the weights: \(\mathbf w(n+1)=\mathbf w(n)\).
- If \(\mathbf w^\top\mathbf x>0\) but \(\mathbf x\) belongs to class 2, subtract: \(\mathbf w(n+1)=\mathbf w(n)-\eta\,\mathbf x(n)\).
- If \(\mathbf w^\top\mathbf x\le 0\) but \(\mathbf x\) belongs to class 1, add: \(\mathbf w(n+1)=\mathbf w(n)+\eta\,\mathbf x(n)\).
With labels \(d(n) = +1\) or \(-1\), both corrections combine into one error-correction rule, where \(y(n)\) is the perceptron’s \(\pm 1\) output. On a mistake, \(d - y = \pm 2\), so this is the same rule with the step doubled:
Convergence theorem. If the classes are linearly separable, the perceptron finds a separating line in a finite number of steps. If they overlap or need a curved boundary, it never settles.
Learning rate. A constant \(\eta > 0\) gives the fixed-increment rule. Other schedules: a constant rate (simple, fine for easy problems), a decreasing rate (smooth convergence, fewer oscillations), an adaptive rate (adjusts itself from the gradients, as Adam does), and cyclic or warm-up schedules.
The batch perceptron
The rule above has no cost function and corrects one sample at a time. The batch version defines the perceptron cost over the set \(\mathcal H\) of misclassified samples. It is zero when nothing is misclassified:
Steepest descent moves against the gradient, so each step adds up all misclassified samples at once:
Worked exercise (slides 19–21)
Data: \(a = (0, 1)\) class −, \(b = (2, 0)\) class −, \(c = (1, 1)\) class +. The perceptron outputs + when \(s = w_0 + w_1x_1 + w_2x_2 > 0\).
(a) By hand. A line that separates the points has slope \(-\tfrac12\) and crosses the \(x_2\) axis at \(\tfrac54\): \(x_2 = \tfrac54 - \tfrac{x_1}{2}\), i.e. \(2x_1 + 4x_2 - 5 = 0\). Taking \(c\)’s side as positive gives \(w_0 = -5,\ w_1 = 2,\ w_2 = 4\). Alternatively, take the normal vector \((1, 2)\) pointing to the positive point; then \(w_0 = 1\cdot(-0.5) + 2\cdot(-1) = -2.5\), giving \((-2.5, 1, 2)\). The two answers differ only by a factor of 2, which does not change the line.
(b) The learning algorithm with \(\eta = 1\) and initial \(\mathbf w = (-1.5, 0, 2)\) visits \(a, b, c\) in order and converges after 12 iterations to \((-2.5, 1, 2)\). Step through it below.
Perceptron stepperReplay the slide exercise one update at a time, or watch the rule fail forever on XOR.Open demoClose
Gradient descent
Imagine being lost in mountains and looking for the valley. Gradient descent feels the slope underfoot and takes a small step downhill, again and again. Applied to a model:
- 1Picture the error as a landscapeEach choice of weights gives an error; high points are bad weights.
- 2Start at a random pointThe initial guess for the weights.
- 3Compute the slope (gradient)How much, and in which direction, the error changes when each weight moves a little.
- 4Step downhill\(w_{\text{new}} = w_{\text{old}} - \eta\,\dfrac{\partial E}{\partial w}\). The derivative gives the direction; \(\eta\) gives the step size.
- 5RepeatUntil the slope is almost zero: a minimum.
Variants: batch gradient descent uses the whole data set per step (slow for large sets); stochastic uses one sample (fast but noisy); mini-batch uses a small batch (a balance of speed and stability).
Limits: it can get stuck in a local minimum (a good start or momentum helps); it needs a differentiable function; it is sensitive to \(\eta\). Too high overshoots the minimum, too low barely moves.
Learning rate labRun 30 steps of gradient descent on a bowl and on a two-valley curve, from any start and learning rate.Open demoClose
| Gradient descent | Perceptron rule | |
|---|---|---|
| Objective | Any differentiable function | Misclassification error (binary) |
| Update | Proportional to the gradient, every step | Only when a prediction is wrong |
| Convergence | Not guaranteed (local minima) | Guaranteed if linearly separable |
| Per iteration | All weights at once | Only for misclassified inputs |
| Applications | Regression, multi-class, deep networks | Simple binary classification |
Both are iterative methods that minimize an error by adjusting weights, and both handle non-linear problems only when combined with feature engineering or hidden layers.
Exam traps
- The bias is a weight on a constant input of +1. Forgetting it forces every line through the origin.
- In the perceptron exercise, augment each sample with \(x_0 = 1\) before adding or subtracting: \(c = (1, 1)\) becomes \((1, 1, 1)\).
- The perceptron updates only on mistakes; gradient descent updates every step. Only the perceptron has a convergence guarantee, and only for linearly separable data.
- Read the firing rule carefully: “\(\ge T\)” and “\(> 0\) with bias” lead to different but equally valid numbers.
- XOR needs at least two layers. A single neuron, whatever its activation, draws one straight line.
Multilayer perceptron and backpropagation
Hidden layers turn straight boundaries into curved ones, so XOR becomes solvable. Backpropagation trains every weight by sending the output error backward through the network with the chain rule.
- An MLP has an input layer, one or more hidden layers and an output layer, each fully connected to the next. MLPs are universal approximators.
- Hidden neurons are feature detectors: they map the input into a new space where the classes are easier to separate. That is how a 2-2-1 network solves XOR.
- Training minimizes \(E=\tfrac{1}{2}\sum (t-o)^2\) by gradient descent. Activations must be differentiable, which is why the step function is replaced by the sigmoid.
- Output neuron: \(\delta = (t-o)\,f'(u)\). Hidden neuron: \(\delta = f'(u)\sum_k w_k\,\delta_k\). Every weight then moves by \(\Delta w=\eta\,\delta\,x\).
- Function signals flow forward; error signals flow backward. One forward pass plus one backward pass gives the gradient for every weight.
Is one perceptron enough?
A perceptron fits a data set with zero training error only when one straight line separates the classes. Concentric circles, XOR and spirals all defeat it. The fix is to stack layers.
A multilayer perceptron is a feedforward network organized in layers:
- one input layer of distribution points (it computes nothing, it only passes the inputs on);
- one or more hidden layers of neurons;
- one output layer of neurons.
Each node connects to every node of the next layer, and each connection has a weight (which may be zero). With enough hidden neurons, MLPs are universal approximators: they can approximate any continuous function as closely as you like.
| Threshold network | Decision regions it can form |
|---|---|
| No hidden layer (a perceptron) | A half-plane bounded by one line |
| One hidden layer | Convex regions: the intersection of several half-planes |
| Two hidden layers | Arbitrary regions, including non-convex and disconnected ones |
The XOR network
XOR outputs 1 when exactly one input is 1. Two hidden threshold neurons and one output neuron solve it:
- \(h_1\) = OR: weights (1, 1), bias −0.5. Fires unless both inputs are 0.
- \(h_2\) = NAND: weights (−1, −1), bias 1.5. Fires unless both inputs are 1.
- \(y = \text{AND}(h_1, h_2)\): weights (1, 1), bias −1.5. Fires only when both hidden neurons fire.
In hidden space the four inputs land on only three points: (0, 0) → (0, 1); (0, 1) and (1, 0) → (1, 1); (1, 1) → (1, 0). Now one line separates the class-1 point from the rest, so the output neuron can finish the job. This is the feature-space idea in its smallest form.
XOR in two spacesSee the two hidden lines in input space, and how hidden space makes XOR linearly separable.Open demoClose
Select a row to follow one input through the network. Blue points have XOR = 1, rose points XOR = 0.
Backpropagation
Updating weights after seeing an input \(x\) changes how the network treats other inputs too, including ones it has never seen. Handling unseen inputs correctly is generalization. Training needs an error function that measures the gap between targets \(t\) and outputs \(o\) over the \(N\) training patterns (with \(m\) output neurons):
Gradient descent needs the partial derivative of \(E\) with respect to every weight, holding the others fixed. The learning rule is then:
A positive derivative means \(E\) grows as the weight grows, so the weight should decrease, and vice versa. \(\eta\) says how far to move.
With a sigmoid, neuron \(i\)’s output for pattern \(p\) is \(o_i=f(u_i)=\frac{1}{1+e^{-a u_i}}\), where \(u_i=\sum_j w_{ij}x_{ij}\) is its weighted input. The derivative is easy to compute from the output itself: \(f'(u_i)=a\,o_i(1-o_i)\). The step function has no useful derivative, which is why it cannot be used here.
This gives the generalized delta rule. For an output neuron, the target is known:
A hidden neuron has no target. It receives a share of the error of every neuron \(k\) it feeds, weighted by the connection:
- 1Forward passCompute every neuron’s weighted input \(u\) and output \(o\), layer by layer.
- 2ErrorCompare the outputs with the targets: \(E=\tfrac12\sum(t-o)^2\).
- 3Output deltas\(\delta=(t-o)\,f'(u)\) for each output neuron.
- 4Hidden deltasPropagate backward: \(\delta=f'(u)\sum_k w_k\delta_k\), using the weights before they are updated.
- 5Update\(w \leftarrow w+\eta\,\delta\,x\) for every weight, where \(x\) is the input that weight carries.
- 6RepeatFor every pattern and over many epochs, until a stopping criterion is met (Lecture 4).
Sign conventions differ between slides and books. Writing \(\delta=(t-o)f'\) and adding \(\eta\delta x\) is the same as computing \(\partial E/\partial w=-(t-o)f'x\) and subtracting \(\eta\,\partial E/\partial w\). Pick one and stay consistent.
The chain rule behind it
Take one sigmoid neuron with input \(x\), weight \(w\) and bias \(b\): \(u=wx+b\), \(o=f(u)\), and the error for one pattern \(E=\tfrac12(t-o)^2\). A change in \(w\) ripples through \(u\), then \(o\), then \(E\), so the chain rule multiplies the three local derivatives:
So \(\Delta w=-\eta\,\partial E/\partial w=\eta\,\delta\,x\): the delta rule. The bias works the same way, because \(\partial u/\partial b=1\). The derivative with respect to the input uses the weight instead, and it is what gets handed back to the neuron that produced \(x\):
When \(x\) is the output \(o_i\) of a hidden neuron \(i\), it reaches the error through every neuron \(k\) of the next layer, so its derivative is a sum over paths. Multiplying by its own slope \(f'(u_i)\) gives the hidden-layer rule from above:
Chain rule, one neuronCheck each derivative against a real nudge, then take gradient steps by hand.Open demoClose
“Nudge check” changes the parameter by 0.001 and measures how much \(E\) really changes. It matches the chain-rule value, which is the whole point: backpropagation computes exactly these slopes, only for thousands of weights at once.
The three worked examples
The lecture runs backpropagation by hand on three small networks. Biases are not updated in any of them. The calculator below reproduces every number and shows each multiplication.
| Example 1 | Example 2 | Example 3 | |
|---|---|---|---|
| Network | 2-2-1, linear, no bias | 2-2-2, sigmoid, biases \(b_1 = 0.25\), \(b_2 = 0.35\) | 2-2-2, sigmoid, biases \(b_1 = 0.35\), \(b_2 = 0.60\) |
| Inputs | (2, 3) | (0.1, 0.5) | (0.05, 0.10) |
| Targets | 1 | (0.05, 0.95) | (0.01, 0.99) |
| \(\eta\) | 0.05 | 0.6 | 0.5 |
| Hidden outputs | 0.85, 0.48 | 0.60108, 0.61538 | 0.593270, 0.596884 |
| Outputs | 0.191 | 0.73492, 0.77955 | 0.751365, 0.772928 |
| Error \(\tfrac12\sum(t-o)^2\) | 0.327 | 0.24908 | 0.298371 |
| After one update | output 0.26 | \(w_5 = 0.45188\), …, \(w_8 = 0.81082\) | error 0.291028 |
The examples label weights differently. In Example 2, \(w_2\) goes from \(i_1\) to \(h_2\) and \(w_3\) from \(i_2\) to \(h_1\); in Example 3, \(w_2\) goes from \(i_2\) to \(h_1\) and \(w_3\) from \(i_1\) to \(h_2\). The calculator keeps each example’s own labels. Slide 40 is titled “Example 3” but belongs to Example 2.
Backpropagation calculatorLoad any of the three slide examples or type your own numbers. Every forward, backward and update value is shown with its arithmetic.Open demoClose
Example 3 continues on the last slide: after the first round the error falls from 0.298371 to 0.291028. After 10,000 rounds it is about 0.000035, with outputs 0.0159 and 0.9841 against targets 0.01 and 0.99. The calculator’s “Train 10,000×” button reproduces this.
Exam traps
- Use the old output-layer weights when computing hidden deltas. Update all weights only after every delta is computed.
- The sigmoid derivative is \(o(1 - o)\), computed from the neuron’s output, not from its weighted input.
- For a linear neuron \(f' = 1\), so \(\delta\) is simply the error (Example 1).
- A hidden neuron that feeds two outputs collects error from both: \(\partial E/\partial\,\mathrm{out}_h\) sums, over the outputs, each output’s error term times its weight.
- The input layer has no neurons and no weights of its own; “a 2-2-1 network” has \(2\times 2 + 2\times 1 = 6\) weights (plus biases).
Cost functions and optimization
The practical side of training: which loss to minimize, how often to update, which optimizer to use, how to prepare inputs and initialize weights, when to stop, and how to keep the network from memorizing the training set.
- Use MSE or MAE for regression, cross-entropy (with softmax) for multi-class classification and binary cross-entropy (with a sigmoid) for two classes.
- Online and stochastic learning update after every sample; batch learning after the whole set; mini-batch after small subsets, the usual compromise.
- Momentum adds a fraction of the previous step to damp oscillations. Adam keeps running averages of the gradient and its square to give every weight its own step size (defaults \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\eta = 0.001\)).
- Normalize the inputs, and initialize weights neither too small nor too large (Xavier for sigmoid/tanh, He for ReLU). Otherwise gradients vanish or explode.
- Stop early when the validation error starts rising. Regularize with L1 (sparse weights), L2 (weight decay), more data or dropout.
Objective (loss) functions
An objective function, also called a loss or cost function, measures the difference between the network’s outputs and the desired outputs. It quantifies the prediction error and guides training: the whole aim is to minimize it.
| Loss | Formula | Used for |
|---|---|---|
| Mean squared error | \(\frac1n\sum_i (y_i-\hat y_i)^2\) | Regression |
| Mean absolute error | \(\frac1n\sum_i |y_i-\hat y_i|\) | Regression; less sensitive to outliers |
| Cross-entropy (log loss) | \(-\frac1n\sum_i\sum_c y_{i,c}\log\hat y_{i,c}\) | Multi-class classification, with softmax outputs and one-hot labels |
| Binary cross-entropy | \(-\frac1n\sum_i\big[y_i\log\hat y_i+(1-y_i)\log(1-\hat y_i)\big]\) | Two classes, one sigmoid output |
| Sparse categorical CE | \(-\frac1n\sum_i\log\hat y_{i,\,y_i}\) | Same as cross-entropy, with labels given as class indices instead of one-hot vectors |
| KL divergence | \(\sum_x P(x)\log\frac{P(x)}{Q(x)}\) | How far a predicted distribution \(Q\) is from a target distribution \(P\) |
With one-hot labels, cross-entropy keeps only one term: minus the log of the probability given to the correct class. A confident wrong answer (\(p\) close to 0) costs a lot, which pushes training harder than MSE does.
Softmax and cross-entropyMove the raw scores, see the probabilities, and compare how cross-entropy and MSE punish mistakes.Open demoClose
When to update: online, stochastic, batch, mini-batch
| Approach | Update after | Strengths | Weaknesses |
|---|---|---|---|
| Online | Each sample | Little storage, exploits redundancy, tracks small changes in the data, simple; useful for large sets | Noisy, sequential |
| Stochastic | Each sample, in random order | Noise can shake the network out of local optima | Less stable than the others |
| Batch | All samples | Computationally efficient (parallel), smoother convergence | One update per epoch; memory-hungry |
| Mini-batch | A small subset | Efficient like batch, but with frequent, finer-grained updates | Batch size becomes one more hyperparameter |
# online / stochastic backpropagation for epoch in 1..E: shuffle(training_set) # stochastic: random order each epoch for (x, t) in training_set: forward(x); compute δ for every neuron w ← w + η·δ·x # update immediately # batch backpropagation for epoch in 1..E: Δw ← 0 for (x, t) in training_set: forward(x); compute δ Δw ← Δw + η·δ·x # accumulate w ← w + Δw # one update per epoch
Batch versus stochastic descentFit a line with batch, mini-batch and stochastic updates, and compare their paths over the error surface.Open demoClose
The model is \(y = wx + b\) on 100 noisy points; the contours show the mean squared error over all \((w, b)\). Batch descent walks straight downhill but needs a full pass for every step. Stochastic descent takes 100 cheap steps for the same cost, stumbling but arriving quickly. Compare all three at the same number of updates, then at the same number of passes over the data.
Optimizers: momentum and Adam
An optimizer is an improved version of gradient descent, packaged as an object in deep learning libraries. Improvements aim to speed up convergence, avoid or overshoot local extrema, and reduce manual tuning, especially of the learning rate. Common optimizers: plain GD, momentum, Nesterov, Adagrad, Adadelta, RMSprop, Adam, AdaMax and Nadam.
Momentum
Backpropagation follows the steepest-descent direction through weight space. A small \(\eta\) gives small, slow changes; a large \(\eta\) speeds learning up but can make the weights oscillate. Adding a momentum term keeps the speed while avoiding the instability. The slides call this form the generalized delta rule:
- Advantages: steadier direction and fewer oscillations; where the gradient keeps the same direction the “ball” rolls faster and faster (with a speed limit since \(\alpha < 1\)), crossing flat areas quickly; with inertia it can roll over narrow local minima.
- Disadvantage: both \(\eta\) and \(\alpha\) must be set by hand, which often takes experiments.
Adam (adaptive moment estimation)
Adam builds on Adagrad and Adadelta. For every weight it keeps two running averages: \(m_t\) of the gradient (the first moment, its mean) and \(v_t\) of the squared gradient (the second moment, its uncentred variance):
Both start at zero, so early on they are biased toward zero, especially when \(\beta_1\) and \(\beta_2\) are close to 1. Bias correction fixes that, and the update divides the corrected mean by the root of the corrected variance:
\(\hat m_t\) is the gradient with momentum; \(\eta/(\sqrt{\hat v_t}+\epsilon)\) acts as each weight’s current learning rate. Default settings: \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\), \(\eta = 0.001\). Adam usually converges quickly; when progress saturates, lower \(\eta\), and after a few reductions it settles in a satisfying minimum.
Optimizer raceSGD, momentum and Adam start from the same point on a ravine, a saddle and a bumpy surface. Click the map to move the start.Open demoClose
When to stop, and how to help backpropagation
Stopping criteria:
- Number of epochs: a maximum is reached.
- Early stopping: the validation error starts to rise after reaching its minimum; cross-validation detects it.
- Plateau: validation performance stops improving beyond a threshold for a set number of epochs.
- Error threshold: training or validation error falls below a target.
- Resources: the compute budget runs out.
- Weight convergence: the weights stop changing significantly for several epochs.
Heuristics that make backpropagation work better:
- Stochastic versus batch: sequential (stochastic) mode is cheaper per iteration but needs more iterations. It shines when the training set is large and redundant.
- Maximize information content: every example should carry as much new information as possible. Prefer examples that give the largest error, or examples radically different from the previous ones, which push the search into new regions of weight space. In practice, shuffle the examples between epochs.
- Normalize the inputs, initialize the weights well and tune the learning rate per layer, as the next sections explain.
Choosing the activation function
| Function | \(f(x)\) | \(f'(x)\) | Advantages | Disadvantages |
|---|---|---|---|---|
| ReLU | \(\max(0,x)\) | \(1\) if \(x > 0\), else \(0\) | Efficient gradient flow; simple and cheap; works well for deep networks | Unbounded outputs; “dead neurons” stuck at 0 |
| Leaky ReLU | \(x\) if \(x > 0\), else \(\alpha x\) | \(1\) or \(\alpha\) | Non-zero gradient for negative inputs; prevents dead neurons | Slightly less efficient; the leak \(\alpha\) needs tuning |
| ELU | \(x\) if \(x > 0\), else \(\alpha(e^{x}-1)\) | \(1\) or \(\alpha e^{x}\) | Smooth output; fast, good gradient flow | More expensive than ReLU or Leaky ReLU |
| Tanh | \(\tanh x\) | \(1-\tanh^2 x\) | Bounded in \((-1, 1)\); zero-centred | Saturates, so gradients vanish |
| Sigmoid | \(\frac{1}{1+e^{-x}}\) | \(\sigma(x)(1-\sigma(x))\) | Outputs read as probabilities; binary classification | Saturates (max slope only 0.25); slower than ReLU |
| Softmax | \(\frac{e^{x_i}}{\sum_j e^{x_j}}\) | see below | Turns scores into a probability distribution over classes | Output layer only |
Softmax turns a vector of real scores into values between 0 and 1 that sum to 1. Its derivative has two cases:
Slide example: scores [1.3, 5.1, 2.2, 0.7, 1.1] become [0.02, 0.90, 0.05, 0.01, 0.02]. The largest score takes almost all the probability because of the exponential.
Normalizing the inputs
Normalized inputs speed up backpropagation. Two measures: the input variables should be uncorrelated (use PCA, ICA or another feature-extraction method), and the decorrelated variables should be scaled to similar variances. The full recipe has three steps: mean removal, decorrelation and covariance equalization.
| Method | Formula | Notes |
|---|---|---|
| Min-max | \(x'=\frac{x-\min}{\max-\min}\) | Into \([0, 1]\); one outlier squeezes everything else |
| Z-score (standardization) | \(x'=\frac{x-\mu}{\sigma}\) | Mean 0, standard deviation 1 |
| Robust | \(x'=\frac{x-\text{median}}{Q_3-Q_1}\) | Median and interquartile range; much less sensitive to outliers |
| Log | \(x'=\log(x+1)\) | For values spanning a wide range; \(+1\) avoids \(\log 0\) |
| Softmax scaling | \(x_i'=\frac{e^{x_i}}{\sum_j e^{x_j}}\) | Values sum to 1 |
| Unit vector | \(\mathbf x'=\frac{\mathbf x}{\lVert\mathbf x\rVert}\) | Each sample gets length 1 |
Scaling calculatorType a feature’s values, add an outlier, and compare every scaling method side by side.Open demoClose
All methods share one axis; a value beyond it is pinned to the edge with its number. With the outlier, min-max squeezes the normal values into a sliver near 0 and z-score into a narrow band; robust scaling keeps them about a unit apart, and \(\log(1 + x)\) pulls the outlier in.
Initializing the weights
Bad initial weights cause four problems:
- Vanishing or exploding gradients: gradients become too small or too large, so training stalls or diverges.
- Saturated neurons: weights that are too large push sigmoid or tanh neurons to their flat ends, where the derivative is almost zero, so learning slows or stops.
- Slow convergence, especially in deep networks.
- Poor performance: lower accuracy on validation and test data.
Vanishing gradients. During backpropagation, gradients tend to get smaller as they move back toward the input. The early layers then learn very slowly or not at all, although they matter most: they detect the simple patterns everything else builds on. Exploding gradients are the opposite: gradients grow layer after layer, early weights change wildly, and the network gives meaningless results.
The reason is the chain rule: every layer multiplies the gradient by a weight and an activation derivative. The sigmoid’s derivative is at most 0.25, so ten sigmoid layers can shrink a gradient by about \(0.25^{10} \approx 10^{-6}\).
| Method | Formula (\(N(0, s)\) has standard deviation \(s\)) | Advantages | Disadvantages |
|---|---|---|---|
| Zero | \(w_{ij}=0\) | Simple and fast | No symmetry breaking: every neuron learns the same feature |
| Random | \(w_{ij}\sim R[-a,a]\), e.g. \([-0.5, 0.5]\) or \(\pm1/\sqrt{n}\) | Easy | Can still vanish or explode |
| Xavier (Glorot) | \(N\big(0,\sqrt{2/(n_{in}+n_{out})}\big)\) | Stable gradients for sigmoid and tanh | Ignores other activations |
| He | \(N\big(0,\sqrt{2/n_{in}}\big)\) | Stable for ReLU, prevents dead neurons | A little more involved than Xavier |
| Orthogonal | \(w_{ij}=Q_{ij}/\sqrt{n_{in}}\) | Less redundancy between neurons | More expensive to compute |
| Kaiming | \(N\big(0,\sqrt{2/(a\,n_{in})}\big)\) | Efficient for ReLU and Leaky ReLU | More complex than Xavier or He |
Xavier keeps the variance of each layer’s output equal to the variance of its input. He initialization doubles it to compensate for ReLU zeroing half the inputs. Transfer learning (Lecture 5) is another “initialization”: start from pre-trained weights. The rule of thumb: weights should be neither too large nor too small.
He and Kaiming initialization are the same method (by Kaiming He); the slides list them separately. The slide text also gives Xavier’s variance as \(2/n\), while the table uses the usual \(2/(n_{in}+n_{out})\).
Vanishing and exploding gradientsPush one gradient back through a deep random network and see how big it is at every layer.Open demoClose
A real forward and backward pass through layers of 64 neurons with random weights and a random input. Bars show the average size of the gradient reaching each layer’s weights, on a log scale: layer 1 is next to the input.
Learning rate per layer
Ideally all neurons learn at the same speed. The last layers usually have larger local gradients than the front layers, so \(\eta\) should be smaller in the last layers. Neurons with many inputs should also get a smaller \(\eta\) than neurons with few. A common suggestion: make each neuron’s \(\eta\) inversely proportional to the square root of its number of incoming connections.
Generalization
Backpropagation aims for weights that generalize: the network maps test data, never seen in training, correctly. Learning can be viewed as curve fitting, with the network as a non-linear input–output mapping; generalization is then good non-linear interpolation between the training points.
How many examples are enough? With \(W\) free parameters (weights and biases) and a permitted fraction \(\varepsilon\) of test errors, the training-set size \(N\) should satisfy
For 10% error, you need about 10 times as many training examples as free parameters.
How much data does my network need?Count the parameters of any MLP and apply \(N \approx W/\varepsilon\).Open demoClose
Overfitting and regularization
Deep models have so many parameters that they overfit easily. Regularization reduces the generalization error. The lecture covers three families: penalties on the parameters (L1, L2), expanding the training set, and dropout.
Parameter penalties
Add a penalty \(\Omega(\theta)\) to the objective, weighted by a hyperparameter \(\alpha \ge 0\). \(\alpha = 0\) means no regularization; larger \(\alpha\) means more:
With L2, each step first shrinks every weight by the factor \((1 - \varepsilon\alpha)\), where \(\varepsilon\) is the learning rate: this is weight decay. L1 subtracts a constant amount in the direction of the sign, which drives small weights exactly to zero.
- L1 behaves like estimating the median of the data, and performs feature selection by zeroing unimportant weights. Useful when there are many features.
- L2 behaves like estimating the mean; it shrinks all weights smoothly but rarely to exactly zero.
Why L1 gives zeros and L2 does notRaise \(\alpha\) and watch the best weights slide toward the origin: L1 lands on an axis, L2 does not.Open demoClose
Ellipses: contours of the unregularized loss \(J\), centred on its minimum (×). Shape: the set of weights with the same penalty, sized to touch the best compromise (●). The dashed line traces the solution as \(\alpha\) grows from 0.
Expanding the data set
The most effective way to prevent overfitting is more training data. Expanding the data set artificially saves time, but depends on the field:
- Images: rotate or scale, as long as the class does not change (rotating a 6 can make a 9).
- Speech: add random noise to the input.
- NLP: replace words with synonyms.
- Noise injection into inputs, hidden layers or outputs. For softmax outputs, label smoothing replaces the 0 and 1 targets with \(\varepsilon/k\) and \(1-\frac{k-1}{k}\varepsilon\) for \(k\) classes.
Dropout
During each training step, dropout temporarily removes a random fraction of a layer’s neurons; only the remaining neurons take part in the forward and backward passes. The rate (between 0 and 1) is the fraction dropped: 0.5 drops half the neurons. Because no neuron can rely on any particular other neuron, the network learns more robust, general features. It works like an ensemble of many thinned networks inside one network.
At test time nothing is dropped, so more neurons are active than during training, and the outputs are scaled to compensate.
The slide says outputs are “scaled down by the dropout rate”. The correct factor is the keep probability, \(1 - \text{rate}\). With rate 0.5 the two coincide; with rate 0.2 you multiply by 0.8, not 0.2. Libraries use “inverted dropout”: they divide by \(1 - \text{rate}\) during training and change nothing at test time.
Dropout, step by stepSample thinned networks and check why test-time outputs must be rescaled.Open demoClose
The output neuron sums 8 hidden activations of 1.0 with weight 1. Over many training steps its average input is \(8\times(1 - \text{rate})\). At test time all 8 are active and it receives 8, so the test output is multiplied by the keep probability \(1 - \text{rate}\) to match what training saw.
Exam traps
- Softmax plus cross-entropy for multi-class; sigmoid plus binary cross-entropy for two classes; linear output plus MSE for regression.
- Adam’s \(v_t\) averages the squared gradient. The bias correction divides by \(1 - \beta^t\), which matters only in the first steps.
- Zero initialization fails because of symmetry, not because zero is “too small”: all neurons in a layer stay identical forever.
- L1 gives sparse weights (feature selection); L2 gives small weights (weight decay).
- Dropout is active only during training. At test time, scale by the keep probability, or use inverted dropout.
- Early stopping watches the validation error, not the training error.
Transfer learning
Reuse the knowledge a network gained on one task for another task, building on existing expertise instead of starting from scratch.
- A pre-trained network has already learned general low-level features (edges, textures, shapes), so a new task needs less data and less training time, and often reaches better accuracy.
- Early and middle layers are reused; the last, task-specific layers are replaced and retrained.
- Four ways: use the model as is; replace the last layer and freeze the rest; replace the last layer and retrain everything; remove several layers and add new ones.
- It makes sense when both tasks have the same kind of input, task A has much more data than task B, and A’s low-level features help with B.
- Risks: domain mismatch, negative transfer and the cost of fine-tuning large models.
Why reuse a trained network
- Faster training: the low-level features are already learned.
- Less data: a small data set is enough for fine-tuning.
- Better performance: pre-trained knowledge often gives better accuracy and generalization than training from scratch.
Potential drawbacks:
- Domain mismatch: what the model learned may not translate perfectly to the new task.
- Negative transfer: biases from pre-training can hurt the new task, so the source model must be chosen and adapted carefully.
- Computational overhead: fine-tuning a large model is still expensive.
How it works. In computer vision, the first layers of a network detect edges, the middle layers detect forms, and the last layers detect task-specific features. Transfer learning keeps the early and middle layers and retrains only the last ones.
The slides compare training a model from scratch with fine-tuning a pre-trained one on the same small data set. From scratch, validation accuracy stays around 0.68 while validation loss climbs from 0.63 to 0.77, a sign of overfitting. The pre-trained model starts at 0.81 accuracy and reaches 0.87, with validation loss falling from 0.42 to 0.34.
Four ways to use a pre-trained model
- 1Use it without modificationWhen the new task is the same as the original, for example classifying the same ImageNet categories.
- 2Replace the last layer, freeze the restTrain only the new classification layer. Suits a small data set on a similar task.
- 3Replace the last layer, retrain the whole networkFine-tuning: all weights start from the pre-trained values. Needs more data.
- 4Remove several layers and add new onesTrain only the added layers, or everything. Needed when the tasks differ more. To retrain the complete network you need enough data and compute.
Example from the slides. Task A recognizes human faces; task B recognizes dog faces. Removing only the last layer is not enough, because the layers before it detect high-level features of human faces. Remove more layers. How many depends on how similar the two tasks are.
When transfer from task A to task B makes sense:
- A and B have the same kind of input (both images, both audio…).
- There is much more data for A than for B.
- Low-level features learned for A could help with B.
Where to find pre-trained models: TensorFlow Hub, Keras Applications, PyTorch (torchvision models), Hugging Face (transformer models), Kaggle and other model zoos.
Which layers should I retrain?Describe your data and task; see the recommended strategy on VGG16 and how many weights you would actually train.Open demoClose
Parameter counts are the real VGG16 values (138 million in total). Grey = frozen, blue = retrained from pre-trained values, amber = new layer trained from random values.
Exam traps
- Freezing means the weights are not updated; they still run in the forward pass.
- The more the tasks differ, the more top layers you replace. Early layers (edges, textures) transfer best.
- Fine-tuning everything on a tiny data set can overfit and destroy the useful pre-trained features.
- Transfer requires the same kind of input: an image network does not help with audio waveforms.
Convolutional neural networks
A dense layer on a photo needs tens of millions of weights. A CNN instead slides small learned filters over the image, sharing their weights everywhere, then applies ReLU and pooling before a final classifier.
- Shallow networks have one or two hidden layers; deep networks have more, sometimes hundreds, and capture more complex patterns at a higher cost.
- Patterns are small and appear anywhere, so a filter only needs a small receptive field, and the same filter can be reused at every position (weight sharing).
- A convolutional layer slides each filter over the input and takes a dot product at each position, producing one feature map per filter. Depth = number of filters.
- Output size: \(W_o=\lfloor (W_i-K+2P)/S\rfloor+1\). Parameters per layer: \((K\cdot K\cdot C_{in}+1)\cdot F\).
- Four operations: convolution, ReLU, pooling (max or average), and a fully connected classifier with softmax.
Shallow and deep networks
| Shallow | Deep | |
|---|---|---|
| Hidden layers | Usually 1–2 | More than 2; modern ones have dozens or hundreds |
| Complexity | Lower; easier to interpret | Higher; captures intricate patterns and representations |
| Training | Faster, fewer parameters | Longer; needs more data and often GPUs |
| Suits | Simple relationships, limited data, when interpretability matters | Images, speech, NLP, any high-dimensional data |
| Examples | Perceptron; MLP with one or two hidden layers | CNN, RNN (LSTM, BLSTM, GRU), GAN, Transformers, ResNet, DenseNet |
Why not a fully connected network?
A 500 × 500 colour image has \(500\times 500\times 3 = 750{,}000\) inputs. Connecting it to just 100 hidden neurons takes 75 million weights. Do we need all those edges, and could some be shared? Two observations say yes:
- Patterns are much smaller than the whole image. A “beak detector” only needs to look at a small region, so it needs few parameters.
- The same pattern appears in different places. An upper-left beak detector and a middle beak detector do the same job, so they can share the same parameters and move around the image.
A convolutional neural network is an MLP designed to recognize two-dimensional shapes with a high degree of invariance to translation, scaling, skewing and other distortions. It is a network with some convolutional layers (and some other layers); each convolutional layer has a number of filters that perform convolutions.
Dense versus convolution: parameter countGrow the image and compare the weights a dense layer and a convolutional layer need.Open demoClose
The convolution operation
A filter (also called a kernel or feature detector) is a small grid of weights, 3 × 3 in the lecture’s example. It slides over the image; at each position, multiply the overlapping values element by element and add them up (a dot product). The grid of results is the feature map. The filter values are the network parameters to learn.
The lecture’s 6 × 6 binary image with Filter 1 (a diagonal detector) at stride 1 gives a 4 × 4 map that starts with 3 and −1. The 3 marks where the diagonal pattern matches perfectly. Repeating with Filter 2 (a vertical line detector) gives a second 4 × 4 map; together they form a 2 × 4 × 4 output. With stride 2 the map shrinks to 2 × 2: \(\left[\begin{smallmatrix}3 & -3\\ -3 & 0\end{smallmatrix}\right]\).
Convolution stepperSlide the slide’s filters over the 6 × 6 image one position at a time, with any stride and padding.Open demoClose
Click any cell of the feature map to jump to that position. You can also edit the filter values when “Custom” is selected.
Colour images and the comparison with dense layers
A colour image has three channels (R, G, B), so each filter also has depth 3: one 3 × 3 slice per channel. At each position the three channel results and the bias are added into one number. In the slide’s worked example the three channels give 308, −498 and 164; with a bias of 1 the output is \(308 - 498 + 164 + 1 = -25\).
Seen as a network, convolution is a dense layer with most connections removed and the rest shared. Each output neuron connects to only 9 inputs, not all 36, and every output neuron uses the same 9 weights. Fewer parameters, then even fewer.
Classic hand-made filters include edge detectors, blur and sharpen. In a CNN the filter values are not designed: the network learns them during training. You still choose the number of filters, their size and the architecture. More filters extract more features and usually recognize unseen images better.
Filters on an imageApply edge, sharpen and blur kernels to a small drawing, then rectify the feature map with ReLU.Open demoClose
Feature map colours: blue positive, rose negative, background zero. ReLU keeps only the blue.
Depth, stride, padding and output size
Three hyperparameters, fixed before training, control the size of the feature maps:
- Depth: the number of filters. Three filters on a boat image give three stacked feature maps, so the output depth is 3.
- Stride: how many pixels the filter jumps each time. Stride 2 jumps two pixels and gives smaller maps.
- Zero-padding: a border of zeros around the input, so the filter can also be applied to the edge pixels.
For the 6 × 6 image with a 3 × 3 filter, no padding and stride 2: \((6 - 3 + 0)/2 + 1 = 2.5\), rounded down to 2.
Slide example. A 28 × 28 × 1 input goes through three convolutions:
| Layer | Kernel | \(P\) | \(S\) | Computation | Output | Parameters |
|---|---|---|---|---|---|---|
| Input | – | – | – | – | 28 × 28 × 1 | – |
| Conv 1 | 16 × (5 × 5 × 1) | 1 | 2 | \(\lfloor(28 - 5 + 2)/2\rfloor + 1 = 13\) | 13 × 13 × 16 | 416 |
| Conv 2 | 2 × (3 × 3 × 16) | 0 | 2 | \(\lfloor(13 - 3)/2\rfloor + 1 = 6\) | 6 × 6 × 2 | 290 |
| Conv 3 | 3 × (3 × 3 × 2) | 0 | 1 | \((6 - 3)/1 + 1 = 4\) | 4 × 4 × 3 | 57 |
A filter’s depth always equals the depth of its input; the output depth equals the number of filters.
Layer size calculatorBuild a chain of convolution and pooling layers and get every output shape and parameter count.Open demoClose
ReLU and pooling
ReLU is applied after every convolution. It works element by element and replaces every negative value in the feature map by zero, which introduces non-linearity. Tanh or sigmoid would also work, but ReLU performs better in most situations.
Pooling (subsampling, down-sampling) shrinks each feature map but keeps the most important information. Max, average and sum pooling all exist. The usual setting is a 2 × 2 window with stride 2. In the slide example:
Why pooling helps:
- Smaller, more manageable representations, so fewer parameters and computations, which also controls overfitting.
- Invariance to small transformations, distortions and translations: a small shift does not change the maximum or average of a neighbourhood.
- A nearly scale-invariant (strictly, equivariant) representation, so objects are detected wherever they are.
Pooling playgroundEdit the feature map and compare max, average and sum pooling.Open demoClose
The whole CNN
Convolution, ReLU and pooling repeat several times. The last maps are flattened into a vector and fed to a fully connected network: a traditional MLP with a softmax output. The convolutional part extracts high-level features; the fully connected part uses them to classify the image.
- 1ConvolutionLearned filters produce feature maps.
- 2Non-linearity (ReLU)Negative values become 0.
- 3PoolingEach map shrinks; steps 1–3 can repeat many times.
- 4Classification (fully connected + softmax)Flattened features become class probabilities: cat, dog, …
Two principles from the lecture explain why this works:
- Feature extraction: each neuron takes its inputs from a local receptive field, so it extracts local features. Once a feature is found, its exact location matters less, as long as its position relative to other features is roughly preserved.
- Feature mapping: each layer has several feature maps. Convolution with a small kernel followed by a non-linearity gives shift invariance, and weight sharing reduces the number of parameters.
A character recognizer (slides 58–61)
The input is 28 × 28, a centred and normalized character. Four hidden layers and an output layer:
- Convolution: 4 feature maps of 24 × 24, each neuron with a 5 × 5 receptive field.
- Subsampling and local averaging: 4 maps of 12 × 12.
- Convolution: 12 maps of 8 × 8; neurons may connect to several maps of the previous layer.
- Subsampling: 12 maps of 4 × 4.
- Output: a final convolution with 26 neurons (one per letter), each with a 4 × 4 receptive field.
The network has about 100,000 connections but only about 2,600 weights, thanks to weight sharing. Fewer free parameters mean less capacity, which improves generalization. It is trained with stochastic backpropagation.
Training a CNN
- 1InitializeAll filters and weights get random values.
- 2Forward passConvolution, ReLU, pooling and the fully connected layer give class probabilities, for example [0.2, 0.4, 0.1, 0.3], random at first.
- 3Total error\(\sum \tfrac12(\text{target} - \text{output})^2\). With target [0, 0, 1, 0]: \(\tfrac12(0.04 + 0.16 + 0.81 + 0.09) = 0.55\).
- 4Backpropagation and gradient descentUpdate every filter value and weight in proportion to its contribution to the error. The same image might now give [0.1, 0.1, 0.7, 0.1], an error of 0.06. The number and size of filters and the architecture stay fixed.
- 5RepeatSteps 2–4 for every image in the training set.
Architectures by task
| Task | Architectures |
|---|---|
| Image classification | AlexNet, VGG (VGG16, VGG19), GoogLeNet (Inception), ResNet (ResNet50, ResNet101), DenseNet, MobileNet |
| Object detection | R-CNN (Fast, Faster R-CNN), YOLO, SSD, RetinaNet |
| Semantic segmentation | FCN, U-Net, SegNet, DeepLab |
| Generative modelling | GAN, VAE, PixelCNN |
GAN applications listed in the lecture: image generation (faces, landscapes, animals), image-to-image translation (day to night, sketch to photo), super-resolution (medical and satellite imaging), video synthesis, 3D objects from 2D images, text-to-image synthesis, style transfer, data augmentation, inpainting, image manipulation (pose, expression, background), colourization and face swapping.
Exam traps
- Round down in the output-size formula: \(\lfloor (W - K + 2P)/S\rfloor + 1\).
- A filter spans the full input depth. Parameters \(= (K\times K\times C_{in} + 1)\times F\); the \(+1\) is the bias of each filter.
- Pooling has no weights to learn. It reduces width and height but keeps the depth.
- ReLU is applied element-wise to the feature map, after the convolution, not to the filter.
- “Connections” and “weights” differ: weight sharing gives many connections but few distinct weights.
Recurrent networks and attention
Sequences carry meaning in their order. An RNN reads them one step at a time and feeds its hidden state back to itself as memory. LSTM and GRU gates protect that memory; attention lets the model look back at any position directly.
- Feedforward networks and CNNs see only the current input and remember nothing. An RNN keeps a hidden state that summarizes everything seen so far, and uses the same weights at every time step.
- \(a^{\langle t\rangle}=g_1(W_{aa}a^{\langle t-1\rangle}+W_{ax}x^{\langle t\rangle}+b_a)\) and \(y^{\langle t\rangle}=g_2(W_{ya}a^{\langle t\rangle}+b_y)\).
- Training unrolls the network over time (backpropagation through time). Gradients are multiplied once per step, so over long sequences they vanish or explode, and long-term dependencies are lost.
- LSTM adds a cell state (long-term memory) controlled by forget, input and output gates. GRU merges this into two gates (update, reset) and is a little faster. Neither always wins.
- Encoder–decoder RNNs map a sequence to another sequence. Attention replaces the single fixed context vector with a weighted sum of all encoder states, with weights from a similarity score.
Sequential data and why RNNs exist
Sequential data are data points in a specific order, where the order itself carries meaning:
- Time series: stock prices, machine sensor readings, daily weather.
- Text: sentences, chat logs, DNA or RNA sequences.
- Audio: speech, music.
- Video: sequences of frames, gesture recognition.
Feedforward networks (regression, classification) and CNNs (vision) cannot handle such data: they consider only the current input and cannot memorize previous ones. A recurrent neural network saves the output of a layer and feeds it back as input for the next step, so the prediction at time \(t\) depends on \(x(t)\) and on what came before.
Applications shown: image captioning, speech to text, machine translation, and video understanding such as object tracking.
The vanilla RNN
Folding the layers of an unrolled network into one recurrent layer gives the vanilla RNN, the simplest form. The slides name its parameters \(A\), \(B\) and \(C\): input-to-hidden, hidden-to-hidden and hidden-to-output weights, reused at every time step. With the more common notation, for each time step \(t\):
\(g_1\) is usually tanh (or ReLU), and \(g_2\) a sigmoid or softmax, depending on the task.
| Advantages | Disadvantages |
|---|---|
| Built for sequential data where order matters | Vanishing and exploding gradients |
| Handles variable-length input | Struggles with long-term dependencies |
| Parameter sharing across time steps: the same pattern is recognized at any position | Hard to keep context over long sequences |
| Memory of previous inputs | Sensitive to input order: reordering changes the output |
| Can be trained online, as new data arrives | No parallelism: one time step after another |
Vanilla RNNs learn short-term dependencies well but struggle with long-term ones: when the word that decides the answer is far back in the sentence, its influence has faded by the time it is needed.
Types of RNN
| Type | Input → output | Example |
|---|---|---|
| One to one | one → one | Image classification |
| One to many | one → sequence | Image captioning: an image becomes a sentence |
| Many to one | sequence → one | Sentiment analysis: a review becomes positive or negative |
| Many to many (same length) | sequence → sequence, aligned | Named entity recognition: a tag for every word |
| Many to many (encoder–decoder) | sequence → sequence, different lengths | Machine translation |
RNN shapes, unrolledSee where inputs enter and outputs leave for each RNN type.Open demoClose
Backpropagation through time
To train an RNN, unroll it over the sequence and apply backpropagation to the unrolled network. Because the weights are shared, the gradient for \(W\) is the sum over all time steps, and the loss at step \(t\) depends on every earlier state:
The term \(\partial s_t/\partial s_k\) is a product of one factor per step between \(k\) and \(t\). If those factors are mostly smaller than 1, the product shrinks exponentially (vanishing gradient) and distant steps stop contributing. If they are larger than 1, it grows exponentially (exploding gradient).
How far back can an RNN learn?Set the recurrent weight and see how much gradient reaches each earlier time step.Open demoClose
A one-neuron RNN, \(h_t = g(w_{aa}\,h_{t-1} + 0.5\,x_t)\), on a random input sequence. Bars show \(\lvert\partial h_T/\partial h_k\rvert\), the gradient the last step sends back to step \(k\), on a log scale.
Variants of the RNN
- LSTM: designed against the vanishing gradient, with three gates (input, forget, output) that decide what to remember and what to forget.
- GRU: also against the vanishing gradient, with two gates. The reset gate decides how much of the previous state to forget; the update gate decides how much of the new state to keep.
- Bidirectional RNN: processes the sequence forward and backward, capturing past and future context.
- Encoder–decoder RNN: one RNN encodes the input sequence into a fixed-length vector; another generates the output sequence from it. Used for translation.
- Attention: lets the network focus on the most relevant parts of a long input instead of treating all parts equally.
Bidirectional RNN
Is reading left to right always enough? Fill the blank:
- “I am ___.”
- “I am ___ hungry.”
- “I am ___ hungry, and I can eat half a cow.”
The right word (happy, not, very) depends on what comes after the blank. A bidirectional RNN combines two independent RNNs, one reading left to right and one right to left. Their outputs at each position are combined by concatenation, sum, average or maximum.
LSTM
A traditional RNN cell has a single layer (a tanh). An LSTM cell has four interacting layers and four components: the cell state and the forget, input and output gates. The hidden state \(h_{t-1}\) is the short-term memory; the cell state \(C_t\) is the long-term memory.
- Cell state: a conveyor belt running through the whole unit. A pointwise multiplication (×) by values between 0 and 1 forgets (0) or keeps (1) each component; an addition (+) writes new information.
- Forget gate: a sigmoid layer looks at \(h_{t-1}\) and \(x_t\) and outputs a number between 0 (forget completely) and 1 (keep completely) for each component of \(C_{t-1}\).
- Input gate: a sigmoid decides which values to update, a tanh layer proposes candidate values \(\tilde C_t\), and their product \(i_t\ast\tilde C_t\) is added to the cell state.
- Output gate: a sigmoid decides which parts of the cell state to output; the cell state goes through tanh (squashing to \([-1, 1]\)) and is multiplied by it.
The key line is \(C_t=f_t\ast C_{t-1}+i_t\ast\tilde C_t\): the memory is updated by adding, not by squashing through a weight and a tanh at every step. When the forget gate stays near 1, information and its gradient pass through many steps almost unchanged.
The conveyor beltWrite one fact into memory at step 1 and see how much is left 40 steps later, in an LSTM and in a vanilla RNN.Open demoClose
At step 1 the input is the fact (value 1); afterwards the inputs are small random noise. Gates are held constant here to isolate their effect; in a trained LSTM they are computed from \(h_{t-1}\) and \(x_t\) at every step.
GRU
The gated recurrent unit is a newer, simpler relative of the LSTM. It drops the cell state and passes information through the hidden state, with two gates:
- Update gate: acts like the LSTM’s forget and input gates together; decides what to throw away and what new information to add.
- Reset gate: decides how much past information to forget.
A GRU has fewer tensor operations, so it trains a little faster. There is no clear winner: practitioners usually try both.
Summary. RNNs suffer from short-term memory. LSTMs and GRUs mitigate it with gates, small neural networks that regulate the flow of information along the sequence. They power speech recognition, speech synthesis and natural language understanding.
Encoder–decoder (sequence to sequence)
A many-to-many RNN with two blocks, used for translation, summarization and speech recognition:
- Encoder: reads the input sequence step by step, updating its hidden state. Its final hidden state summarizes the whole input: the fixed-size context. Usually an LSTM or GRU.
- Decoder: starts from the encoder’s final state and generates the output sequence one element at a time, predicting the next element at each step.
Training versus prediction. During training, the decoder receives the true previous output as its next input (both sequences are known). This is called teacher forcing. During prediction, the decoder’s own output at time \(t\) becomes its input at \(t + 1\). Every target sequence is wrapped in special tokens: SOS (start of sequence) starts the decoding and EOS (end of sequence) tells the network the output is complete.
| Advantages | Disadvantages |
|---|---|
| Versatile across sequence-to-sequence tasks | Limited memory: long sequences lose information (vanishing and exploding gradients) |
| Input and output lengths can differ | One fixed-size context vector cannot hold the global context of a long input |
| Compresses the input into a summary, keeping the essentials | Expensive training, especially with variable lengths |
| Handles temporal dependencies | Sensitive to hyperparameters; struggles with rare or out-of-vocabulary sequences |
Attention
Even LSTMs and GRUs lose performance on long sentences: a translator must link words dozens of positions apart, and squeezing everything into one context vector is a bottleneck. Attention lets the decoder focus on the relevant parts of the input when producing each part of the output, as people do.
- Context vector: in a plain encoder–decoder, one fixed-size vector passed from encoder to decoder.
- Attention weights: computed dynamically for every input position, from a score measuring how relevant each encoder hidden state is to the current decoder state. Common scores: dot product, additive and multiplicative attention.
- Weighted sum: the weights (after softmax) combine all encoder hidden states into a context for the current step, used together with the decoder state to predict the output.
Self-attention, worked
In self-attention, every element of one sequence attends to every element of the same sequence. Each input row \(x\) is projected three times: a key \(k=xW_K\), a query \(q=xW_Q\) and a value \(v=xW_V\). The output for one input is the softmax of its query’s dot products with all keys, applied to the values.
Slide example: three inputs \(X\) = [1, 0, 1, 0], [0, 2, 0, 2], [1, 1, 1, 1] give keys \(K\) = [0, 1, 1], [4, 4, 0], [2, 3, 1], queries \(Q\) = [1, 0, 2], [2, 2, 2], [2, 1, 3] and values \(V\) = [1, 2, 3], [2, 8, 0], [2, 6, 3]. For input 1, the scores are \(q_1\cdot k_j\) = [2, 4, 4], the softmax is [0.063, 0.468, 0.468], and the output is \(0.063\,v_1 + 0.468\,v_2 + 0.468\,v_3\) = [1.94, 6.68, 1.60].
Self-attention calculatorRecompute the slide example for each input, with or without the \(\sqrt d\) scaling transformers use.Open demoClose
The slide does not scale the scores. Transformers divide by \(\sqrt d\) so that large dot products do not push the softmax to one-hot weights.
Exam traps
- An RNN uses the same weights at every time step; unrolling does not create new parameters.
- Hidden state = short-term memory; cell state = long-term memory. Only the LSTM has a cell state.
- LSTM has 3 gates (forget, input, output); GRU has 2 (update, reset).
- Teacher forcing applies during training only; at prediction time the decoder feeds itself its own outputs.
- In self-attention the softmax is over the scores of one query against all keys; the output is a weighted sum of values, not keys.
Formula sheet
Every formula from the slides, grouped by lecture, each with a one-line reminder or a worked number. Search by name or symbol.
Exam prep
The calculations the lectures train you to do by hand, with full answers, then the comparison questions that come up most, then a scored quiz.
Calculations, worked
C1Perceptron: train on \(a = (0, 1)\) −, \(b = (2, 0)\) −, \(c = (1, 1)\) + with \(\eta = 1\) from \(\mathbf w = (-1.5, 0, 2)\).
Augment every sample with \(x_0 = 1\). Output + if \(s = \mathbf w\cdot\mathbf x > 0\). Add \(\mathbf x\) when a + sample gets \(s \le 0\); subtract \(\mathbf x\) when a − sample gets \(s > 0\).
| It. | \(\mathbf w\) | Sample | \(s\) | Action |
|---|---|---|---|---|
| 1 | (−1.5, 0, 2) | a − | +0.5 | subtract (1, 0, 1) |
| 2 | (−2.5, 0, 1) | b − | −2.5 | none |
| 3 | (−2.5, 0, 1) | c + | −1.5 | add (1, 1, 1) |
| 4 | (−1.5, 1, 2) | a − | +0.5 | subtract |
| 5 | (−2.5, 1, 1) | b − | −0.5 | none |
| 6 | (−2.5, 1, 1) | c + | −0.5 | add |
| 7 | (−1.5, 2, 2) | a − | +0.5 | subtract |
| 8 | (−2.5, 2, 1) | b − | +1.5 | subtract (1, 2, 0) |
| 9 | (−3.5, 0, 1) | c + | −2.5 | add |
| 10–12 | (−2.5, 1, 2) | a, b, c | −0.5, −0.5, +0.5 | none: converged |
Final weights \((-2.5, 1, 2)\): the line \(x_1 + 2x_2 = 2.5\). By hand, \((-5, 2, 4)\) is the same line.
C2Design McCulloch–Pitts neurons for AND, OR, NOT, NAND and NOR. Why not XOR?
With “fire if \(\sum_j w_j x_j \ge T\)”: AND \(w = (1, 1),\ T = 2\); OR \(w = (1, 1),\ T = 1\); NOT \(w = -1,\ T = 0\); NAND \(w = (-1, -1),\ T = -1\); NOR \(w = (-1, -1),\ T = 0\); AND-NOT \(w = (2, -1),\ T = 2\).
XOR would need \(w_1 \ge T\), \(w_2 \ge T\), \(0 < T\) and \(w_1 + w_2 < T\). Adding the first two gives \(w_1 + w_2 \ge 2T > T\), a contradiction. No single line separates the classes; a hidden layer is needed.
C3Give weights for a 2-2-1 threshold network that computes XOR.
\(h_1 = \operatorname{step}(x_1 + x_2 - 0.5)\) (OR), \(h_2 = \operatorname{step}(-x_1 - x_2 + 1.5)\) (NAND), \(y = \operatorname{step}(h_1 + h_2 - 1.5)\) (AND), with \(\operatorname{step}(v) = 1\) if \(v \ge 0\).
Check: \((0, 0) \to h = (0, 1) \to y = 0\); \((0, 1)\) and \((1, 0) \to h = (1, 1) \to y = 1\); \((1, 1) \to h = (1, 0) \to y = 0\).
C4One backpropagation step on Example 3 (inputs 0.05, 0.10; targets 0.01, 0.99; \(\eta = 0.5\)). Find \(E\), new \(w_5\) and new \(w_1\).
Forward. Hidden layer, then output layer:
Error.
\(w_5\) (output layer).
\(w_1\) (hidden layer). Here \(\delta = \partial E/\partial\,\mathrm{net}\), the slide’s sign convention for this example.
Check every other weight in the backpropagation calculator.
C5Normalize \(X_1 = (2, 120)\) … \(X_5 = (18, 1023)\) and explain why.
Divide \(f_1\) by its maximum 18 and \(f_2\) by 1121: \(X_1 = (0.111, 0.107)\), \(X_2 = (0.444, 0.475)\), \(X_3 = (0.056, 0.880)\), \(X_4 = (0.833, 1.000)\), \(X_5 = (1.000, 0.913)\).
Before scaling, \(d(X_1, X_2) = \sqrt{6^2 + 413^2} \approx 413.04\): feature 2 accounts for 99.98% of the squared distance. After scaling, both features contribute.
C6Edit distance from TRAIN to CRANE, with the table.
The bottom-right cell is 3: substitute T→C, keep R and A, substitute I→N and N→E.
C7Convolve the 6 × 6 image with Filter 1 (stride 1), then give the output size for stride 2.
Top-left: multiply the window by the filter element by element, then add:
The full map is
Stride 2: \(\lfloor(6 - 3 + 0)/2\rfloor + 1 = 2\), so the map is 2 × 2: \(\left[\begin{smallmatrix}3 & -3\\ -3 & 0\end{smallmatrix}\right]\).
C8A 28 × 28 × 1 input passes conv(16 × 5 × 5, \(P = 1\), \(S = 2\)), conv(2 × 3 × 3, \(P = 0\), \(S = 2\)), conv(3 × 3 × 3, \(P = 0\), \(S = 1\)). Give each output size and parameter count.
\(\lfloor(28 - 5 + 2)/2\rfloor + 1 = 13\) → 13 × 13 × 16, parameters \((5\cdot 5\cdot 1 + 1)\cdot 16 = 416\).
\(\lfloor(13 - 3)/2\rfloor + 1 = 6\) → 6 × 6 × 2, parameters \((3\cdot 3\cdot 16 + 1)\cdot 2 = 290\).
\((6 - 3)/1 + 1 = 4\) → 4 × 4 × 3, parameters \((3\cdot 3\cdot 2 + 1)\cdot 3 = 57\). Total 763.
C9Softmax of [1.3, 5.1, 2.2, 0.7, 1.1] and the cross-entropy if class 2 is correct.
\(e^{z}\) = [3.67, 164.0, 9.03, 2.01, 3.00], sum 181.7. Probabilities [0.020, 0.903, 0.050, 0.011, 0.017], rounded on the slide to [0.02, 0.90, 0.05, 0.01, 0.02].
Cross-entropy \(= -\ln 0.903 = \mathbf{0.102}\). If class 3 were correct instead: \(-\ln 0.050 = 3.0\).
C10Self-attention output for input 1 in the slide example.
\(q_1\) = [1, 0, 2]; scores with \(k_1\) = [0, 1, 1], \(k_2\) = [4, 4, 0], \(k_3\) = [2, 3, 1] are 2, 4, 4. Softmax: [0.063, 0.468, 0.468].
Output:
Inputs 2 and 3 give [2.00, 7.96, 0.05] and [2.00, 7.76, 0.36].
C11Momentum: \(\Delta w(n-1) = 0.2\), \(\alpha = 0.9\), \(\eta = 0.1\), \(\delta = 0.5\), input \(y = 0.8\). Find \(\Delta w(n)\).
Without momentum the step would be only 0.04.
C12Adam’s first step with gradient \(g_1 = 0.5\) and default settings.
\(m_1 = 0.1\times 0.5 = 0.05\); \(v_1 = 0.001\times 0.25 = 0.00025\). Bias correction: \(\hat m_1 = 0.05/(1 - 0.9) = 0.5\); \(\hat v_1 = 0.00025/(1 - 0.999) = 0.25\).
Step \(= 0.001\times 0.5/(\sqrt{0.25} + 10^{-8}) = \mathbf{0.001}\). Adam’s first step is about \(\eta\) in the direction of \(-\operatorname{sign}(g)\), whatever the size of the gradient.
C13How many training examples for a 784-16-16-10 network with at most 10% test error?
C14Label smoothing with \(\varepsilon = 0.1\) and \(k = 4\) classes: what are the new targets?
Wrong classes: \(\varepsilon/k = 0.025\). Correct class: \(1 - (k-1)\,\varepsilon/k = 1 - 0.075 = 0.925\). They still sum to 1.
C15A neuron’s total input is 10 at test time. During training, dropout with rate 0.2 was applied to its inputs. What should it receive?
During training it saw on average 80% of its inputs, so at test time scale by the keep probability: \(10\times 0.8 = \mathbf{8}\). With inverted dropout, training inputs were already divided by 0.8 and the test input stays 10.
Concept questions
Q1Describe the steps of a machine learning system.
- Data collection (sources, noise, quantity).
- Preprocessing: cleaning, normalization, feature extraction and selection.
- Model selection.
- Training: minimize the error between predictions and labels.
- Evaluation on unseen test data (accuracy, precision, recall, F1).
- Hyperparameter tuning (grid or random search on the validation set).
- Deployment (API, integration).
Illustrate with salmon versus sea bass: camera, features (length, lightness), classifier, evaluation.
Q2Compare the perceptron rule and gradient descent.
- Objective: misclassification error versus any differentiable function.
- Update: only on mistakes versus every step, proportional to the gradient.
- Convergence: guaranteed for linearly separable data versus no guarantee (local minima).
- Scope: binary linear classification versus regression, multi-class and deep networks.
Both are iterative and minimize an error by adjusting weights.
Q3Why can a single perceptron not solve XOR, and how does an MLP solve it?
A perceptron draws one line; XOR’s positive points (0,1) and (1,0) are on opposite corners and cannot be separated from (0,0) and (1,1) by one line. An MLP’s hidden neurons map the inputs into a new space (OR and NAND outputs) where the classes are linearly separable, and the output neuron draws the line there.
Q4Explain backpropagation and the role of the learning rate.
Forward pass to compute outputs; error at the output; output deltas \(\delta = (t - o)\,f'(u)\); hidden deltas \(\delta = f'(u)\sum w\,\delta\) propagated backward with the old weights; update every weight by \(\eta\,\delta\,x\); repeat over patterns and epochs. \(\eta\) sets the step size: too small is slow, too large oscillates or diverges. Activations must be differentiable.
Q5What are vanishing and exploding gradients, and how do you fight them?
Backpropagation multiplies one factor per layer (or per time step in an RNN). Factors below 1 shrink the gradient exponentially, so early layers barely learn; factors above 1 make it explode and training diverges.
Remedies: ReLU-family activations, Xavier or He initialization, input normalization, per-layer learning rates, and for sequences LSTM or GRU gates. (Gradient clipping and batch normalization are common outside the slides.)
Q6Compare online, stochastic, batch and mini-batch learning.
Online: update after each sample (little memory, tracks change, noisy). Stochastic: the same in random order (can escape local optima, less stable). Batch: one update per pass (parallel, smooth, slow per update). Mini-batch: update per small subset, the usual compromise.
Q7Momentum versus Adam.
Momentum adds \(\alpha\,\Delta w(n-1)\) to each step: smoother direction, faster on consistent slopes, can roll over small minima; \(\eta\) and \(\alpha\) need tuning. Adam keeps running averages of \(g\) and \(g^2\) with bias correction and divides by \(\sqrt{\hat v}\), giving each weight its own adaptive step; the defaults \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\), \(\eta = 0.001\) usually work.
Q8L1 versus L2 regularization, and other ways to fight overfitting.
L1 adds \(\alpha\lVert w\rVert_1\): gradient \(\alpha\,\operatorname{sign}(w)\), drives weights to exactly zero (feature selection, like a median). L2 adds \(\tfrac12\alpha\lVert w\rVert_2^2\): weight decay \(w \leftarrow (1 - \varepsilon\alpha)\,w - \varepsilon\nabla J\), small but non-zero weights (like a mean).
Others: more data and augmentation, noise injection and label smoothing, dropout, early stopping.
Q9When and how would you use transfer learning?
When tasks share the input type, the source task has much more data, and its low-level features help. Ways: use as is; replace the last layer and freeze the rest (small, similar data); replace the last layer and fine-tune all (more data); remove several layers and add new ones (different task, e.g. human faces → dog faces). Risks: domain mismatch, negative transfer, compute cost.
Q10Why are CNNs better than MLPs for images?
A dense layer on a 500 × 500 × 3 image with 100 neurons needs 75 million weights. CNNs use local receptive fields (patterns are small) and weight sharing (patterns appear anywhere), so a layer of 100 filters of 5 × 5 × 3 needs 7,600 parameters. Pooling adds invariance to small shifts and reduces size; the result generalizes better with fewer parameters.
Q11Explain the LSTM cell and how it differs from a GRU.
The cell state is a conveyor belt of long-term memory; the hidden state is short-term memory. The forget gate scales the old cell state, the input gate adds tanh candidates, and the output gate decides what of \(\tanh(C)\) becomes \(h\). Additive updates let information and gradients survive many steps.
The GRU has no cell state and two gates (update, reset). It is cheaper and slightly faster; neither is always better.
Q12Why was attention added to encoder–decoder RNNs?
The encoder squeezes the whole input into one fixed-size context vector, which loses information for long sentences. Attention computes, at each decoder step, a score between the decoder state and every encoder state, normalizes the scores with softmax, and uses the weighted sum of encoder states as a fresh context, so the decoder can focus on the relevant words. Transformers keep only attention and drop recurrence, which makes training parallel.
Q13Three ways to build invariance into a neural network.
By structure (wire the weights so transformed inputs give the same output; too many connections), by training (data augmentation; cost and overfitting), by feature space (invariant features; fewer features and guaranteed invariance, but needs feature engineering).
Quiz
34 multiple-choice questions across all seven lectures. Answer in any order; nothing is marked until you check. Each answer comes with a one-line explanation.