How the ten labs fit together
Every lab builds one piece of a single pipeline: images go in, a network guesses, and the guesses are used to improve the network. See where each piece fits first.
| Lesson | Lab | You learn |
|---|---|---|
| 1 | Lab 1 | Jupyter: cells, modes, shortcuts |
| 2 | Lab 2 | matplotlib: line plots, legend, subplots |
| 3 | Lab 3 | NumPy: arrays, shape, ranges, axis, broadcasting |
| 4 | Lab 4 | Tensors: create, join, multiply, share memory |
| 5 | Lab 5 (+6) | Autograd: gradients, backward, step, freezing |
| 6 | Labs 6–7 | Layers: Linear, Conv2d, pooling, activations |
| 7 | Lab 7 | Building a network class and tracking its shapes |
| 8 | Lab 8 | Datasets, transforms, DataLoaders |
| 9 | Lab 9 | Training and testing the CIFAR-10 classifier |
| 10 | Lab 10 | Data augmentation |
| 11 | Lab 6 + script | Transfer learning, train/eval mode |
Each lesson ends with quick-check cards: tap one to see the answer. The chips under each title jump to the mock exam questions that test that lesson.
Jupyter Notebook
The notebook is where all the labs run: a page of cells, each one either code or text.
a = 32
b = 2*a
print("%d + %d" % (a, b))32 + 64
A code cell runs Python and shows its output right below it. A markdown cell holds formatted text. You switch between two modes:
- Edit mode: you are typing inside a cell. Press Enter (or double-click) to get in.
- Command mode: you act on whole cells (change type, select, move). Press Esc to get out.
The shortcuts
| Keys | Does |
|---|---|
| Shift + Enter | run the cell and select the cell below |
| Ctrl/Cmd + Enter | run the cell and stay on it |
| Alt + Enter | run the cell and insert a new one below |
| Esc / Enter | command mode / edit mode |
| os.listdir? | show the help of a function |
| os.listdir?? | show its source code |
Which keys evaluate the current cell and select the one below?
How do you read the documentation of transforms.Normalize inside a notebook?
transforms.Normalize? (two question marks show the source).Plotting with matplotlib
Plots are how you look at data and results: a curve, a histogram, a grid of images.
import matplotlib.pyplot as plt
A line plot
plt.plot(x, y, fmt, color=None, label=None)
- x, y
- two lists of the same length: the coordinates of the points
- fmt
- optional short style string, e.g.
'red'or'r' - color
- line color:
'r','g','b','k'(black)… - label
- the name shown in the legend (only after
plt.legend())
Simplified to the arguments the lab uses.
X = [0.01*i for i in range(0, 201)]
Y = [x**2 for x in X]
Z = [x for x in X]
plt.plot(X, Y, color='r', label='quadratic')
plt.plot(X, Z, color='g', label='linear')
plt.xlabel("x"); plt.ylabel("y"); plt.title("Multiple plots")
plt.legend() # without this, the labels are never shown
Why a curve can look jagged
plt.plot joins consecutive points with straight lines. Few points give a bent line; many close points look smooth.
X = [0.01*i for i in range(0, 201)].Python bit: [0.01*i for i in range(0, 201)] is a list comprehension: "for every i from 0 to 200, compute 0.01·i, collect the results in a list". range(0, 201) stops at 200.
Several plots in one figure
plt.subplot(nrows, ncols, index)
- nrows, ncols
- split the figure into a grid of rows × columns
- index
- which cell the next plot goes in, counted left → right, then top → bottom, starting at 1
Other chart types
| Call | Draws |
|---|---|
| plt.hist(data, bins=9) | histogram: how often each value range occurs |
| plt.bar(x, heights, width=0.8) | bar chart (default bar width 0.8) |
| plt.pie(sizes, labels=names) | pie chart; each slice = size / total |
| plt.figure(1) | start (or select) figure number 1 |
| plt.imshow(img, cmap="gray") | show an image (Labs 8–10) |
Which call selects the bottom-left panel of a 2 × 2 grid?
plt.subplot(2, 2, 3). Bottom-right is plt.subplot(2, 2, 4).You passed label='quadratic' but no label appears. Why?
plt.legend().Why does y = x² from x = 0, 1, 2 look jagged, and what is the fix?
NumPy arrays
NumPy arrays are the model for PyTorch tensors: same shapes, same indexing, same axis rules. Learn them once here.
import numpy as np
Creating arrays
np.array(object, dtype=None)
- object
- a list, or a list of lists for 2-D:
[[7, 2], [3, 4], [5, 6]]is 3 rows × 2 columns - dtype
- element type (
int,float…). Everything is converted to it
| Call | Gives |
|---|---|
| np.zeros((2, 4)) | 2 × 4 of 0.0 |
| np.ones((3, 2)) | 3 × 2 of 1.0 |
| np.empty((3, 4)) | 3 × 4 of garbage: memory reserved but not cleaned |
np.empty often prints zeros, but that's luck. Its values are whatever was in RAM. "np.empty gives zeros" is false.Attributes
| Attribute | For a = np.array([[7,2],[3,4],[5,6]]) |
|---|---|
| a.shape | (3, 2): size of every dimension |
| a.ndim | 2: number of dimensions |
| a.size | 6: total elements |
| a.dtype | int64 |
| len(a) | 3: only the first dimension |
Ranges: step or count?
np.arange(start, stop, step)
- step
- distance between values. stop is excluded, like Python's
range. The step may be a float.
np.linspace(start, stop, num=50)
- num
- how many evenly spaced values. stop is included.
NumPy functions work on whole arrays at once: Y = np.exp(-X) * np.cos(2*pi*X) computes every value of X in one line.
min, max, mean… and the axis
np.amin(a, axis=None)
- axis
None: one result for the whole array.0: go down each column.1: go across each row. Same fornp.amax,np.mean,np.median,np.std,np.var.
np.amin(a, axis=0) → [3 2] (one per column). np.amin(a, axis=1) → [2 3 5] (one per row). Rule: the axis you name is the one that disappears from the shape: (3, 2) with axis=1 becomes (3,).Random numbers and reshaping
| Call | Gives |
|---|---|
| np.random.uniform(1, 10) | one random float between 1 and 10 (defaults 0 and 1) |
| np.random.randint(3, 500, 10) | 10 random integers from 3 to 499: the high value is excluded (that's why Lab 3 writes b+1) |
| np.random.randn(3, 10, 4) | shape (3, 10, 4) from the standard normal (mean 0, std 1) |
| A.reshape((4, 5, 10)) | same 200 elements, new shape (was 10 × 20) |
Broadcasting
When two shapes differ, NumPy compares them from the right. Matching sizes, or a missing dimension, are fine, and the smaller array is reused across the missing dimension. Nothing is actually copied.
x * y with x = [2, 3]: each row of y is multiplied element by element by [2, 3]. It is not a matrix product; that would be y @ x → [11 48 63].Does np.empty((3, 4)) give zeros?
np.zeros for zeros.np.arange(10, 50, 10) vs np.linspace(10, 50, 5)?
[10 20 30 40] (step, stop excluded) vs [10. 20. 30. 40. 50.] (count, stop included).np.amin([[7,2],[3,4],[5,6]], axis=1)?
[2 3 5]: one minimum per row. With axis=0: [3 2].Shape of x * y for x (2,) and y (3, 2)? What makes it work?
PyTorch tensors
A tensor is a NumPy-style array that can also run on a GPU and track gradients. Inputs, outputs and weights are all tensors.
import torch
Creating tensors
| Call | Gives | Memory |
|---|---|---|
| torch.tensor([[1, 2], [3, 4]]) | from a list; type inferred | copy |
| torch.from_numpy(np_array) | from a NumPy array; keeps its dtype | shared |
| torch.ones(2, 3) / torch.zeros(2, 3) | filled with 1. / 0. | new |
| torch.rand(2, 3) | random in [0, 1) | new |
| torch.randn(5, 3) | random from the standard normal | new |
| torch.ones_like(x) | same shape and dtype as x, filled with 1 | new |
Attributes: t.shape → torch.Size([3, 4]), t.dtype → torch.float32 (the default), t.device → cpu.
Indexing works like NumPy
t = torch.ones(4, 4) t[1, 1] = 6 # row 1, column 1 t[:, 2] = 3 # every row, column 2 t[3, :] = 0 # row 3, every column
Joining: torch.cat
torch.cat(tensors, dim=0)
- tensors
- a list of tensors, e.g.
[x, x] - dim
- which size in the shape gets added up. Every other size must match and stays the same.
Multiplying: element-wise or matrix?
| For a = [[1, 2], [3, 4]] | Code | Result |
|---|---|---|
| element-wise | a * a or a.mul(a) | [[1, 4], [9, 16]] |
| matrix product | a @ a or a.matmul(a) | [[7, 10], [15, 22]] |
In place, and memory shared with NumPy
A method ending in _ changes the tensor itself: t.add_(5), t.sub_(3), t.t_(), t.copy_(5). Without the underscore, a new tensor is returned and t is untouched. In-place ops save memory but are discouraged around gradients.
t = torch.ones(5) n = t.numpy() # n shares t's memory t.add_(2) # t: tensor([3., 3., 3., 3., 3.]) n: [3. 3. 3. 3. 3.] n2 = np.ones(5) t2 = torch.from_numpy(n2) np.add(n2, 1, out=n2) # t2: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
float64 appears because NumPy's default decimal type is float64 and from_numpy keeps it. PyTorch's default is float32, which isn't printed.
Which one shares memory: torch.tensor(arr) or torch.from_numpy(arr)?
from_numpy (and t.numpy()). torch.tensor always copies.x has shape (2, 3). torch.cat([x, x], dim=1)?
What does the trailing _ in add_ mean?
Autograd: gradients, backward and step
torch.autograd is PyTorch's automatic differentiation engine. It works out which way to move every weight to reduce the error.
A gradient is a slope
One weight w, loss = (w − 3)². The best w is 3. The gradient 2(w − 3) is the slope at the current w. Positive slope: decrease w. Negative slope: increase w. Gradient descent always steps against the slope: w_new = w − lr × gradient.
| Step | w | loss | gradient | update (lr = 0.25) |
|---|---|---|---|---|
| 0 | 5 | 4 | +4 | 5 − 1 = 4 |
| 1 | 4 | 1 | +2 | 4 − 0.5 = 3.5 |
| 2 | 3.5 | 0.25 | +1 | 3.5 − 0.25 = 3.25 |
Tracking: requires_grad and grad_fn
Every operation on a tracked tensor is recorded in a computational graph. A tensor produced by an operation gets a grad_fn: a receipt of how it was made. Tensors you create yourself are leaves and have grad_fn = None. The output of an operation requires grad if any input does.
loss.backward() walks it back and stops at the tensors that asked for gradients (w and b). x and y never get a gradient.x = torch.ones(5) # input y = torch.zeros(3) # target w = torch.randn(5, 3, requires_grad=True) b = torch.randn(3, requires_grad=True) z = torch.matmul(x, w) + b # z.grad_fn = <AddBackward0> loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)
backward() computes, step() changes
loss.backward()
Walks the graph backward (chain rule) and stores ∂loss/∂w in w.grad for every leaf with requires_grad=True. Values do not change. Gradients are added to what is already in .grad.
torch.optim.SGD(params, lr, momentum=0)
- params
- the tensors to update:
[w, b]ornet.parameters() - lr
- learning rate: the step size
- momentum
- keeps some of the previous step's direction (0.9 in Lab 9)
optimizer.step() applies w ← w − lr × w.grad. optimizer.zero_grad() empties every .grad.
| Moment (Lab 5 numbers) | w[0][0] | w.grad[0][0] |
|---|---|---|
| after creating w | 0.2070 | None |
after loss.backward() | 0.2070 (same) | 0.3062 |
after optimizer.step(), lr = 0.001 | 0.2067 | 0.3062 |
Check by hand (Lab 6): with a = [2., 3.], b = [6., 4.] and Q = 3a³ − b², the derivatives are ∂Q/∂a = 9a² and ∂Q/∂b = −2b. After Q.backward(gradient=torch.tensor([1., 1.])): a.grad = [36., 81.], b.grad = [-12., -8.]. (The gradient= argument is needed because Q is a vector, not a single number.)
Freezing and switching tracking off
requires_grad=Falsefreezes a tensor: its.gradstaysNoneandoptimizer.step()skips it. Lab 5's example: w frozen, b trainable → only b changes.with torch.no_grad():switches tracking off for a block. Use it when you only need predictions (testing): no graph, less memory, faster.
z = torch.matmul(x, w) + b
print(z.requires_grad) # True
with torch.no_grad():
z = torch.matmul(x, w) + b
print(z.requires_grad) # False
Right after loss.backward(), have w's values changed?
w.grad is filled. optimizer.step() changes the values.Why does z have a grad_fn but x doesn't?
w frozen (requires_grad=False), b trainable. After backward + step?
w.grad is None and w is unchanged; only b is updated. No error.Shape of w.grad when w is 5 × 3?
torch.Size([5, 3]): one slope per weight.Layers and activation functions
A neural network is a stack of simple functions with learnable weights. These are the pieces you stack.
import torch.nn as nn import torch.nn.functional as F
Linear (fully connected)
nn.Linear(in_features, out_features, bias=True)
- in_features
- length of each input vector
- out_features
- length of each output vector
Computes y = x·Aᵀ + b. Only the last dimension changes: Lab 7 feeds a (3, 2, 4) tensor into nn.Linear(4, 3) and gets (3, 2, 3).
Convolution: Conv2d
nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0)
- in_channels
- layers in the input: 3 for RGB, 1 for grayscale
- out_channels
- number of filters = number of output feature maps
- kernel_size
- each filter looks at a k × k window
- stride
- how many pixels the window jumps each move
- padding
- empty pixels added around the border
h_out = (h_in + 2p − D(k − 1) − 1) / s + 1With dilation D = 1:
h_out = (h_in + 2p − k) / s + 1. For 32, k = 5, p = 0, s = 1: (32 − 5)/1 + 1 = 28.Lab 7's small example: a 1 × 4 × 4 input through nn.Conv2d(1, 1, kernel_size=2) gives 1 × 3 × 3, because (4 − 2)/1 + 1 = 3.
Max pooling
nn.MaxPool2d(kernel_size, stride=None)
- stride
- defaults to kernel_size.
MaxPool2d(2, 2)halves height and width.
Function version: F.max_pool2d(x, 2) or F.max_pool2d(x, (2, 2)).
Activation functions
| Name | Formula | In PyTorch |
|---|---|---|
| ReLU | max(0, x) | F.relu(x) |
| sigmoid | 1 / (1 + e−x) | torch.sigmoid(x) |
| tanh | (ex − e−x) / (ex + e−x) | torch.tanh(x) |
| softmax | exi / Σj exj | nn.Softmax(dim=1) |
Softmax turns raw scores (logits) into probabilities between 0 and 1 that sum to 1: scores [2.0, 1.0, 0.1] → [0.659, 0.242, 0.099]. The order never changes, so the biggest score is always the biggest probability.
What do the numbers in nn.Conv2d(3, 6, 5) mean?
Output size of a 32 × 32 image after nn.Conv2d(3, 6, 5)?
Which formula is ReLU? Sigmoid? Softmax?
Building a network with nn.Module
A network is a Python class: __init__ creates the layers and forward says how data flows through them. Then you track the shape at every step.
The class, line by line
class Net(nn.Module): # 1. a new kind of nn.Module
def __init__(self): # 2. runs once, at net = Net()
super(Net, self).__init__() # set up nn.Module first (required)
self.conv1 = nn.Conv2d(3, 6, 5) # 3. layers are stored on self
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 10)
def forward(self, x): # 4. runs at net(x)
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5) # flatten
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x) # raw scores: no softmax
return x
net = Net()
outputs = net(images) # calls forward
You only write forward. The backward pass is created automatically by autograd. Each layer's "in" number equals the previous layer's "out": conv1 out 6 → conv2 in 6, fc1 out 120 → fc2 in 120.
Tracking the shape
- Conv, kernel 5, no padding: size − 4. Channels become the number of filters.
- MaxPool2d(2, 2): size ÷ 2 (rounded down). Channels stay.
- ReLU: no shape change.
- view: flattens 16 × 5 × 5 = 400, which must equal
fc1.in_features.
x.view(*shape)
Same data, new shape, no copy (shares memory). The total number of elements must stay the same. One size may be -1: "work it out". torch.arange(6).view(2, 3) → 2 × 3; x.view(-1, 400) on [16, 16, 5, 5] → [16, 400].
Lab 7's own network (grayscale, 3 × 3 kernels) on a 1 × 32 × 32 input: conv 3 → 30, pool → 15, conv 3 → 13, pool → 6 (13 ÷ 2 rounded down). So fc1 takes 16 × 6 × 6 = 576, which matches the printed torch.Size([1, 576]).
Other things Lab 7 shows
net.parameters()gives every weight and bias tensor. For Lab 7's net that's 10 (5 layers × weight + bias);params[0].size()is[6, 1, 3, 3].torch.nnonly accepts batches: [N, C, H, W]. For a single image, add a fake batch dimension withinput.unsqueeze(0).- A loss compares output and target:
criterion = nn.MSELoss(), thenloss = criterion(output, target). - The full update:
optimizer.zero_grad(), forward, loss,loss.backward(),optimizer.step().optim.Adam(net.parameters(), lr=0.01)is used the same way as SGD.
Do you write a backward() method in your Net class?
forward; autograd builds backward.Input [16, 3, 32, 32]: shape after each stage?
Why must fc1 have 400 input features?
Datasets, transforms and DataLoaders
How images get from disk into the network: load one sample, prepare it, group samples into batches.
for loop takes them one at a time.Loading a dataset
datasets.FashionMNIST(root, train=True, download=False, transform=None, target_transform=None)
- root
- folder where the files are stored, e.g.
"data" - train
True: training split (60,000 images).False: test split (10,000)- download
- download the files if they are not already in root
- transform
- applied to each image when it is loaded
- target_transform
- applied to each label
datasets.CIFAR10(...) takes the same arguments (50,000 / 10,000 images, 3 × 32 × 32).
A Dataset works like a list: img, label = training_data[7] gives one image and its label. FashionMNIST images are 1 × 28 × 28, grayscale, 10 classes (T-shirt, trouser, … ankle boot).
Transforms
| Transform | Does |
|---|---|
| transforms.Compose([t1, t2, …]) | chains transforms; they run in list order |
| transforms.ToTensor() | PIL image / ndarray (H × W × C, 0–255) → FloatTensor (C × H × W, 0.0–1.0) |
| transforms.Normalize(mean, std) | per channel: (x − mean) / std. Tensors only, no PIL images |
Order matters: Normalize needs a tensor with values in [0, 1], and only ToTensor produces one, so ToTensor goes first.
The DataLoader
DataLoader(dataset, batch_size=1, shuffle=False, num_workers=0)
- batch_size
- samples per batch
- shuffle
True: new random order every epoch (for training; reduces overfitting).False: same order every time (for testing and viewing)- num_workers
- helper processes that load data in parallel (0 = main process only)
Taking one batch and showing an image
train_features, train_labels = next(iter(train_dataloader)) print(train_features.size()) # torch.Size([64, 1, 28, 28]) print(train_labels.size()) # torch.Size([64]) img = train_features[0].squeeze() # (1, 28, 28) → (28, 28) plt.imshow(img, cmap="gray")
- [64, 1, 28, 28] = [batch, channels, height, width]: 64 grayscale images of 28 × 28.
iter(loader)gets the loader ready to hand out batches;next(...)takes the next one..squeeze()removes every size-1 dimension.plt.imshowwants (28, 28), not (1, 28, 28). The opposite is.unsqueeze(0).
What does ToTensor() do to pixel values?
Range after Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))?
torch.Size([64, 1, 28, 28]) means?
Which method turns (1, 28, 28) into (28, 28)?
.squeeze()Training and testing a classifier
Everything comes together on CIFAR-10: 10 classes of 3 × 32 × 32 color images. This lesson is worth the most points.
The five steps of Lab 9
- Load and normalize CIFAR-10:
Compose([ToTensor(), Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))]); train loader batch 16, shuffled; test loader batch 4;num_workers=2. - Define the CNN: the
Netclass from Lesson 7. - Define the loss and optimizer (below).
- Train: 10 epochs of the loop below.
- Test on the test set: accuracy 56%.
Loss and optimizer
nn.CrossEntropyLoss()
For classification. Takes raw scores [batch, 10] and labels [batch], applies softmax internally, and returns the mean loss over the batch.
optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
Lab 9's exact values. The official PyTorch tutorial uses 0.001; your lab uses 0.01.
The training loop
for epoch in range(n_epochs): # 10 times
running_loss = 0.0
for i, data in enumerate(trainloader, 0): # 3,125 times per epoch
inputs, labels = data
optimizer.zero_grad() # 1. clear
outputs = net(inputs) # 2. predict [16, 10]
loss = criterion(outputs, labels) # 3. score one number
loss.backward() # 4. blame fills .grad
optimizer.step() # 5. fix weights change
running_loss += loss.item() # .item(): tensor → Python number
if i % 2000 == 1999: # every 2000 batches
print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
running_loss = 0.0
Batch, iteration, epoch
| Word | Means | CIFAR-10, Lab 9 |
|---|---|---|
| batch | samples processed together | 16 images |
| iteration | one pass of the inner loop with one batch = one weight update | i counts them |
| epoch | one complete pass over the whole training set | 50,000 ÷ 16 = 3,125 iterations |
| training | several epochs | 10 epochs = 31,250 updates |
Why zero_grad, once per batch
.grad is a whiteboard. backward() adds its number to the board without erasing it; step() reads the board. Follow one weight:
| Moment | With zero_grad() | Without |
|---|---|---|
| batch 0: backward gives +2 | 0 + 2 = +2 | 0 + 2 = +2 |
| batch 0: step reads | +2 ✓ | +2 ✓ |
| batch 1: wipe | 0 | (skipped) still +2 |
| batch 1: backward gives −1 | 0 − 1 = −1 | 2 − 1 = +1 |
| batch 1: step reads | −1 ✓ increase w | +1 ✗ decrease w |
So: once per batch, at the start of the inner loop. Not per image: the 16 images give one averaged loss and one gradient. Not per epoch: step() runs every batch, so the board must be clean every batch.
Saving and reloading
PATH = './cifar_net.pth' torch.save(net.state_dict(), PATH) # save the learned weights trainednet = Net() # a fresh network… trainednet.load_state_dict(torch.load(PATH)) # …with the saved weights
Testing: from scores to a predicted class
torch.max(input, dim) → (values, indices)
Along dim, returns the largest values and their positions. With outputs [4, 10] and dim = 1: for each of the 4 images, the best score and its class number.
| img | 0 plane | 1 car | 2 bird | 3 cat | 4 deer | 5 dog | 6 frog | 7 horse | 8 ship | 9 truck | predicted | truth |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | −1.62 | −2.81 | 1.07 | 1.37 | 3.70 | 2.04 | 1.17 | 2.12 | −5.44 | −1.58 | 4 deer | frog |
| 1 | −0.41 | −2.13 | 0.77 | 1.49 | 0.75 | 1.61 | −0.48 | −0.00 | −1.00 | −1.42 | 5 dog | deer |
| 2 | 4.16 | 2.14 | 0.44 | −2.51 | −2.95 | −4.90 | −1.70 | −5.63 | 6.97 | 1.65 | 8 ship | ship |
| 3 | −0.33 | −3.72 | 2.81 | 2.55 | −1.32 | 1.97 | 0.40 | −0.67 | −0.92 | −3.06 | 2 bird | bird |
The real outputs Lab 9 printed. The lab calls these scores "energies"; they are logits, not probabilities (negative values, rows don't sum to 1). predicted = tensor([4, 5, 8, 2]); 2 of 4 correct.
correct, total = 0, 0
with torch.no_grad(): # testing: no gradients needed
for images, labels in testloader:
outputs = trainednet(images)
_, predicted = torch.max(outputs, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
print(100 * correct / total) # 56 % (random guessing: 10 %)
On a GPU: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu"), then net.to(device), and each batch inputs, labels = data[0].to(device), data[1].to(device).
Showing images: img = img / 2 + 0.5 undoes the normalization ([−1, 1] → [0, 1]), and np.transpose(npimg, (1, 2, 0)) reorders C × H × W to H × W × C for plt.imshow.
Order of the 5 lines in each iteration?
optimizer.zero_grad() → outputs = net(inputs) → loss = criterion(outputs, labels) → loss.backward() → optimizer.step().What goes wrong without zero_grad()?
What is in predicted after _, predicted = torch.max(outputs, 1)?
Lab 9: loss, lr, momentum, batch sizes, epochs, accuracy?
One epoch is…?
Data augmentation
More variety for free: make new training images by changing existing ones, so the model learns what stays the same.
What and why
Data augmentation creates new training examples by modifying existing ones (flip, crop, rotate, recolor) without changing their meaning: the label stays correct. Models are sensitive to small details; seeing many versions of the same thing teaches them the patterns that stay consistent, which builds robustness on unseen data. In PyTorch it happens in the Dataset's transform, usually a Compose.
The four families
| Family | Changes | Examples |
|---|---|---|
| Geometric | position, scale, orientation | HorizontalFlip, VerticalFlip, RandomRotation, RandomCrop, CenterCrop, Affine, Perspective |
| Color & appearance | lighting, sensor, pixel look | ColorJitter, Grayscale, GaussianBlur, noise, Posterize, Solarize |
| Occlusion / region | hides parts of the image | Cutout, RandomErasing, Hide-and-Seek |
| Sample mixing | blends images and labels | Mixup, CutMix, FMix |
The transforms used in the lab
| Signature | Does |
|---|---|
transforms.RandomCrop(size, padding=None, padding_mode='constant') | crop at a random spot; padding adds a border first |
transforms.CenterCrop(size) | crop the middle |
transforms.RandomHorizontalFlip(p=0.5) | mirror left–right with probability p |
transforms.RandomVerticalFlip(p=0.5) | mirror top–bottom with probability p |
transforms.RandomRotation(degrees) | 15 → rotate by −15° to +15° |
transforms.ColorJitter(brightness=0, contrast=0, saturation=0, hue=0) | random color changes |
transforms.Grayscale(num_output_channels=1) | remove color; batch becomes (4, 1, 32, 32) |
transforms.v2.MixUp(alpha=1.0, num_classes=None) | blend pairs of images and their labels (works on batches) |
transform_DACombined = transforms.Compose([
transforms.RandomCrop(25, padding=3),
transforms.RandomHorizontalFlip(),
transforms.RandomVerticalFlip(),
transforms.ToTensor(),
])
c10 = CIFAR10('./data', train=True, download=True, transform=transform_DACombined)
loader = DataLoader(c10, batch_size=4, shuffle=False) # same images every run
Rules from the lab
- Visualizing? Use
shuffle=False"so that we always see the same images" and can compare them. - Keep the image size. The network expects a fixed size (fc1 = 400 depends on 32 × 32). RandomCrop with padding keeps it.
- Augment training data only. The transfer-learning script uses flips, rotation and ColorJitter for training, but only ToTensor + Normalize for validation.
- Don't Mixup unrelated classes (dog + airplane) blindly.
- For display,
data[0].swapaxes(0,2).swapaxes(0,1)turns C × H × W into H × W × C.
What makes an augmentation good rather than harmful?
RandomCrop belongs to which family? Cutout? ColorJitter? Mixup?
Which transform gives a single-channel image?
transforms.Grayscale(num_output_channels=1)shuffle=True or False when visualizing augmented samples?
Transfer learning, train and eval mode
Reuse a network already trained on millions of images, retrain only its last layer, and switch it correctly between training and testing.
model = torchvision.models.resnet18(weights=None) # the structure
model.load_state_dict(torch.load(PATH)) # pretrained weights (ImageNet, 1000 classes)
for param in model.parameters(): # freeze everything
param.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 10) # new head: 512 → 10, trainable by default
optimizer = optim.Adam(model.fc.parameters(), lr=0.001) # optimize only the head
resnet18(weights=ResNet18_Weights.DEFAULT)downloads ImageNet weights;weights=Nonebuilds the same structure with random weights.- state_dict: a Python dictionary mapping each layer name to its tensor (
conv1.weight: [64, 3, 7, 7], …).torch.save/load_state_dictuse it. - The new layer is created after the freezing loop, so it is trainable. The script's head is
nn.Sequential(nn.Dropout(0.5), nn.Linear(model.fc.in_features, 10)). - Fine-tuning: also unfreeze the last block,
for param in model.layer4.parameters(): param.requires_grad = True.
train(), eval() and no_grad()
| Phase | Call | Dropout | BatchNorm | Gradients |
|---|---|---|---|---|
| training | model.train() | on (drops neurons) | uses batch stats, updates averages | tracked |
| validation / test | model.eval() | off | uses saved averages | still tracked! |
| validation / test | with torch.no_grad(): | (no effect) | (no effect) | off: less memory, faster |
for epoch in range(num_epochs):
model.train() # training phase
for images, labels in train_loader:
outputs = model(images)
loss = criterion(outputs, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
model.eval() # validation phase
with torch.no_grad():
for images, labels in val_loader:
outputs = model(images)
_, preds = torch.max(outputs, 1)
Forgetting them: no eval() → Dropout and BatchNorm keep training behaviour, so test results are noisy and unreliable. No no_grad() → correct results but wasted memory and time. No train() after validating → the next epoch trains with Dropout off and BatchNorm frozen. The Lab 9 net has neither Dropout nor BatchNorm, but ResNet has BatchNorm, which is why the script calls both.
How do you freeze a pretrained network's layers?
for param in model.parameters(): param.requires_grad = False, then replace model.fc with a new trainable layer.Does model.eval() stop gradient tracking?
torch.no_grad() stops tracking. Use both when testing.What is a state_dict?
Answer first. True/False and multiple choice check instantly; for written questions, reveal the model answer and tick what you covered. Sections 3 and 4 are worth 66 of the 120 points. Each question links to the lesson that teaches it.
Last-hour cheat sheet
Numbers to know by heart
Formulas
| Conv output | (h_in + 2p − k) / s + 1 |
| Full form (dilation D) | (h_in + 2p − D(k−1) − 1) / s + 1 |
| MaxPool2d(2, 2) | h ÷ 2 (rounded down) |
| Normalize | (x − mean) / std |
| Gradient descent | w ← w − lr × w.grad |
| Linear | y = x·Aᵀ + b |
| ReLU · sigmoid | max(0, x) · 1/(1 + e−x) |
| tanh · softmax | (ex−e−x)/(ex+e−x) · exi/Σexj |
The two loops
for epoch in range(10):
for i, data in enumerate(trainloader, 0):
inputs, labels = data
optimizer.zero_grad() # clear
outputs = net(inputs) # predict
loss = criterion(outputs, labels) # score
loss.backward() # blame: fills .grad
optimizer.step() # fix: changes weights
correct, total = 0, 0
with torch.no_grad():
for images, labels in testloader:
_, predicted = torch.max(net(images), 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
Save: torch.save(net.state_dict(), PATH) · Load: net = Net(); net.load_state_dict(torch.load(PATH))
Signatures: what each argument means
nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0)
nn.Linear(in_features, out_features, bias=True)
nn.MaxPool2d(kernel_size, stride=None)
DataLoader(dataset, batch_size=1, shuffle=False, num_workers=0)
datasets.CIFAR10(root, train=True, download=False, transform=None)
transforms.Normalize(mean, std)
optim.SGD(params, lr, momentum=0)
torch.max(input, dim) → (values, indices)
torch.cat(tensors, dim=0)
np.linspace(start, stop, num=50)
np.arange(start, stop, step)
plt.subplot(nrows, ncols, index)
Pairs people mix up
| loss.backward() | computes .grad, values unchanged | optimizer.step() | changes values |
| torch.tensor(a) | copy | torch.from_numpy(a) | shared memory |
| np.arange | step, stop excluded | np.linspace | count, stop included |
| axis=0 | down columns → one per column | axis=1 | across rows → one per row |
| cat dim=0 | (2,3)+(2,3) → (4,3) | cat dim=1 | → (2,6) |
| a * b | element-wise | a @ b | matrix product |
| t.add(2) | new tensor | t.add_(2) | in place |
| model.eval() | Dropout/BN test mode | torch.no_grad() | no gradient tracking |
| logits (Lab 9 outputs) | any value, raw scores | probabilities | 0–1, sum to 1 (softmax) |
| shuffle=True | training | shuffle=False | testing, visualizing |
| .squeeze() | removes size-1 dims | .unsqueeze(0) | adds a batch dim |
| RandomCrop | geometric | Cutout | occlusion |
True/False traps from the mock
np.emptygives zeros → False (garbage).backward()computes and updates → False (step updates).- You must write
backward()in your nn.Module → False (automatic). - Lab 9's forward ends with softmax → False (raw logits).
- Visualize augmentations with
shuffle=True→ False. from_numpyshares memory → True. ToTensor scales to [0, 1] → True.- linspace includes the end, arange doesn't → True. Augmentation must keep the label → True.
- Absolute words ("all", "both", "must") often signal False, but check each one.
Open-question skeletons (5 pts each)
- Jagged plot (Lab 2)
- plot joins points with straight lines → 3 points = 2 segments → use 201 points,
X = [0.01*i for i in range(0, 201)]. - backward vs step
- backward: chain rule, gradients into
.grad, values unchanged · step:w ← w − lr·grad· only one → no learning / stale or no update. - train / eval / no_grad
- train: Dropout/BN on · eval: off, saved stats · no_grad: no graph, saves memory · eval ≠ no_grad, use both · consequences.
- Augmentation
- definition · why (variety, robustness, less overfitting) · good = label stays true · harmful: 6 → 9, crop out the object.
- Lab 9 pipeline
- data + Normalize · Net · CrossEntropyLoss + SGD(0.01, 0.9) · zero_grad/forward/loss/backward/step · state_dict save/load · no_grad + torch.max → 56%.
- in_features
- formula · trace 32 → 28 → 14 → 10 → 5 · 16 channels · 16 × 5 × 5 = 400 · image size must not change.