{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "19649fbb",
   "metadata": {},
   "source": [
    "# Revision for IN400 - Applied Mathematics & Statistics for Computer Science\n",
    "\n",
    "## Vectors and Matrices: Theory and Practice with NumPy\n",
    "\n",
    "This notebook covers the fundamental concepts of vectors and matrices, essential for AI and machine learning.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa1411cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NumPy - run this cell first!\n",
    "import numpy as np\n",
    "\n",
    "print(\"NumPy version:\", np.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85dde9e7",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 1. Vectors\n",
    "\n",
    "A **vector** is an ordered collection of numbers (called components or elements) that can represent points in space, directions, magnitudes, or other quantities. Vectors are fundamental in AI for representing data, features, and transformations.\n",
    "\n",
    "### 1.1 Vector Representation\n",
    "\n",
    "**Mathematical notation:**\n",
    "\n",
    "- Column vector: $\\mathbf{v} = \\begin{bmatrix} v_1 \\\\ v_2 \\\\ \\vdots \\\\ v_n \\end{bmatrix}$\n",
    "- Row vector: $\\mathbf{v}^T = [v_1, v_2, \\ldots, v_n]$\n",
    "\n",
    "**Example:** In 2D space, $\\mathbf{v} = \\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}$ represents a point at coordinates (3, 4) or a direction from the origin.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc029e18",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating vectors\n",
    "v1 = np.array([3, 4])  # 2D vector\n",
    "v2 = np.array([1, 2, 3])  # 3D vector\n",
    "v3 = np.array([1, 2, 3, 4, 5])  # 5D vector\n",
    "\n",
    "print(\"v1:\", v1)\n",
    "print(\"v2:\", v2)\n",
    "print(\"Shape of v2:\", v2.shape)  # (3,) means 1D array with 3 elements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9975c086",
   "metadata": {},
   "source": [
    "### 1.2 Vector Operations\n",
    "\n",
    "#### 1.2.1 Vector Addition and Subtraction\n",
    "\n",
    "Two vectors can be added/subtracted **element-wise** if they have the same dimension.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{a} + \\mathbf{b} = \\begin{bmatrix} a_1 \\\\ a_2 \\\\ \\vdots \\\\ a_n \\end{bmatrix} + \\begin{bmatrix} b_1 \\\\ b_2 \\\\ \\vdots \\\\ b_n \\end{bmatrix} = \\begin{bmatrix} a_1 + b_1 \\\\ a_2 + b_2 \\\\ \\vdots \\\\ a_n + b_n \\end{bmatrix}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 \\\\ 2 \\\\ 3 \\end{bmatrix} + \\begin{bmatrix} 4 \\\\ 5 \\\\ 6 \\end{bmatrix} = \\begin{bmatrix} 5 \\\\ 7 \\\\ 9 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28b8830c",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "\n",
    "addition = a + b\n",
    "subtraction = a - b\n",
    "\n",
    "print(\"a + b =\", addition)  # [5 7 9]\n",
    "print(\"a - b =\", subtraction)  # [-3 -3 -3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6316c23",
   "metadata": {},
   "source": [
    "#### 1.2.2 Scalar Multiplication\n",
    "\n",
    "Multiplying a vector by a scalar (number) scales each component.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$c \\cdot \\mathbf{v} = c \\cdot \\begin{bmatrix} v_1 \\\\ v_2 \\\\ \\vdots \\\\ v_n \\end{bmatrix} = \\begin{bmatrix} c \\cdot v_1 \\\\ c \\cdot v_2 \\\\ \\vdots \\\\ c \\cdot v_n \\end{bmatrix}$$\n",
    "\n",
    "**Example:** $3 \\cdot \\begin{bmatrix} 1 \\\\ 2 \\end{bmatrix} = \\begin{bmatrix} 3 \\\\ 6 \\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c42950ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([1, 2, 3])\n",
    "scalar = 3\n",
    "\n",
    "result = scalar * v\n",
    "print(\"3 * v =\", result)  # [3 6 9]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96873e4b",
   "metadata": {},
   "source": [
    "#### 1.2.3 Dot Product (Inner Product)\n",
    "\n",
    "The dot product combines two vectors into a **single number**. It measures how much two vectors \"align\" with each other.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{a} \\cdot \\mathbf{b} = a_1b_1 + a_2b_2 + \\ldots + a_nb_n = \\sum_{i=1}^{n} a_i b_i$$\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 \\\\ 2 \\\\ 3 \\end{bmatrix} \\cdot \\begin{bmatrix} 4 \\\\ 5 \\\\ 6 \\end{bmatrix} = 1(4) + 2(5) + 3(6) = 4 + 10 + 18 = 32$$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- If $\\mathbf{a} \\cdot \\mathbf{b} = 0$, the vectors are **orthogonal** (perpendicular)\n",
    "- If $\\mathbf{a} \\cdot \\mathbf{b} > 0$, they point in similar directions\n",
    "- If $\\mathbf{a} \\cdot \\mathbf{b} < 0$, they point in opposite directions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0223cf9",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "\n",
    "# Method 1: Using np.dot()\n",
    "dot_product = np.dot(a, b)\n",
    "print(\"a · b =\", dot_product)  # 32\n",
    "\n",
    "# Method 2: Using @ operator\n",
    "dot_product = a @ b\n",
    "print(\"a · b =\", dot_product)  # 32\n",
    "\n",
    "# Method 3: Element-wise multiply then sum\n",
    "dot_product = np.sum(a * b)\n",
    "print(\"a · b =\", dot_product)  # 32"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b998426c",
   "metadata": {},
   "source": [
    "#### 1.2.4 Vector Magnitude (Norm)\n",
    "\n",
    "The **magnitude** (or length) of a vector is the distance from the origin to the point.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\|\\mathbf{v}\\| = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2} = \\sqrt{\\mathbf{v} \\cdot \\mathbf{v}}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\left\\|\\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}\\right\\| = \\sqrt{3^2 + 4^2} = \\sqrt{9 + 16} = \\sqrt{25} = 5$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9fca6fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([3, 4])\n",
    "\n",
    "# Method 1: Using np.linalg.norm()\n",
    "magnitude = np.linalg.norm(v)\n",
    "print(\"||v|| =\", magnitude)  # 5.0\n",
    "\n",
    "# Method 2: Manual calculation\n",
    "magnitude = np.sqrt(np.sum(v**2))\n",
    "print(\"||v|| =\", magnitude)  # 5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c189002",
   "metadata": {},
   "source": [
    "#### 1.2.5 Unit Vector (Normalization)\n",
    "\n",
    "A **unit vector** has magnitude 1 and indicates direction only.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\hat{\\mathbf{v}} = \\frac{\\mathbf{v}}{\\|\\mathbf{v}\\|}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{v} = \\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}, \\quad \\hat{\\mathbf{v}} = \\frac{1}{5}\\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix} = \\begin{bmatrix} 0.6 \\\\ 0.8 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a560a52",
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([3, 4])\n",
    "\n",
    "# Normalize the vector\n",
    "unit_vector = v / np.linalg.norm(v)\n",
    "print(\"Unit vector:\", unit_vector)  # [0.6 0.8]\n",
    "print(\"Magnitude of unit vector:\", np.linalg.norm(unit_vector))  # 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f800d451",
   "metadata": {},
   "source": [
    "#### 1.2.6 Cross Product (Vector Product)\n",
    "\n",
    "The **cross product** is an operation on two 3D vectors that produces a third vector **perpendicular** to both input vectors. It's only defined for 3D vectors.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{a} \\times \\mathbf{b} = \\begin{bmatrix} a_2b_3 - a_3b_2 \\\\ a_3b_1 - a_1b_3 \\\\ a_1b_2 - a_2b_1 \\end{bmatrix}$$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- The result is perpendicular to both input vectors\n",
    "- The magnitude is: $\\|\\mathbf{a} \\times \\mathbf{b}\\| = \\|\\mathbf{a}\\| \\|\\mathbf{b}\\| \\sin(\\theta)$, where $\\theta$ is the angle between vectors\n",
    "- **Not commutative**: $\\mathbf{a} \\times \\mathbf{b} = -(\\mathbf{b} \\times \\mathbf{a})$\n",
    "- If vectors are parallel: $\\mathbf{a} \\times \\mathbf{b} = \\mathbf{0}$\n",
    "\n",
    "**Applications:**\n",
    "\n",
    "- Finding normal vectors (computer graphics, physics)\n",
    "- Computing torque and angular momentum\n",
    "- Determining the orientation of surfaces\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 \\\\ 2 \\\\ 3 \\end{bmatrix} \\times \\begin{bmatrix} 4 \\\\ 5 \\\\ 6 \\end{bmatrix} = \\begin{bmatrix} 2(6) - 3(5) \\\\ 3(4) - 1(6) \\\\ 1(5) - 2(4) \\end{bmatrix} = \\begin{bmatrix} -3 \\\\ 6 \\\\ -3 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af810f1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "\n",
    "# Cross product using np.cross()\n",
    "cross_product = np.cross(a, b)\n",
    "print(\"a × b =\", cross_product)  # [-3  6 -3]\n",
    "\n",
    "# Verify: the result is perpendicular to both a and b\n",
    "# Dot product with perpendicular vectors should be 0\n",
    "print(\"\\n(a × b) · a =\", np.dot(cross_product, a))  # Should be 0\n",
    "print(\"(a × b) · b =\", np.dot(cross_product, b))  # Should be 0\n",
    "\n",
    "# Example: Find a vector perpendicular to the xy-plane\n",
    "x_axis = np.array([1, 0, 0])\n",
    "y_axis = np.array([0, 1, 0])\n",
    "z_direction = np.cross(x_axis, y_axis)\n",
    "print(\"\\nx_axis × y_axis =\", z_direction)  # [0 0 1] - points in z direction\n",
    "\n",
    "# Example: Parallel vectors have zero cross product\n",
    "parallel_a = np.array([2, 4, 6])\n",
    "parallel_b = np.array([1, 2, 3])  # parallel_b = 0.5 * parallel_a\n",
    "result = np.cross(parallel_b, parallel_a)\n",
    "print(\"\\nCross product of parallel vectors:\", result)  # [0 0 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f958abca",
   "metadata": {},
   "source": [
    "#### 1.2.7 Vector Projection\n",
    "\n",
    "**Vector projection** projects one vector onto another, finding the component of one vector in the direction of another.\n",
    "\n",
    "**Mathematical definition:**\n",
    "\n",
    "Projection of $\\mathbf{a}$ onto $\\mathbf{b}$:\n",
    "$$\\text{proj}_{\\mathbf{b}}\\mathbf{a} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\|\\mathbf{b}\\|^2}\\mathbf{b} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\mathbf{b} \\cdot \\mathbf{b}}\\mathbf{b}$$\n",
    "\n",
    "The **scalar projection** (length of the projection):\n",
    "$$\\text{comp}_{\\mathbf{b}}\\mathbf{a} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\|\\mathbf{b}\\|}$$\n",
    "\n",
    "**Geometric interpretation:**\n",
    "\n",
    "- The projection shows \"how much\" of vector $\\mathbf{a}$ points in the direction of $\\mathbf{b}$\n",
    "- If perpendicular: projection = 0\n",
    "- If parallel: projection = $\\mathbf{a}$ (or $-\\mathbf{a}$)\n",
    "\n",
    "**Applications:**\n",
    "\n",
    "- Linear regression (projecting data onto a line/plane)\n",
    "- Component analysis\n",
    "- Shadow calculations in graphics\n",
    "- Signal decomposition\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{a} = \\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}, \\quad \\mathbf{b} = \\begin{bmatrix} 1 \\\\ 0 \\end{bmatrix}$$\n",
    "$$\\text{proj}_{\\mathbf{b}}\\mathbf{a} = \\frac{3(1) + 4(0)}{1^2 + 0^2}\\begin{bmatrix} 1 \\\\ 0 \\end{bmatrix} = 3\\begin{bmatrix} 1 \\\\ 0 \\end{bmatrix} = \\begin{bmatrix} 3 \\\\ 0 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bcc0eb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 1: Project vector a onto vector b\n",
    "a = np.array([3, 4])\n",
    "b = np.array([1, 0])\n",
    "\n",
    "# Calculate projection\n",
    "proj_b_a = (np.dot(a, b) / np.dot(b, b)) * b\n",
    "print(\"Vector a:\", a)\n",
    "print(\"Vector b:\", b)\n",
    "print(\"Projection of a onto b:\", proj_b_a)  # [3 0]\n",
    "\n",
    "# Scalar projection (length)\n",
    "scalar_proj = np.dot(a, b) / np.linalg.norm(b)\n",
    "print(\"Scalar projection (length):\", scalar_proj)  # 3.0\n",
    "\n",
    "# Example 2: 3D projection\n",
    "a_3d = np.array([1, 2, 3])\n",
    "b_3d = np.array([1, 1, 0])\n",
    "\n",
    "proj_3d = (np.dot(a_3d, b_3d) / np.dot(b_3d, b_3d)) * b_3d\n",
    "print(\"\\n3D Example:\")\n",
    "print(\"Vector a:\", a_3d)\n",
    "print(\"Vector b:\", b_3d)\n",
    "print(\"Projection of a onto b:\", proj_3d)\n",
    "\n",
    "# Example 3: Perpendicular vectors (projection = 0)\n",
    "perpendicular_a = np.array([1, 0, 0])\n",
    "perpendicular_b = np.array([0, 1, 0])\n",
    "\n",
    "proj_perp = (\n",
    "    np.dot(perpendicular_a, perpendicular_b) / np.dot(perpendicular_b, perpendicular_b)\n",
    ") * perpendicular_b\n",
    "print(\"\\nPerpendicular vectors:\")\n",
    "print(\"Projection:\", proj_perp)  # [0 0 0]\n",
    "\n",
    "# Example 4: Decompose a vector into parallel and perpendicular components\n",
    "a_decomp = np.array([3, 4])\n",
    "b_decomp = np.array([1, 0])\n",
    "\n",
    "parallel_component = (\n",
    "    np.dot(a_decomp, b_decomp) / np.dot(b_decomp, b_decomp)\n",
    ") * b_decomp\n",
    "perpendicular_component = a_decomp - parallel_component\n",
    "\n",
    "print(\"\\nVector decomposition:\")\n",
    "print(\"Original vector:\", a_decomp)\n",
    "print(\"Parallel component:\", parallel_component)\n",
    "print(\"Perpendicular component:\", perpendicular_component)\n",
    "print(\"Sum (should equal original):\", parallel_component + perpendicular_component)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c31d5fd",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Matrices\n",
    "\n",
    "A **matrix** is a 2D array of numbers arranged in rows and columns. Matrices are essential in AI for representing datasets, transformations, and neural network weights.\n",
    "\n",
    "### 2.1 Matrix Representation\n",
    "\n",
    "**Mathematical notation:**\n",
    "$$\\mathbf{A} = \\begin{bmatrix} a_{11} & a_{12} & \\cdots & a_{1n} \\\\ a_{21} & a_{22} & \\cdots & a_{2n} \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ a_{m1} & a_{m2} & \\cdots & a_{mn} \\end{bmatrix}$$\n",
    "\n",
    "- An **m × n matrix** has m rows and n columns\n",
    "- Element at row i, column j is denoted as $a_{ij}$ or $A[i,j]$\n",
    "\n",
    "**Example:** A 2×3 matrix:\n",
    "$$\\mathbf{A} = \\begin{bmatrix} 1 & 2 & 3 \\\\ 4 & 5 & 6 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd0f0fb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating matrices\n",
    "A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3 matrix\n",
    "\n",
    "B = np.array([[1, 2], [3, 4], [5, 6]])  # 3x2 matrix\n",
    "\n",
    "print(\"Matrix A:\")\n",
    "print(A)\n",
    "print(\"Shape:\", A.shape)  # (2, 3) means 2 rows, 3 columns\n",
    "\n",
    "# Identity matrix (diagonal of 1s)\n",
    "I = np.eye(3)\n",
    "print(\"\\nIdentity matrix:\")\n",
    "print(I)\n",
    "\n",
    "# Zero matrix\n",
    "Z = np.zeros((2, 3))\n",
    "print(\"\\nZero matrix:\")\n",
    "print(Z)\n",
    "\n",
    "# Random matrix\n",
    "R = np.random.rand(2, 3)  # Random values between 0 and 1\n",
    "print(\"\\nRandom matrix:\")\n",
    "print(R)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff2fc432",
   "metadata": {},
   "source": [
    "### 2.2 Matrix Operations\n",
    "\n",
    "#### 2.2.1 Matrix Addition and Subtraction\n",
    "\n",
    "Matrices of the **same dimensions** can be added/subtracted element-wise.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{A} + \\mathbf{B} = \\begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\\\ a_{21} + b_{21} & a_{22} + b_{22} \\end{bmatrix}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix} + \\begin{bmatrix} 5 & 6 \\\\ 7 & 8 \\end{bmatrix} = \\begin{bmatrix} 6 & 8 \\\\ 10 & 12 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb0a8a49",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])\n",
    "B = np.array([[5, 6], [7, 8]])\n",
    "\n",
    "addition = A + B\n",
    "subtraction = A - B\n",
    "\n",
    "print(\"A + B =\")\n",
    "print(addition)\n",
    "# [[6  8]\n",
    "#  [10 12]]\n",
    "\n",
    "print(\"\\nA - B =\")\n",
    "print(subtraction)\n",
    "# [[-4 -4]\n",
    "#  [-4 -4]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd21fdbb",
   "metadata": {},
   "source": [
    "#### 2.2.2 Scalar Multiplication\n",
    "\n",
    "Multiply each element by a scalar.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$c \\cdot \\mathbf{A} = \\begin{bmatrix} c \\cdot a_{11} & c \\cdot a_{12} \\\\ c \\cdot a_{21} & c \\cdot a_{22} \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b4c1d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])\n",
    "result = 3 * A\n",
    "\n",
    "print(\"3 * A =\")\n",
    "print(result)\n",
    "# [[3  6]\n",
    "#  [9 12]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "279954cc",
   "metadata": {},
   "source": [
    "#### 2.2.3 Matrix Multiplication\n",
    "\n",
    "**Key rule:** To multiply $\\mathbf{A}$ (m×n) by $\\mathbf{B}$ (p×q), we need **n = p**. The result is an m×q matrix.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$(\\mathbf{AB})_{ij} = \\sum_{k=1}^{n} a_{ik} b_{kj}$$\n",
    "\n",
    "The element at position (i,j) in the result is the dot product of row i of A with column j of B.\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix} \\times \\begin{bmatrix} 5 & 6 \\\\ 7 & 8 \\end{bmatrix} = \\begin{bmatrix} 1(5)+2(7) & 1(6)+2(8) \\\\ 3(5)+4(7) & 3(6)+4(8) \\end{bmatrix} = \\begin{bmatrix} 19 & 22 \\\\ 43 & 50 \\end{bmatrix}$$\n",
    "\n",
    "**Important:** Matrix multiplication is **not commutative**: $\\mathbf{AB} \\neq \\mathbf{BA}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f76060c",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])  # 2x2\n",
    "\n",
    "B = np.array([[5, 6], [7, 8]])  # 2x2\n",
    "\n",
    "# Matrix multiplication\n",
    "C = np.dot(A, B)\n",
    "# Or using @ operator\n",
    "C = A @ B\n",
    "\n",
    "print(\"A × B =\")\n",
    "print(C)\n",
    "# [[19 22]\n",
    "#  [43 50]]\n",
    "\n",
    "# Matrix-vector multiplication\n",
    "v = np.array([1, 2])\n",
    "result = A @ v\n",
    "\n",
    "print(\"\\nA × v =\")\n",
    "print(result)  # [5 11]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d9baef6",
   "metadata": {},
   "source": [
    "#### 2.2.4 Matrix Transpose\n",
    "\n",
    "**Transpose** flips a matrix over its diagonal, converting rows to columns.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$(\\mathbf{A}^T)_{ij} = A_{ji}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{A} = \\begin{bmatrix} 1 & 2 & 3 \\\\ 4 & 5 & 6 \\end{bmatrix}, \\quad \\mathbf{A}^T = \\begin{bmatrix} 1 & 4 \\\\ 2 & 5 \\\\ 3 & 6 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "514c5129",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3\n",
    "\n",
    "A_transpose = A.T\n",
    "\n",
    "print(\"A =\")\n",
    "print(A)\n",
    "print(\"\\nA^T =\")\n",
    "print(A_transpose)  # 3x2 matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a04b9792",
   "metadata": {},
   "source": [
    "#### 2.2.5 Matrix Inverse\n",
    "\n",
    "The **inverse** of a square matrix $\\mathbf{A}$ is denoted $\\mathbf{A}^{-1}$ and satisfies:\n",
    "$$\\mathbf{A} \\mathbf{A}^{-1} = \\mathbf{A}^{-1} \\mathbf{A} = \\mathbf{I}$$\n",
    "\n",
    "**Important:** Not all matrices have an inverse! A matrix is **invertible** if its determinant is non-zero.\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{A} = \\begin{bmatrix} 4 & 7 \\\\ 2 & 6 \\end{bmatrix}, \\quad \\mathbf{A}^{-1} = \\begin{bmatrix} 0.6 & -0.7 \\\\ -0.2 & 0.4 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c32ddbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[4, 7], [2, 6]])\n",
    "\n",
    "# Calculate inverse\n",
    "A_inv = np.linalg.inv(A)\n",
    "\n",
    "print(\"A =\")\n",
    "print(A)\n",
    "print(\"\\nA^(-1) =\")\n",
    "print(A_inv)\n",
    "\n",
    "# Verify: A × A^(-1) = I\n",
    "identity = A @ A_inv\n",
    "print(\"\\nA × A^(-1) =\")\n",
    "print(identity)  # Should be close to [[1, 0], [0, 1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "521e894d",
   "metadata": {},
   "source": [
    "#### 2.2.6 Matrix Determinant\n",
    "\n",
    "The **determinant** is a scalar value that can be computed from a square matrix. It provides important information about the matrix:\n",
    "\n",
    "- If det(A) ≠ 0, the matrix is **invertible**\n",
    "- If det(A) = 0, the matrix is **singular** (not invertible)\n",
    "- The determinant represents the volume scaling factor of the linear transformation\n",
    "\n",
    "**Mathematical definition:**\n",
    "\n",
    "For a 2×2 matrix:\n",
    "$$\\text{det}\\begin{bmatrix} a & b \\\\ c & d \\end{bmatrix} = ad - bc$$\n",
    "\n",
    "For a 3×3 matrix:\n",
    "$$\\text{det}\\begin{bmatrix} a & b & c \\\\ d & e & f \\\\ g & h & i \\end{bmatrix} = a(ei - fh) - b(di - fg) + c(dh - eg)$$\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "2×2: $\\text{det}\\begin{bmatrix} 3 & 8 \\\\ 4 & 6 \\end{bmatrix} = 3(6) - 8(4) = 18 - 32 = -14$\n",
    "\n",
    "3×3: $\\text{det}\\begin{bmatrix} 1 & 2 & 3 \\\\ 0 & 1 & 4 \\\\ 5 & 6 & 0 \\end{bmatrix} = 1(1 \\cdot 0 - 4 \\cdot 6) - 2(0 \\cdot 0 - 4 \\cdot 5) + 3(0 \\cdot 6 - 1 \\cdot 5) = 1(-24) - 2(-20) + 3(-5) = -24 + 40 - 15 = 1$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2268cef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 1: Determinant of a 2x2 matrix\n",
    "A_2x2 = np.array([[3, 8], [4, 6]])\n",
    "\n",
    "det_A = np.linalg.det(A_2x2)\n",
    "print(\"Matrix A (2x2):\")\n",
    "print(A_2x2)\n",
    "print(\"Determinant of A:\", det_A)  # -14.0\n",
    "\n",
    "# Example 2: Determinant of a 3x3 matrix\n",
    "A_3x3 = np.array([[1, 2, 3], [0, 1, 4], [5, 6, 0]])\n",
    "\n",
    "det_B = np.linalg.det(A_3x3)\n",
    "print(\"\\nMatrix B (3x3):\")\n",
    "print(A_3x3)\n",
    "print(\"Determinant of B:\", det_B)  # 1.0\n",
    "\n",
    "# Example 3: Check if a matrix is invertible\n",
    "singular_matrix = np.array([[2, 4], [1, 2]])\n",
    "\n",
    "det_singular = np.linalg.det(singular_matrix)\n",
    "print(\"\\nSingular Matrix:\")\n",
    "print(singular_matrix)\n",
    "print(\"Determinant:\", det_singular)  # 0.0 (or very close to 0)\n",
    "\n",
    "if abs(det_singular) < 1e-10:\n",
    "    print(\"This matrix is SINGULAR (not invertible)\")\n",
    "else:\n",
    "    print(\"This matrix is INVERTIBLE\")\n",
    "\n",
    "# Example 4: Manual calculation for 2x2 (to verify)\n",
    "a, b = 3, 8\n",
    "c, d = 4, 6\n",
    "manual_det = a * d - b * c\n",
    "print(f\"\\nManual calculation: {a}*{d} - {b}*{c} = {manual_det}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09a0d008",
   "metadata": {},
   "source": [
    "#### 2.2.7 Eigenvalues and Eigenvectors\n",
    "\n",
    "**Eigenvalues** and **eigenvectors** are fundamental concepts in linear algebra. For a square matrix $\\mathbf{A}$, an eigenvector $\\mathbf{v}$ and its corresponding eigenvalue $\\lambda$ satisfy:\n",
    "\n",
    "$$\\mathbf{A}\\mathbf{v} = \\lambda\\mathbf{v}$$\n",
    "\n",
    "**Interpretation:**\n",
    "\n",
    "- When matrix $\\mathbf{A}$ is applied to eigenvector $\\mathbf{v}$, the result is the same vector scaled by $\\lambda$\n",
    "- The eigenvector's **direction** is preserved, only the **magnitude** changes\n",
    "- $\\lambda$ (eigenvalue) is the scaling factor\n",
    "\n",
    "**Finding eigenvalues:**\n",
    "Solve the characteristic equation:\n",
    "$$\\text{det}(\\mathbf{A} - \\lambda\\mathbf{I}) = 0$$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- An n×n matrix has n eigenvalues (counting multiplicities)\n",
    "- Eigenvectors corresponding to different eigenvalues are linearly independent\n",
    "- For symmetric matrices, eigenvalues are real and eigenvectors are orthogonal\n",
    "\n",
    "**Applications in AI/ML:**\n",
    "\n",
    "- **Principal Component Analysis (PCA)**: Dimensionality reduction\n",
    "- **Google PageRank**: Website ranking algorithm\n",
    "- **Markov Chains**: State transitions and steady states\n",
    "- **Image Compression**: Spectral decomposition\n",
    "- **Stability Analysis**: Neural network dynamics\n",
    "- **Graph Analysis**: Community detection\n",
    "\n",
    "**Example:**\n",
    "For matrix $\\mathbf{A} = \\begin{bmatrix} 4 & 2 \\\\ 1 & 3 \\end{bmatrix}$:\n",
    "\n",
    "- Eigenvalues: $\\lambda_1 = 5, \\lambda_2 = 2$\n",
    "- Eigenvector for $\\lambda_1 = 5$: $\\mathbf{v}_1 = \\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$\n",
    "- Verification: $\\begin{bmatrix} 4 & 2 \\\\ 1 & 3 \\end{bmatrix}\\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix} = \\begin{bmatrix} 10 \\\\ 5 \\end{bmatrix} = 5\\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$ ✓\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "312a6398",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 1: Basic eigenvalue and eigenvector calculation\n",
    "A = np.array([[4, 2], [1, 3]])\n",
    "\n",
    "# Calculate eigenvalues and eigenvectors\n",
    "eigenvalues, eigenvectors = np.linalg.eig(A)\n",
    "\n",
    "print(\"Matrix A:\")\n",
    "print(A)\n",
    "print(\"\\nEigenvalues:\", eigenvalues)\n",
    "print(\"\\nEigenvectors:\")\n",
    "print(eigenvectors)\n",
    "\n",
    "# Verify: A @ v = λ * v for the first eigenvector\n",
    "v1 = eigenvectors[:, 0]  # First eigenvector (column 0)\n",
    "lambda1 = eigenvalues[0]\n",
    "\n",
    "left_side = A @ v1\n",
    "right_side = lambda1 * v1\n",
    "\n",
    "print(f\"\\nVerification for λ₁ = {lambda1:.4f}:\")\n",
    "print(f\"A @ v₁ = {left_side}\")\n",
    "print(f\"λ₁ * v₁ = {right_side}\")\n",
    "print(f\"Equal? {np.allclose(left_side, right_side)}\")\n",
    "\n",
    "# Example 2: Symmetric matrix (real eigenvalues, orthogonal eigenvectors)\n",
    "symmetric_matrix = np.array([[2, 1], [1, 2]])\n",
    "\n",
    "eig_vals, eig_vecs = np.linalg.eig(symmetric_matrix)\n",
    "\n",
    "print(\"\\n\\nSymmetric Matrix:\")\n",
    "print(symmetric_matrix)\n",
    "print(\"\\nEigenvalues:\", eig_vals)\n",
    "print(\"\\nEigenvectors:\")\n",
    "print(eig_vecs)\n",
    "\n",
    "# Check orthogonality: dot product should be 0\n",
    "dot_product = np.dot(eig_vecs[:, 0], eig_vecs[:, 1])\n",
    "print(f\"\\nDot product of eigenvectors: {dot_product:.10f} (should be ~0)\")\n",
    "\n",
    "# Example 3: 3x3 matrix\n",
    "A_3x3 = np.array([[6, -1, 0], [-1, 6, -1], [0, -1, 6]])\n",
    "\n",
    "eig_vals_3d, eig_vecs_3d = np.linalg.eig(A_3x3)\n",
    "\n",
    "print(\"\\n\\n3x3 Matrix:\")\n",
    "print(A_3x3)\n",
    "print(\"\\nEigenvalues:\", eig_vals_3d)\n",
    "\n",
    "\n",
    "# Example 4: Application - Power iteration to find dominant eigenvalue\n",
    "def power_iteration(A, num_iterations=10):\n",
    "    \"\"\"Find the dominant eigenvalue and eigenvector using power iteration\"\"\"\n",
    "    n = A.shape[0]\n",
    "    v = np.random.rand(n)  # Random initial vector\n",
    "\n",
    "    for i in range(num_iterations):\n",
    "        # Multiply by matrix\n",
    "        v = A @ v\n",
    "        # Normalize\n",
    "        v = v / np.linalg.norm(v)\n",
    "\n",
    "    # Calculate eigenvalue: λ = (v^T A v) / (v^T v)\n",
    "    eigenvalue = (v.T @ A @ v) / (v.T @ v)\n",
    "    return eigenvalue, v\n",
    "\n",
    "\n",
    "dominant_eval, dominant_evec = power_iteration(A, num_iterations=20)\n",
    "print(f\"\\n\\nPower Iteration (finds dominant eigenvalue):\")\n",
    "print(f\"Dominant eigenvalue: {dominant_eval:.4f}\")\n",
    "print(f\"Compare with np.linalg.eig: {max(eigenvalues):.4f}\")\n",
    "\n",
    "# Example 5: Trace and determinant relation to eigenvalues\n",
    "print(f\"\\n\\nTrace-Eigenvalue relation:\")\n",
    "print(f\"Trace of A: {np.trace(A):.4f}\")\n",
    "print(f\"Sum of eigenvalues: {np.sum(eigenvalues):.4f}\")\n",
    "print(f\"Determinant of A: {np.linalg.det(A):.4f}\")\n",
    "print(f\"Product of eigenvalues: {np.prod(eigenvalues):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a36c303",
   "metadata": {},
   "source": [
    "#### 2.2.8 Element-wise Operations\n",
    "\n",
    "Sometimes we need element-wise operations (Hadamard product), not matrix multiplication.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d61117ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])\n",
    "B = np.array([[5, 6], [7, 8]])\n",
    "\n",
    "# Element-wise multiplication\n",
    "element_wise = A * B\n",
    "print(\"A ⊙ B (element-wise) =\")\n",
    "print(element_wise)\n",
    "# [[5  12]\n",
    "#  [21 32]]\n",
    "\n",
    "# Element-wise division\n",
    "element_div = A / B\n",
    "print(\"\\nA ⊘ B (element-wise) =\")\n",
    "print(element_div)\n",
    "\n",
    "# Element-wise power\n",
    "powered = A**2\n",
    "print(\"\\nA^2 (element-wise) =\")\n",
    "print(powered)\n",
    "# [[1  4]\n",
    "#  [9 16]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "164090bf",
   "metadata": {},
   "source": [
    "### 2.3 Special Matrices\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed5e9b21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identity matrix (3x3)\n",
    "I = np.eye(3)\n",
    "print(\"Identity matrix:\")\n",
    "print(I)\n",
    "\n",
    "# Diagonal matrix\n",
    "diag = np.diag([1, 2, 3])\n",
    "print(\"\\nDiagonal matrix:\")\n",
    "print(diag)\n",
    "\n",
    "# Upper triangular\n",
    "upper = np.triu(np.ones((3, 3)))\n",
    "print(\"\\nUpper triangular:\")\n",
    "print(upper)\n",
    "\n",
    "# Lower triangular\n",
    "lower = np.tril(np.ones((3, 3)))\n",
    "print(\"\\nLower triangular:\")\n",
    "print(lower)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3d4e532",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Practical Applications in AI\n",
    "\n",
    "### 3.1 Data Representation\n",
    "\n",
    "- **Vectors**: Each data point (e.g., image, text) is represented as a vector\n",
    "- **Matrices**: A dataset is a matrix where each row is a data sample\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "478aea83",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataset: 5 students with 3 test scores each\n",
    "grades = np.array(\n",
    "    [\n",
    "        [85, 90, 88],  # Student 1\n",
    "        [78, 85, 80],  # Student 2\n",
    "        [92, 95, 93],  # Student 3\n",
    "        [70, 75, 72],  # Student 4\n",
    "        [88, 87, 90],\n",
    "    ]\n",
    ")  # Student 5\n",
    "\n",
    "print(\"Dataset shape:\", grades.shape)  # (5, 3)\n",
    "\n",
    "# Calculate mean score for each student\n",
    "mean_per_student = np.mean(grades, axis=1)\n",
    "print(\"Mean per student:\", mean_per_student)\n",
    "\n",
    "# Calculate mean score for each test\n",
    "mean_per_test = np.mean(grades, axis=0)\n",
    "print(\"Mean per test:\", mean_per_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d006b28",
   "metadata": {},
   "source": [
    "### 3.2 Linear Transformations\n",
    "\n",
    "Matrix multiplication applies transformations to vectors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50856926",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rotation matrix (90 degrees counterclockwise)\n",
    "theta = np.pi / 2  # 90 degrees in radians\n",
    "rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])\n",
    "\n",
    "# Original point\n",
    "point = np.array([1, 0])\n",
    "\n",
    "# Rotate the point\n",
    "rotated_point = rotation @ point\n",
    "print(\"Original point:\", point)\n",
    "print(\"Rotated point:\", rotated_point)  # Should be close to [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1897d31d",
   "metadata": {},
   "source": [
    "### 3.3 Solving Linear Systems\n",
    "\n",
    "Systems of linear equations can be solved using matrices.\n",
    "\n",
    "**System:**\n",
    "$$2x + 3y = 8$$\n",
    "$$4x + y = 10$$\n",
    "\n",
    "**Matrix form:** $\\mathbf{Ax} = \\mathbf{b}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee9fd4a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coefficient matrix\n",
    "A = np.array([[2, 3], [4, 1]])\n",
    "\n",
    "# Constants vector\n",
    "b = np.array([8, 10])\n",
    "\n",
    "# Solve for x\n",
    "x = np.linalg.solve(A, b)\n",
    "print(\"Solution:\", x)  # x = [2, 2] means x=2, y=2\n",
    "\n",
    "# Verify solution\n",
    "print(\"Verification A×x =\", A @ x)  # Should equal b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee4a8fe",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 4. Summary Cheat Sheet\n",
    "\n",
    "| Operation             | Math Notation                                               | NumPy Code                    |\n",
    "| --------------------- | ----------------------------------------------------------- | ----------------------------- |\n",
    "| Create vector         | $\\mathbf{v} = [1, 2, 3]$                                    | `v = np.array([1, 2, 3])`     |\n",
    "| Create matrix         | $\\mathbf{A} = \\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix}$ | `A = np.array([[1,2],[3,4]])` |\n",
    "| Vector addition       | $\\mathbf{a} + \\mathbf{b}$                                   | `a + b`                       |\n",
    "| Scalar multiplication | $c \\cdot \\mathbf{v}$                                        | `c * v`                       |\n",
    "| Dot product           | $\\mathbf{a} \\cdot \\mathbf{b}$                               | `np.dot(a, b)` or `a @ b`     |\n",
    "| Cross product (3D)    | $\\mathbf{a} \\times \\mathbf{b}$                              | `np.cross(a, b)`              |\n",
    "| Vector magnitude      | $\\|\\mathbf{v}\\|$                                            | `np.linalg.norm(v)`           |\n",
    "| Vector projection     | $\\text{proj}_{\\mathbf{b}}\\mathbf{a}$                        | `(a@b / b@b) * b`             |\n",
    "| Matrix multiplication | $\\mathbf{AB}$                                               | `A @ B` or `np.dot(A, B)`     |\n",
    "| Element-wise mult.    | $\\mathbf{A} \\odot \\mathbf{B}$                               | `A * B`                       |\n",
    "| Transpose             | $\\mathbf{A}^T$                                              | `A.T`                         |\n",
    "| Determinant           | $\\text{det}(\\mathbf{A})$ or $\\|\\mathbf{A}\\|$                | `np.linalg.det(A)`            |\n",
    "| Eigenvalues/vectors   | $\\mathbf{Av} = \\lambda\\mathbf{v}$                           | `np.linalg.eig(A)`            |\n",
    "| Inverse               | $\\mathbf{A}^{-1}$                                           | `np.linalg.inv(A)`            |\n",
    "| Identity matrix       | $\\mathbf{I}_n$                                              | `np.eye(n)`                   |\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
