{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cc95c139",
   "metadata": {},
   "source": [
    "# IN400 - Linear Algebra & NumPy Reference Guide\n",
    "\n",
    "**Applied Mathematics & Statistics for Computer Science**\n",
    "\n",
    "A comprehensive guide covering vectors, matrices, and advanced linear algebra concepts with both mathematical theory and practical NumPy implementations. Essential reference for AI and machine learning applications.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b771428",
   "metadata": {},
   "source": [
    "## 📑 Table of Contents\n",
    "\n",
    "1. [Vectors](#1-vectors)\n",
    "   - Vector Representation & Creation\n",
    "   - Vector Operations (Addition, Scalar Multiplication, Dot Product)\n",
    "   - Vector Magnitude & Normalization\n",
    "   - Cross Product & Vector Projection\n",
    "2. [Matrices](#2-matrices)\n",
    "   - Matrix Representation & Creation\n",
    "   - Matrix Operations (Addition, Multiplication, Transpose)\n",
    "   - Determinant, Inverse, and Rank\n",
    "   - Eigenvalues & Eigenvectors\n",
    "3. [Advanced Topics](#3-advanced-topics)\n",
    "   - Principal Component Analysis (PCA)\n",
    "   - Singular Value Decomposition (SVD)\n",
    "4. [Practical Applications](#4-practical-applications)\n",
    "   - Data Representation\n",
    "   - Linear Transformations\n",
    "   - Solving Linear Systems\n",
    "5. [Quick Reference Cheat Sheet](#5-quick-reference-cheat-sheet)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a220250",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NumPy library\n",
    "import numpy as np\n",
    "\n",
    "print(\"NumPy version:\", np.__version__)\n",
    "print(\"Ready to go! 🚀\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68b28056",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# 1. Vectors\n",
    "\n",
    "A **vector** is an ordered collection of numbers representing:\n",
    "\n",
    "- Points in space\n",
    "- Directions and magnitudes\n",
    "- Data features in machine learning\n",
    "- Physical quantities (velocity, force, etc.)\n",
    "\n",
    "## 1.1 Vector Representation\n",
    "\n",
    "### Mathematical Notation\n",
    "\n",
    "**Column vector:**\n",
    "$$\\mathbf{v} = \\begin{bmatrix} v_1 \\\\ v_2 \\\\ \\vdots \\\\ v_n \\end{bmatrix}$$\n",
    "\n",
    "**Row vector:**\n",
    "$$\\mathbf{v}^T = [v_1, v_2, \\ldots, v_n]$$\n",
    "\n",
    "### Example\n",
    "\n",
    "In 2D space: $\\mathbf{v} = \\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}$ represents point $(3, 4)$ or a direction from the origin.\n",
    "\n",
    "### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10612350",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating vectors in NumPy\n",
    "v1 = np.array([3, 4])  # 2D vector\n",
    "v2 = np.array([1, 2, 3])  # 3D vector\n",
    "v3 = np.array([1, 2, 3, 4, 5])  # 5D vector\n",
    "\n",
    "print(\"2D vector:\", v1)\n",
    "print(\"3D vector:\", v2)\n",
    "print(\"5D vector:\", v3)\n",
    "print(\"\\nShape of v2:\", v2.shape)  # (3,) means 1D array with 3 elements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5df788b",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 1.2 Vector Operations\n",
    "\n",
    "### 1.2.1 Vector Addition and Subtraction\n",
    "\n",
    "Vectors can be added or subtracted **element-wise** if they have the same dimension.\n",
    "\n",
    "#### Mathematical Definition\n",
    "\n",
    "$$\\mathbf{a} + \\mathbf{b} = \\begin{bmatrix} a_1 \\\\ a_2 \\\\ \\vdots \\\\ a_n \\end{bmatrix} + \\begin{bmatrix} b_1 \\\\ b_2 \\\\ \\vdots \\\\ b_n \\end{bmatrix} = \\begin{bmatrix} a_1 + b_1 \\\\ a_2 + b_2 \\\\ \\vdots \\\\ a_n + b_n \\end{bmatrix}$$\n",
    "\n",
    "#### Example\n",
    "\n",
    "$$\\begin{bmatrix} 1 \\\\ 2 \\\\ 3 \\end{bmatrix} + \\begin{bmatrix} 4 \\\\ 5 \\\\ 6 \\end{bmatrix} = \\begin{bmatrix} 5 \\\\ 7 \\\\ 9 \\end{bmatrix}$$\n",
    "\n",
    "#### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2accabce",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "\n",
    "addition = a + b\n",
    "subtraction = a - b\n",
    "\n",
    "print(\"a + b =\", addition)  # [5 7 9]\n",
    "print(\"a - b =\", subtraction)  # [-3 -3 -3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35b88f10",
   "metadata": {},
   "source": [
    "#### 1.2.2 Scalar Multiplication\n",
    "\n",
    "Multiplying a vector by a scalar (number) scales each component.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$c \\cdot \\mathbf{v} = c \\cdot \\begin{bmatrix} v_1 \\\\ v_2 \\\\ \\vdots \\\\ v_n \\end{bmatrix} = \\begin{bmatrix} c \\cdot v_1 \\\\ c \\cdot v_2 \\\\ \\vdots \\\\ c \\cdot v_n \\end{bmatrix}$$\n",
    "\n",
    "**Example:** $3 \\cdot \\begin{bmatrix} 1 \\\\ 2 \\end{bmatrix} = \\begin{bmatrix} 3 \\\\ 6 \\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cf907d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([1, 2, 3])\n",
    "scalar = 3\n",
    "\n",
    "result = scalar * v\n",
    "print(\"3 * v =\", result)  # [3 6 9]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69877ff",
   "metadata": {},
   "source": [
    "#### 1.2.3 Dot Product (Inner Product)\n",
    "\n",
    "The dot product combines two vectors into a **single number**. It measures how much two vectors \"align\" with each other.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{a} \\cdot \\mathbf{b} = a_1b_1 + a_2b_2 + \\ldots + a_nb_n = \\sum_{i=1}^{n} a_i b_i$$\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 \\\\ 2 \\\\ 3 \\end{bmatrix} \\cdot \\begin{bmatrix} 4 \\\\ 5 \\\\ 6 \\end{bmatrix} = 1(4) + 2(5) + 3(6) = 4 + 10 + 18 = 32$$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- If $\\mathbf{a} \\cdot \\mathbf{b} = 0$, the vectors are **orthogonal** (perpendicular)\n",
    "- If $\\mathbf{a} \\cdot \\mathbf{b} > 0$, they point in similar directions\n",
    "- If $\\mathbf{a} \\cdot \\mathbf{b} < 0$, they point in opposite directions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "551bfae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "\n",
    "# Method 1: Using np.dot()\n",
    "dot_product = np.dot(a, b)\n",
    "print(\"a · b =\", dot_product)  # 32\n",
    "\n",
    "# Method 2: Using @ operator\n",
    "dot_product = a @ b\n",
    "print(\"a · b =\", dot_product)  # 32\n",
    "\n",
    "# Method 3: Element-wise multiply then sum\n",
    "dot_product = np.sum(a * b)\n",
    "print(\"a · b =\", dot_product)  # 32"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f825cc42",
   "metadata": {},
   "source": [
    "#### 1.2.4 Vector Magnitude (Norm)\n",
    "\n",
    "The **magnitude** (or length) of a vector is the distance from the origin to the point.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\|\\mathbf{v}\\| = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2} = \\sqrt{\\mathbf{v} \\cdot \\mathbf{v}}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\left\\|\\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}\\right\\| = \\sqrt{3^2 + 4^2} = \\sqrt{9 + 16} = \\sqrt{25} = 5$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da5edf9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([3, 4])\n",
    "\n",
    "# Method 1: Using np.linalg.norm()\n",
    "magnitude = np.linalg.norm(v)\n",
    "print(\"||v|| =\", magnitude)  # 5.0\n",
    "\n",
    "# Method 2: Manual calculation\n",
    "magnitude = np.sqrt(np.sum(v**2))\n",
    "print(\"||v|| =\", magnitude)  # 5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17595c68",
   "metadata": {},
   "source": [
    "#### 1.2.5 Unit Vector (Normalization)\n",
    "\n",
    "A **unit vector** has magnitude 1 and indicates direction only.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\hat{\\mathbf{v}} = \\frac{\\mathbf{v}}{\\|\\mathbf{v}\\|}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{v} = \\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}, \\quad \\hat{\\mathbf{v}} = \\frac{1}{5}\\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix} = \\begin{bmatrix} 0.6 \\\\ 0.8 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be9cc94d",
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([3, 4])\n",
    "\n",
    "# Normalize the vector\n",
    "unit_vector = v / np.linalg.norm(v)\n",
    "print(\"Unit vector:\", unit_vector)  # [0.6 0.8]\n",
    "print(\"Magnitude of unit vector:\", np.linalg.norm(unit_vector))  # 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d60a7fb",
   "metadata": {},
   "source": [
    "#### 1.2.6 Cross Product (Vector Product)\n",
    "\n",
    "The **cross product** is an operation on two 3D vectors that produces a third vector **perpendicular** to both input vectors. It's only defined for 3D vectors.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{a} \\times \\mathbf{b} = \\begin{bmatrix} a_2b_3 - a_3b_2 \\\\ a_3b_1 - a_1b_3 \\\\ a_1b_2 - a_2b_1 \\end{bmatrix}$$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- The result is perpendicular to both input vectors\n",
    "- The magnitude is: $\\|\\mathbf{a} \\times \\mathbf{b}\\| = \\|\\mathbf{a}\\| \\|\\mathbf{b}\\| \\sin(\\theta)$, where $\\theta$ is the angle between vectors\n",
    "- **Not commutative**: $\\mathbf{a} \\times \\mathbf{b} = -(\\mathbf{b} \\times \\mathbf{a})$\n",
    "- If vectors are parallel: $\\mathbf{a} \\times \\mathbf{b} = \\mathbf{0}$\n",
    "\n",
    "**Applications:**\n",
    "\n",
    "- Finding normal vectors (computer graphics, physics)\n",
    "- Computing torque and angular momentum\n",
    "- Determining the orientation of surfaces\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 \\\\ 2 \\\\ 3 \\end{bmatrix} \\times \\begin{bmatrix} 4 \\\\ 5 \\\\ 6 \\end{bmatrix} = \\begin{bmatrix} 2(6) - 3(5) \\\\ 3(4) - 1(6) \\\\ 1(5) - 2(4) \\end{bmatrix} = \\begin{bmatrix} -3 \\\\ 6 \\\\ -3 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4348037e",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3])\n",
    "b = np.array([4, 5, 6])\n",
    "\n",
    "# Cross product using np.cross()\n",
    "cross_product = np.cross(a, b)\n",
    "print(\"a × b =\", cross_product)  # [-3  6 -3]\n",
    "\n",
    "# Verify: the result is perpendicular to both a and b\n",
    "# Dot product with perpendicular vectors should be 0\n",
    "print(\"\\n(a × b) · a =\", np.dot(cross_product, a))  # Should be 0\n",
    "print(\"(a × b) · b =\", np.dot(cross_product, b))  # Should be 0\n",
    "\n",
    "# Example: Find a vector perpendicular to the xy-plane\n",
    "x_axis = np.array([1, 0, 0])\n",
    "y_axis = np.array([0, 1, 0])\n",
    "z_direction = np.cross(x_axis, y_axis)\n",
    "print(\"\\nx_axis × y_axis =\", z_direction)  # [0 0 1] - points in z direction\n",
    "\n",
    "# Example: Parallel vectors have zero cross product\n",
    "parallel_a = np.array([2, 4, 6])\n",
    "parallel_b = np.array([1, 2, 3])  # parallel_b = 0.5 * parallel_a\n",
    "result = np.cross(parallel_b, parallel_a)\n",
    "print(\"\\nCross product of parallel vectors:\", result)  # [0 0 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3dc3410",
   "metadata": {},
   "source": [
    "#### 1.2.7 Vector Projection\n",
    "\n",
    "**Vector projection** projects one vector onto another, finding the component of one vector in the direction of another.\n",
    "\n",
    "**Mathematical definition:**\n",
    "\n",
    "Projection of $\\mathbf{a}$ onto $\\mathbf{b}$:\n",
    "$$\\text{proj}_{\\mathbf{b}}\\mathbf{a} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\|\\mathbf{b}\\|^2}\\mathbf{b} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\mathbf{b} \\cdot \\mathbf{b}}\\mathbf{b}$$\n",
    "\n",
    "The **scalar projection** (length of the projection):\n",
    "$$\\text{comp}_{\\mathbf{b}}\\mathbf{a} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\|\\mathbf{b}\\|}$$\n",
    "\n",
    "**Geometric interpretation:**\n",
    "\n",
    "- The projection shows \"how much\" of vector $\\mathbf{a}$ points in the direction of $\\mathbf{b}$\n",
    "- If perpendicular: projection = 0\n",
    "- If parallel: projection = $\\mathbf{a}$ (or $-\\mathbf{a}$)\n",
    "\n",
    "**Applications:**\n",
    "\n",
    "- Linear regression (projecting data onto a line/plane)\n",
    "- Component analysis\n",
    "- Shadow calculations in graphics\n",
    "- Signal decomposition\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{a} = \\begin{bmatrix} 3 \\\\ 4 \\end{bmatrix}, \\quad \\mathbf{b} = \\begin{bmatrix} 1 \\\\ 0 \\end{bmatrix}$$\n",
    "$$\\text{proj}_{\\mathbf{b}}\\mathbf{a} = \\frac{3(1) + 4(0)}{1^2 + 0^2}\\begin{bmatrix} 1 \\\\ 0 \\end{bmatrix} = 3\\begin{bmatrix} 1 \\\\ 0 \\end{bmatrix} = \\begin{bmatrix} 3 \\\\ 0 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b60861ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 1: Project vector a onto vector b\n",
    "a = np.array([3, 4])\n",
    "b = np.array([1, 0])\n",
    "\n",
    "# Calculate projection\n",
    "proj_b_a = (np.dot(a, b) / np.dot(b, b)) * b\n",
    "print(\"Vector a:\", a)\n",
    "print(\"Vector b:\", b)\n",
    "print(\"Projection of a onto b:\", proj_b_a)  # [3 0]\n",
    "\n",
    "# Scalar projection (length)\n",
    "scalar_proj = np.dot(a, b) / np.linalg.norm(b)\n",
    "print(\"Scalar projection (length):\", scalar_proj)  # 3.0\n",
    "\n",
    "# Example 2: 3D projection\n",
    "a_3d = np.array([1, 2, 3])\n",
    "b_3d = np.array([1, 1, 0])\n",
    "\n",
    "proj_3d = (np.dot(a_3d, b_3d) / np.dot(b_3d, b_3d)) * b_3d\n",
    "print(\"\\n3D Example:\")\n",
    "print(\"Vector a:\", a_3d)\n",
    "print(\"Vector b:\", b_3d)\n",
    "print(\"Projection of a onto b:\", proj_3d)\n",
    "\n",
    "# Example 3: Perpendicular vectors (projection = 0)\n",
    "perpendicular_a = np.array([1, 0, 0])\n",
    "perpendicular_b = np.array([0, 1, 0])\n",
    "\n",
    "proj_perp = (\n",
    "    np.dot(perpendicular_a, perpendicular_b) / np.dot(perpendicular_b, perpendicular_b)\n",
    ") * perpendicular_b\n",
    "print(\"\\nPerpendicular vectors:\")\n",
    "print(\"Projection:\", proj_perp)  # [0 0 0]\n",
    "\n",
    "# Example 4: Decompose a vector into parallel and perpendicular components\n",
    "a_decomp = np.array([3, 4])\n",
    "b_decomp = np.array([1, 0])\n",
    "\n",
    "parallel_component = (\n",
    "    np.dot(a_decomp, b_decomp) / np.dot(b_decomp, b_decomp)\n",
    ") * b_decomp\n",
    "perpendicular_component = a_decomp - parallel_component\n",
    "\n",
    "print(\"\\nVector decomposition:\")\n",
    "print(\"Original vector:\", a_decomp)\n",
    "print(\"Parallel component:\", parallel_component)\n",
    "print(\"Perpendicular component:\", perpendicular_component)\n",
    "print(\"Sum (should equal original):\", parallel_component + perpendicular_component)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e8924a1",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Matrices\n",
    "\n",
    "A **matrix** is a 2D array of numbers arranged in rows and columns. Matrices are essential in AI for representing datasets, transformations, and neural network weights.\n",
    "\n",
    "### 2.1 Matrix Representation\n",
    "\n",
    "**Mathematical notation:**\n",
    "$$\\mathbf{A} = \\begin{bmatrix} a_{11} & a_{12} & \\cdots & a_{1n} \\\\ a_{21} & a_{22} & \\cdots & a_{2n} \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ a_{m1} & a_{m2} & \\cdots & a_{mn} \\end{bmatrix}$$\n",
    "\n",
    "- An **m × n matrix** has m rows and n columns\n",
    "- Element at row i, column j is denoted as $a_{ij}$ or $A[i,j]$\n",
    "\n",
    "**Example:** A 2×3 matrix:\n",
    "$$\\mathbf{A} = \\begin{bmatrix} 1 & 2 & 3 \\\\ 4 & 5 & 6 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac6797b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating matrices\n",
    "A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3 matrix\n",
    "\n",
    "B = np.array([[1, 2], [3, 4], [5, 6]])  # 3x2 matrix\n",
    "\n",
    "print(\"Matrix A:\")\n",
    "print(A)\n",
    "print(\"Shape:\", A.shape)  # (2, 3) means 2 rows, 3 columns\n",
    "\n",
    "# Identity matrix (diagonal of 1s)\n",
    "I = np.eye(3)\n",
    "print(\"\\nIdentity matrix:\")\n",
    "print(I)\n",
    "\n",
    "# Zero matrix\n",
    "Z = np.zeros((2, 3))\n",
    "print(\"\\nZero matrix:\")\n",
    "print(Z)\n",
    "\n",
    "# Random matrix\n",
    "R = np.random.rand(2, 3)  # Random values between 0 and 1\n",
    "print(\"\\nRandom matrix:\")\n",
    "print(R)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e84744b6",
   "metadata": {},
   "source": [
    "### 2.2 Matrix Operations\n",
    "\n",
    "#### 2.2.1 Matrix Addition and Subtraction\n",
    "\n",
    "Matrices of the **same dimensions** can be added/subtracted element-wise.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$\\mathbf{A} + \\mathbf{B} = \\begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\\\ a_{21} + b_{21} & a_{22} + b_{22} \\end{bmatrix}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix} + \\begin{bmatrix} 5 & 6 \\\\ 7 & 8 \\end{bmatrix} = \\begin{bmatrix} 6 & 8 \\\\ 10 & 12 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e8b1b9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])\n",
    "B = np.array([[5, 6], [7, 8]])\n",
    "\n",
    "addition = A + B\n",
    "subtraction = A - B\n",
    "\n",
    "print(\"A + B =\")\n",
    "print(addition)\n",
    "# [[6  8]\n",
    "#  [10 12]]\n",
    "\n",
    "print(\"\\nA - B =\")\n",
    "print(subtraction)\n",
    "# [[-4 -4]\n",
    "#  [-4 -4]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74f2fd98",
   "metadata": {},
   "source": [
    "#### 2.2.2 Scalar Multiplication\n",
    "\n",
    "Multiply each element by a scalar.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$c \\cdot \\mathbf{A} = \\begin{bmatrix} c \\cdot a_{11} & c \\cdot a_{12} \\\\ c \\cdot a_{21} & c \\cdot a_{22} \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88b35d7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])\n",
    "result = 3 * A\n",
    "\n",
    "print(\"3 * A =\")\n",
    "print(result)\n",
    "# [[3  6]\n",
    "#  [9 12]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8330412d",
   "metadata": {},
   "source": [
    "#### 2.2.3 Matrix Multiplication\n",
    "\n",
    "**Key rule:** To multiply $\\mathbf{A}$ (m×n) by $\\mathbf{B}$ (p×q), we need **n = p**. The result is an m×q matrix.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$(\\mathbf{AB})_{ij} = \\sum_{k=1}^{n} a_{ik} b_{kj}$$\n",
    "\n",
    "The element at position (i,j) in the result is the dot product of row i of A with column j of B.\n",
    "\n",
    "**Example:**\n",
    "$$\\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix} \\times \\begin{bmatrix} 5 & 6 \\\\ 7 & 8 \\end{bmatrix} = \\begin{bmatrix} 1(5)+2(7) & 1(6)+2(8) \\\\ 3(5)+4(7) & 3(6)+4(8) \\end{bmatrix} = \\begin{bmatrix} 19 & 22 \\\\ 43 & 50 \\end{bmatrix}$$\n",
    "\n",
    "**Important:** Matrix multiplication is **not commutative**: $\\mathbf{AB} \\neq \\mathbf{BA}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908c9e30",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])  # 2x2\n",
    "\n",
    "B = np.array([[5, 6], [7, 8]])  # 2x2\n",
    "\n",
    "# Matrix multiplication\n",
    "C = np.dot(A, B)\n",
    "# Or using @ operator\n",
    "C = A @ B\n",
    "\n",
    "print(\"A × B =\")\n",
    "print(C)\n",
    "# [[19 22]\n",
    "#  [43 50]]\n",
    "\n",
    "# Matrix-vector multiplication\n",
    "v = np.array([1, 2])\n",
    "result = A @ v\n",
    "\n",
    "print(\"\\nA × v =\")\n",
    "print(result)  # [5 11]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2fe021d",
   "metadata": {},
   "source": [
    "#### 2.2.4 Matrix Transpose\n",
    "\n",
    "**Transpose** flips a matrix over its diagonal, converting rows to columns.\n",
    "\n",
    "**Mathematical definition:**\n",
    "$$(\\mathbf{A}^T)_{ij} = A_{ji}$$\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{A} = \\begin{bmatrix} 1 & 2 & 3 \\\\ 4 & 5 & 6 \\end{bmatrix}, \\quad \\mathbf{A}^T = \\begin{bmatrix} 1 & 4 \\\\ 2 & 5 \\\\ 3 & 6 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c70e90f",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3\n",
    "\n",
    "A_transpose = A.T\n",
    "\n",
    "print(\"A =\")\n",
    "print(A)\n",
    "print(\"\\nA^T =\")\n",
    "print(A_transpose)  # 3x2 matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a21e80be",
   "metadata": {},
   "source": [
    "#### 2.2.5 Matrix Inverse\n",
    "\n",
    "The **inverse** of a square matrix $\\mathbf{A}$ is denoted $\\mathbf{A}^{-1}$ and satisfies:\n",
    "$$\\mathbf{A} \\mathbf{A}^{-1} = \\mathbf{A}^{-1} \\mathbf{A} = \\mathbf{I}$$\n",
    "\n",
    "**Important:** Not all matrices have an inverse! A matrix is **invertible** if its determinant is non-zero.\n",
    "\n",
    "**Example:**\n",
    "$$\\mathbf{A} = \\begin{bmatrix} 4 & 7 \\\\ 2 & 6 \\end{bmatrix}, \\quad \\mathbf{A}^{-1} = \\begin{bmatrix} 0.6 & -0.7 \\\\ -0.2 & 0.4 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2afa62f",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[4, 7], [2, 6]])\n",
    "\n",
    "# Calculate inverse\n",
    "A_inv = np.linalg.inv(A)\n",
    "\n",
    "print(\"A =\")\n",
    "print(A)\n",
    "print(\"\\nA^(-1) =\")\n",
    "print(A_inv)\n",
    "\n",
    "# Verify: A × A^(-1) = I\n",
    "identity = A @ A_inv\n",
    "print(\"\\nA × A^(-1) =\")\n",
    "print(identity)  # Should be close to [[1, 0], [0, 1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc6a2c05",
   "metadata": {},
   "source": [
    "#### 2.2.6 Matrix Determinant\n",
    "\n",
    "The **determinant** is a scalar value that can be computed from a square matrix. It provides important information about the matrix:\n",
    "\n",
    "- If det(A) ≠ 0, the matrix is **invertible**\n",
    "- If det(A) = 0, the matrix is **singular** (not invertible)\n",
    "- The determinant represents the volume scaling factor of the linear transformation\n",
    "\n",
    "**Mathematical definition:**\n",
    "\n",
    "For a 2×2 matrix:\n",
    "$$\\text{det}\\begin{bmatrix} a & b \\\\ c & d \\end{bmatrix} = ad - bc$$\n",
    "\n",
    "For a 3×3 matrix:\n",
    "$$\\text{det}\\begin{bmatrix} a & b & c \\\\ d & e & f \\\\ g & h & i \\end{bmatrix} = a(ei - fh) - b(di - fg) + c(dh - eg)$$\n",
    "\n",
    "**Examples:**\n",
    "\n",
    "2×2: $\\text{det}\\begin{bmatrix} 3 & 8 \\\\ 4 & 6 \\end{bmatrix} = 3(6) - 8(4) = 18 - 32 = -14$\n",
    "\n",
    "3×3: $\\text{det}\\begin{bmatrix} 1 & 2 & 3 \\\\ 0 & 1 & 4 \\\\ 5 & 6 & 0 \\end{bmatrix} = 1(1 \\cdot 0 - 4 \\cdot 6) - 2(0 \\cdot 0 - 4 \\cdot 5) + 3(0 \\cdot 6 - 1 \\cdot 5) = 1(-24) - 2(-20) + 3(-5) = -24 + 40 - 15 = 1$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa8963de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 1: Determinant of a 2x2 matrix\n",
    "A_2x2 = np.array([[3, 8], [4, 6]])\n",
    "\n",
    "det_A = np.linalg.det(A_2x2)\n",
    "print(\"Matrix A (2x2):\")\n",
    "print(A_2x2)\n",
    "print(\"Determinant of A:\", det_A)  # -14.0\n",
    "\n",
    "# Example 2: Determinant of a 3x3 matrix\n",
    "A_3x3 = np.array([[1, 2, 3], [0, 1, 4], [5, 6, 0]])\n",
    "\n",
    "det_B = np.linalg.det(A_3x3)\n",
    "print(\"\\nMatrix B (3x3):\")\n",
    "print(A_3x3)\n",
    "print(\"Determinant of B:\", det_B)  # 1.0\n",
    "\n",
    "# Example 3: Check if a matrix is invertible\n",
    "singular_matrix = np.array([[2, 4], [1, 2]])\n",
    "\n",
    "det_singular = np.linalg.det(singular_matrix)\n",
    "print(\"\\nSingular Matrix:\")\n",
    "print(singular_matrix)\n",
    "print(\"Determinant:\", det_singular)  # 0.0 (or very close to 0)\n",
    "\n",
    "if abs(det_singular) < 1e-10:\n",
    "    print(\"This matrix is SINGULAR (not invertible)\")\n",
    "else:\n",
    "    print(\"This matrix is INVERTIBLE\")\n",
    "\n",
    "# Example 4: Manual calculation for 2x2 (to verify)\n",
    "a, b = 3, 8\n",
    "c, d = 4, 6\n",
    "manual_det = a * d - b * c\n",
    "print(f\"\\nManual calculation: {a}*{d} - {b}*{c} = {manual_det}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a7cf0fa",
   "metadata": {},
   "source": [
    "#### 2.2.7 Eigenvalues and Eigenvectors\n",
    "\n",
    "**Eigenvalues** and **eigenvectors** are fundamental concepts in linear algebra. For a square matrix $\\mathbf{A}$, an eigenvector $\\mathbf{v}$ and its corresponding eigenvalue $\\lambda$ satisfy:\n",
    "\n",
    "$$\\mathbf{A}\\mathbf{v} = \\lambda\\mathbf{v}$$\n",
    "\n",
    "**Interpretation:**\n",
    "\n",
    "- When matrix $\\mathbf{A}$ is applied to eigenvector $\\mathbf{v}$, the result is the same vector scaled by $\\lambda$\n",
    "- The eigenvector's **direction** is preserved, only the **magnitude** changes\n",
    "- $\\lambda$ (eigenvalue) is the scaling factor\n",
    "\n",
    "**Finding eigenvalues:**\n",
    "Solve the characteristic equation:\n",
    "$$\\text{det}(\\mathbf{A} - \\lambda\\mathbf{I}) = 0$$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- An n×n matrix has n eigenvalues (counting multiplicities)\n",
    "- Eigenvectors corresponding to different eigenvalues are linearly independent\n",
    "- For symmetric matrices, eigenvalues are real and eigenvectors are orthogonal\n",
    "\n",
    "**Applications in AI/ML:**\n",
    "\n",
    "- **Principal Component Analysis (PCA)**: Dimensionality reduction\n",
    "- **Google PageRank**: Website ranking algorithm\n",
    "- **Markov Chains**: State transitions and steady states\n",
    "- **Image Compression**: Spectral decomposition\n",
    "- **Stability Analysis**: Neural network dynamics\n",
    "- **Graph Analysis**: Community detection\n",
    "\n",
    "**Example:**\n",
    "For matrix $\\mathbf{A} = \\begin{bmatrix} 4 & 2 \\\\ 1 & 3 \\end{bmatrix}$:\n",
    "\n",
    "- Eigenvalues: $\\lambda_1 = 5, \\lambda_2 = 2$\n",
    "- Eigenvector for $\\lambda_1 = 5$: $\\mathbf{v}_1 = \\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$\n",
    "- Verification: $\\begin{bmatrix} 4 & 2 \\\\ 1 & 3 \\end{bmatrix}\\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix} = \\begin{bmatrix} 10 \\\\ 5 \\end{bmatrix} = 5\\begin{bmatrix} 2 \\\\ 1 \\end{bmatrix}$ ✓\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b489630",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example 1: Basic eigenvalue and eigenvector calculation\n",
    "A = np.array([[4, 2], [1, 3]])\n",
    "\n",
    "# Calculate eigenvalues and eigenvectors\n",
    "eigenvalues, eigenvectors = np.linalg.eig(A)\n",
    "\n",
    "print(\"Matrix A:\")\n",
    "print(A)\n",
    "print(\"\\nEigenvalues:\", eigenvalues)\n",
    "print(\"\\nEigenvectors:\")\n",
    "print(eigenvectors)\n",
    "\n",
    "# Verify: A @ v = λ * v for the first eigenvector\n",
    "v1 = eigenvectors[:, 0]  # First eigenvector (column 0)\n",
    "lambda1 = eigenvalues[0]\n",
    "\n",
    "left_side = A @ v1\n",
    "right_side = lambda1 * v1\n",
    "\n",
    "print(f\"\\nVerification for λ₁ = {lambda1:.4f}:\")\n",
    "print(f\"A @ v₁ = {left_side}\")\n",
    "print(f\"λ₁ * v₁ = {right_side}\")\n",
    "print(f\"Equal? {np.allclose(left_side, right_side)}\")\n",
    "\n",
    "# Example 2: Symmetric matrix (real eigenvalues, orthogonal eigenvectors)\n",
    "symmetric_matrix = np.array([[2, 1], [1, 2]])\n",
    "\n",
    "eig_vals, eig_vecs = np.linalg.eig(symmetric_matrix)\n",
    "\n",
    "print(\"\\n\\nSymmetric Matrix:\")\n",
    "print(symmetric_matrix)\n",
    "print(\"\\nEigenvalues:\", eig_vals)\n",
    "print(\"\\nEigenvectors:\")\n",
    "print(eig_vecs)\n",
    "\n",
    "# Check orthogonality: dot product should be 0\n",
    "dot_product = np.dot(eig_vecs[:, 0], eig_vecs[:, 1])\n",
    "print(f\"\\nDot product of eigenvectors: {dot_product:.10f} (should be ~0)\")\n",
    "\n",
    "# Example 3: 3x3 matrix\n",
    "A_3x3 = np.array([[6, -1, 0], [-1, 6, -1], [0, -1, 6]])\n",
    "\n",
    "eig_vals_3d, eig_vecs_3d = np.linalg.eig(A_3x3)\n",
    "\n",
    "print(\"\\n\\n3x3 Matrix:\")\n",
    "print(A_3x3)\n",
    "print(\"\\nEigenvalues:\", eig_vals_3d)\n",
    "\n",
    "\n",
    "# Example 4: Application - Power iteration to find dominant eigenvalue\n",
    "def power_iteration(A, num_iterations=10):\n",
    "    \"\"\"Find the dominant eigenvalue and eigenvector using power iteration\"\"\"\n",
    "    n = A.shape[0]\n",
    "    v = np.random.rand(n)  # Random initial vector\n",
    "\n",
    "    for i in range(num_iterations):\n",
    "        # Multiply by matrix\n",
    "        v = A @ v\n",
    "        # Normalize\n",
    "        v = v / np.linalg.norm(v)\n",
    "\n",
    "    # Calculate eigenvalue: λ = (v^T A v) / (v^T v)\n",
    "    eigenvalue = (v.T @ A @ v) / (v.T @ v)\n",
    "    return eigenvalue, v\n",
    "\n",
    "\n",
    "dominant_eval, dominant_evec = power_iteration(A, num_iterations=20)\n",
    "print(f\"\\n\\nPower Iteration (finds dominant eigenvalue):\")\n",
    "print(f\"Dominant eigenvalue: {dominant_eval:.4f}\")\n",
    "print(f\"Compare with np.linalg.eig: {max(eigenvalues):.4f}\")\n",
    "\n",
    "# Example 5: Trace and determinant relation to eigenvalues\n",
    "print(f\"\\n\\nTrace-Eigenvalue relation:\")\n",
    "print(f\"Trace of A: {np.trace(A):.4f}\")\n",
    "print(f\"Sum of eigenvalues: {np.sum(eigenvalues):.4f}\")\n",
    "print(f\"Determinant of A: {np.linalg.det(A):.4f}\")\n",
    "print(f\"Product of eigenvalues: {np.prod(eigenvalues):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2dbafc9",
   "metadata": {},
   "source": [
    "#### 2.2.8 Element-wise Operations\n",
    "\n",
    "Sometimes we need element-wise operations (Hadamard product), not matrix multiplication.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d29f2a40",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [3, 4]])\n",
    "B = np.array([[5, 6], [7, 8]])\n",
    "\n",
    "# Element-wise multiplication\n",
    "element_wise = A * B\n",
    "print(\"A ⊙ B (element-wise) =\")\n",
    "print(element_wise)\n",
    "# [[5  12]\n",
    "#  [21 32]]\n",
    "\n",
    "# Element-wise division\n",
    "element_div = A / B\n",
    "print(\"\\nA ⊘ B (element-wise) =\")\n",
    "print(element_div)\n",
    "\n",
    "# Element-wise power\n",
    "powered = A**2\n",
    "print(\"\\nA^2 (element-wise) =\")\n",
    "print(powered)\n",
    "# [[1  4]\n",
    "#  [9 16]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ac85475",
   "metadata": {},
   "source": [
    "### 2.3 Special Matrices\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22ef6746",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identity matrix (3x3)\n",
    "I = np.eye(3)\n",
    "print(\"Identity matrix:\")\n",
    "print(I)\n",
    "\n",
    "# Diagonal matrix\n",
    "diag = np.diag([1, 2, 3])\n",
    "print(\"\\nDiagonal matrix:\")\n",
    "print(diag)\n",
    "\n",
    "# Upper triangular\n",
    "upper = np.triu(np.ones((3, 3)))\n",
    "print(\"\\nUpper triangular:\")\n",
    "print(upper)\n",
    "\n",
    "# Lower triangular\n",
    "lower = np.tril(np.ones((3, 3)))\n",
    "print(\"\\nLower triangular:\")\n",
    "print(lower)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ce86364",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Practical Applications in AI\n",
    "\n",
    "### 3.1 Data Representation\n",
    "\n",
    "- **Vectors**: Each data point (e.g., image, text) is represented as a vector\n",
    "- **Matrices**: A dataset is a matrix where each row is a data sample\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b606d59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataset: 5 students with 3 test scores each\n",
    "grades = np.array(\n",
    "    [\n",
    "        [85, 90, 88],  # Student 1\n",
    "        [78, 85, 80],  # Student 2\n",
    "        [92, 95, 93],  # Student 3\n",
    "        [70, 75, 72],  # Student 4\n",
    "        [88, 87, 90],\n",
    "    ]\n",
    ")  # Student 5\n",
    "\n",
    "print(\"Dataset shape:\", grades.shape)  # (5, 3)\n",
    "\n",
    "# Calculate mean score for each student\n",
    "mean_per_student = np.mean(grades, axis=1)\n",
    "print(\"Mean per student:\", mean_per_student)\n",
    "\n",
    "# Calculate mean score for each test\n",
    "mean_per_test = np.mean(grades, axis=0)\n",
    "print(\"Mean per test:\", mean_per_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbc568a1",
   "metadata": {},
   "source": [
    "### 3.2 Linear Transformations\n",
    "\n",
    "Matrix multiplication applies transformations to vectors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df250949",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rotation matrix (90 degrees counterclockwise)\n",
    "theta = np.pi / 2  # 90 degrees in radians\n",
    "rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])\n",
    "\n",
    "# Original point\n",
    "point = np.array([1, 0])\n",
    "\n",
    "# Rotate the point\n",
    "rotated_point = rotation @ point\n",
    "print(\"Original point:\", point)\n",
    "print(\"Rotated point:\", rotated_point)  # Should be close to [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd386e59",
   "metadata": {},
   "source": [
    "### 3.3 Solving Linear Systems\n",
    "\n",
    "Systems of linear equations can be solved using matrices.\n",
    "\n",
    "**System:**\n",
    "$$2x + 3y = 8$$\n",
    "$$4x + y = 10$$\n",
    "\n",
    "**Matrix form:** $\\mathbf{Ax} = \\mathbf{b}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a3e78bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coefficient matrix\n",
    "A = np.array([[2, 3], [4, 1]])\n",
    "\n",
    "# Constants vector\n",
    "b = np.array([8, 10])\n",
    "\n",
    "# Solve for x\n",
    "x = np.linalg.solve(A, b)\n",
    "print(\"Solution:\", x)  # x = [2, 2] means x=2, y=2\n",
    "\n",
    "# Verify solution\n",
    "print(\"Verification A×x =\", A @ x)  # Should equal b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad865468",
   "metadata": {},
   "source": [
    "#### 2.2.9 Matrix Rank\n",
    "\n",
    "The **rank** of a matrix is the maximum number of linearly independent rows or columns.\n",
    "\n",
    "**Mathematical Definition:**\n",
    "\n",
    "- Rank indicates the dimension of the vector space spanned by the matrix's rows/columns\n",
    "- For an m×n matrix: $0 \\leq \\text{rank}(\\mathbf{A}) \\leq \\min(m, n)$\n",
    "\n",
    "**Properties:**\n",
    "\n",
    "- Full rank: rank equals the smaller dimension\n",
    "- Rank-deficient: rank is less than the smaller dimension\n",
    "- $\\text{rank}(\\mathbf{A}) = \\text{rank}(\\mathbf{A}^T)$\n",
    "\n",
    "#### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1464d472",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example matrices with different ranks\n",
    "A_full_rank = np.array([[1, 2], [3, 4]])\n",
    "A_rank_deficient = np.array([[1, 2], [2, 4]])  # Second row is 2x first row\n",
    "\n",
    "print(\"Full rank matrix:\")\n",
    "print(A_full_rank)\n",
    "print(\"Rank:\", np.linalg.matrix_rank(A_full_rank))  # 2\n",
    "\n",
    "print(\"\\nRank-deficient matrix:\")\n",
    "print(A_rank_deficient)\n",
    "print(\"Rank:\", np.linalg.matrix_rank(A_rank_deficient))  # 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29b96523",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# 3. Advanced Topics\n",
    "\n",
    "## 3.1 Principal Component Analysis (PCA)\n",
    "\n",
    "**PCA** is a dimensionality reduction technique that transforms data into a new coordinate system where the greatest variances lie on the first coordinates (principal components).\n",
    "\n",
    "### Mathematical Foundation\n",
    "\n",
    "Given a dataset $\\mathbf{X}$ (n samples × m features):\n",
    "\n",
    "**Step 1: Center the data**\n",
    "$$\\mathbf{X}_{\\text{centered}} = \\mathbf{X} - \\boldsymbol{\\mu}$$\n",
    "where $\\boldsymbol{\\mu}$ is the mean of each feature.\n",
    "\n",
    "**Step 2: Compute covariance matrix**\n",
    "$$\\mathbf{C} = \\frac{1}{n-1}\\mathbf{X}_{\\text{centered}}^T \\mathbf{X}_{\\text{centered}}$$\n",
    "\n",
    "**Step 3: Find eigenvectors and eigenvalues**\n",
    "Solve: $\\mathbf{C}\\mathbf{v} = \\lambda\\mathbf{v}$\n",
    "\n",
    "**Step 4: Sort by eigenvalues**\n",
    "\n",
    "- Larger eigenvalues → more variance captured\n",
    "- Eigenvectors become principal components\n",
    "\n",
    "**Step 5: Project data**\n",
    "$$\\mathbf{X}_{\\text{PCA}} = \\mathbf{X}_{\\text{centered}} \\mathbf{V}_k$$\n",
    "where $\\mathbf{V}_k$ contains the top k eigenvectors.\n",
    "\n",
    "### Applications in ML/AI\n",
    "\n",
    "- **Dimensionality reduction** — Reduce features while preserving variance\n",
    "- **Data visualization** — Project high-D data to 2D/3D\n",
    "- **Noise reduction** — Remove low-variance components\n",
    "- **Feature extraction** — Create uncorrelated features\n",
    "- **Compression** — Store data more efficiently\n",
    "\n",
    "### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce924c8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate sample data: 100 samples, 3 features\n",
    "np.random.seed(42)\n",
    "X = np.random.randn(100, 3)\n",
    "X[:, 1] = X[:, 0] * 2 + np.random.randn(100) * 0.5  # Correlated features\n",
    "\n",
    "print(\"Original data shape:\", X.shape)\n",
    "print(\"First 5 samples:\")\n",
    "print(X[:5])\n",
    "\n",
    "# Step 1: Center the data (subtract mean)\n",
    "X_mean = np.mean(X, axis=0)\n",
    "X_centered = X - X_mean\n",
    "\n",
    "print(\"\\nData mean:\", X_mean)\n",
    "print(\"Centered data mean (should be ~0):\", np.mean(X_centered, axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1473d539",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2: Compute covariance matrix\n",
    "cov_matrix = np.cov(X_centered.T)  # Note: transpose to get feature covariance\n",
    "\n",
    "print(\"Covariance matrix shape:\", cov_matrix.shape)\n",
    "print(\"\\nCovariance matrix:\")\n",
    "print(cov_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19f452b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3: Compute eigenvalues and eigenvectors\n",
    "eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)\n",
    "\n",
    "print(\"Eigenvalues:\", eigenvalues)\n",
    "print(\"\\nEigenvectors (principal components):\")\n",
    "print(eigenvectors)\n",
    "\n",
    "# Step 4: Sort by eigenvalues (descending order)\n",
    "idx = eigenvalues.argsort()[::-1]  # Sort indices in descending order\n",
    "eigenvalues_sorted = eigenvalues[idx]\n",
    "eigenvectors_sorted = eigenvectors[:, idx]\n",
    "\n",
    "print(\"\\nSorted eigenvalues:\", eigenvalues_sorted)\n",
    "\n",
    "# Calculate explained variance ratio\n",
    "explained_variance_ratio = eigenvalues_sorted / np.sum(eigenvalues_sorted)\n",
    "print(\"\\nExplained variance ratio:\", explained_variance_ratio)\n",
    "print(\"Cumulative variance:\", np.cumsum(explained_variance_ratio))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a09b6c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5: Project data onto principal components\n",
    "# Let's reduce from 3D to 2D (keep first 2 components)\n",
    "k = 2\n",
    "principal_components = eigenvectors_sorted[:, :k]\n",
    "\n",
    "X_pca = X_centered @ principal_components\n",
    "\n",
    "print(f\"Reduced data shape: {X_pca.shape}\")\n",
    "print(\"\\nFirst 5 samples in PCA space:\")\n",
    "print(X_pca[:5])\n",
    "\n",
    "print(f\"\\nVariance preserved: {np.sum(explained_variance_ratio[:k]):.2%}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01d6adfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Complete PCA function\n",
    "def pca_transform(X, n_components):\n",
    "    \"\"\"\n",
    "    Perform PCA dimensionality reduction\n",
    "\n",
    "    Parameters:\n",
    "    X: (n_samples, n_features) data matrix\n",
    "    n_components: number of principal components to keep\n",
    "\n",
    "    Returns:\n",
    "    X_pca: transformed data\n",
    "    components: principal components (eigenvectors)\n",
    "    explained_var: variance explained by each component\n",
    "    \"\"\"\n",
    "    # Center data\n",
    "    X_mean = np.mean(X, axis=0)\n",
    "    X_centered = X - X_mean\n",
    "\n",
    "    # Covariance matrix\n",
    "    cov_matrix = np.cov(X_centered.T)\n",
    "\n",
    "    # Eigendecomposition\n",
    "    eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)\n",
    "\n",
    "    # Sort by eigenvalues\n",
    "    idx = eigenvalues.argsort()[::-1]\n",
    "    eigenvalues = eigenvalues[idx]\n",
    "    eigenvectors = eigenvectors[:, idx]\n",
    "\n",
    "    # Select top k components\n",
    "    components = eigenvectors[:, :n_components]\n",
    "\n",
    "    # Project data\n",
    "    X_pca = X_centered @ components\n",
    "\n",
    "    # Explained variance\n",
    "    explained_var = eigenvalues[:n_components] / np.sum(eigenvalues)\n",
    "\n",
    "    return X_pca, components, explained_var\n",
    "\n",
    "\n",
    "# Test the function\n",
    "X_pca, components, explained_var = pca_transform(X, n_components=2)\n",
    "print(\"PCA result shape:\", X_pca.shape)\n",
    "print(\"Explained variance:\", explained_var)\n",
    "print(f\"Total variance explained: {np.sum(explained_var):.2%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3c01ea3",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3.2 Singular Value Decomposition (SVD)\n",
    "\n",
    "**SVD** decomposes any matrix into three simpler matrices, revealing its fundamental structure.\n",
    "\n",
    "### Mathematical Definition\n",
    "\n",
    "For any matrix $\\mathbf{A}$ (m×n):\n",
    "$$\\mathbf{A} = \\mathbf{U} \\boldsymbol{\\Sigma} \\mathbf{V}^T$$\n",
    "\n",
    "Where:\n",
    "\n",
    "- $\\mathbf{U}$ (m×m): Left singular vectors (orthonormal)\n",
    "- $\\boldsymbol{\\Sigma}$ (m×n): Diagonal matrix of singular values $\\sigma_1 \\geq \\sigma_2 \\geq \\ldots \\geq 0$\n",
    "- $\\mathbf{V}^T$ (n×n): Right singular vectors (orthonormal)\n",
    "\n",
    "### Key Properties\n",
    "\n",
    "**Relationship to eigendecomposition:**\n",
    "\n",
    "- Singular values of $\\mathbf{A}$ = square roots of eigenvalues of $\\mathbf{A}^T\\mathbf{A}$\n",
    "- $\\mathbf{V}$ contains eigenvectors of $\\mathbf{A}^T\\mathbf{A}$\n",
    "- $\\mathbf{U}$ contains eigenvectors of $\\mathbf{AA}^T$\n",
    "\n",
    "**Important facts:**\n",
    "\n",
    "- Every matrix has an SVD (even non-square!)\n",
    "- Singular values are always non-negative\n",
    "- Rank of $\\mathbf{A}$ = number of non-zero singular values\n",
    "\n",
    "### Low-Rank Approximation\n",
    "\n",
    "Best rank-k approximation of $\\mathbf{A}$:\n",
    "$$\\mathbf{A}_k = \\mathbf{U}_k \\boldsymbol{\\Sigma}_k \\mathbf{V}_k^T$$\n",
    "\n",
    "Where we keep only the k largest singular values.\n",
    "\n",
    "### Applications in ML/AI\n",
    "\n",
    "- **Image compression** — Store images with fewer bytes\n",
    "- **Recommender systems** — Matrix factorization (Netflix, Amazon)\n",
    "- **Latent Semantic Analysis** — Text mining and NLP\n",
    "- **Noise reduction** — Remove small singular values\n",
    "- **Data visualization** — Similar to PCA\n",
    "- **Pseudoinverse** — Solve ill-conditioned linear systems\n",
    "\n",
    "### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c6ed858",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a sample matrix\n",
    "A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])\n",
    "\n",
    "print(\"Original matrix A:\")\n",
    "print(A)\n",
    "print(\"Shape:\", A.shape)  # 4×3\n",
    "\n",
    "# Perform SVD\n",
    "U, S, Vt = np.linalg.svd(A, full_matrices=True)\n",
    "\n",
    "print(\"\\nU (left singular vectors):\")\n",
    "print(U)\n",
    "print(\"Shape:\", U.shape)  # 4×4\n",
    "\n",
    "print(\"\\nSingular values:\")\n",
    "print(S)\n",
    "print(\"Shape:\", S.shape)  # (3,) - returned as 1D array\n",
    "\n",
    "print(\"\\nV^T (right singular vectors transposed):\")\n",
    "print(Vt)\n",
    "print(\"Shape:\", Vt.shape)  # 3×3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c78d531e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reconstruct original matrix\n",
    "# Need to create full Sigma matrix from singular values\n",
    "Sigma = np.zeros((U.shape[0], Vt.shape[0]))\n",
    "Sigma[: S.shape[0], : S.shape[0]] = np.diag(S)\n",
    "\n",
    "print(\"Sigma (diagonal matrix):\")\n",
    "print(Sigma)\n",
    "print(\"Shape:\", Sigma.shape)\n",
    "\n",
    "# Reconstruct A\n",
    "A_reconstructed = U @ Sigma @ Vt\n",
    "\n",
    "print(\"\\nReconstructed A:\")\n",
    "print(A_reconstructed)\n",
    "\n",
    "print(\"\\nReconstruction error:\")\n",
    "print(np.linalg.norm(A - A_reconstructed))  # Should be very small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81346dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Low-rank approximation\n",
    "k = 2  # Keep top 2 singular values\n",
    "\n",
    "U_k = U[:, :k]\n",
    "S_k = S[:k]\n",
    "Vt_k = Vt[:k, :]\n",
    "\n",
    "# Create k×k diagonal matrix\n",
    "Sigma_k = np.diag(S_k)\n",
    "\n",
    "# Low-rank approximation\n",
    "A_k = U_k @ Sigma_k @ Vt_k\n",
    "\n",
    "print(f\"Rank-{k} approximation:\")\n",
    "print(A_k)\n",
    "\n",
    "print(f\"\\nOriginal matrix:\")\n",
    "print(A)\n",
    "\n",
    "print(f\"\\nApproximation error:\")\n",
    "error = np.linalg.norm(A - A_k)\n",
    "print(f\"Frobenius norm: {error:.4f}\")\n",
    "\n",
    "# Compression ratio\n",
    "original_elements = A.shape[0] * A.shape[1]\n",
    "compressed_elements = k * (U_k.shape[0] + Vt_k.shape[1] + 1)\n",
    "print(\n",
    "    f\"\\nCompression ratio: {compressed_elements}/{original_elements} = {compressed_elements/original_elements:.2%}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bba4155",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: Image compression using SVD\n",
    "# Create a simple \"image\" (grayscale matrix)\n",
    "np.random.seed(42)\n",
    "image = np.random.rand(50, 50)\n",
    "\n",
    "# Add some structure (diagonal pattern)\n",
    "for i in range(50):\n",
    "    image[i, i] = 1.0\n",
    "    if i < 49:\n",
    "        image[i, i + 1] = 0.8\n",
    "        image[i + 1, i] = 0.8\n",
    "\n",
    "print(\"Original image shape:\", image.shape)\n",
    "\n",
    "# Perform SVD\n",
    "U, S, Vt = np.linalg.svd(image, full_matrices=False)\n",
    "\n",
    "# Try different compression levels\n",
    "for k in [5, 10, 20]:\n",
    "    # Reconstruct with k components\n",
    "    image_k = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]\n",
    "\n",
    "    # Calculate error and compression\n",
    "    error = np.linalg.norm(image - image_k)\n",
    "    compression = (k * (image.shape[0] + image.shape[1])) / (\n",
    "        image.shape[0] * image.shape[1]\n",
    "    )\n",
    "\n",
    "    print(f\"\\nk={k} singular values:\")\n",
    "    print(f\"  Reconstruction error: {error:.4f}\")\n",
    "    print(f\"  Storage needed: {compression:.2%} of original\")\n",
    "    print(f\"  Variance explained: {np.sum(S[:k]**2) / np.sum(S**2):.2%}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7877eef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SVD vs PCA relationship\n",
    "# They're closely related! For centered data:\n",
    "\n",
    "# Generate centered data\n",
    "np.random.seed(42)\n",
    "X_data = np.random.randn(100, 5)\n",
    "X_centered = X_data - np.mean(X_data, axis=0)\n",
    "\n",
    "# Method 1: PCA via eigendecomposition\n",
    "cov_matrix = np.cov(X_centered.T)\n",
    "eigenvalues_pca, eigenvectors_pca = np.linalg.eig(cov_matrix)\n",
    "idx = eigenvalues_pca.argsort()[::-1]\n",
    "eigenvalues_pca = eigenvalues_pca[idx]\n",
    "\n",
    "# Method 2: PCA via SVD\n",
    "U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)\n",
    "eigenvalues_svd = (S**2) / (X_centered.shape[0] - 1)\n",
    "\n",
    "print(\"Eigenvalues from PCA:\", eigenvalues_pca)\n",
    "print(\"Eigenvalues from SVD:\", eigenvalues_svd)\n",
    "print(\"\\nAre they equal?\", np.allclose(eigenvalues_pca, eigenvalues_svd))\n",
    "\n",
    "# The principal components are the same (up to sign)\n",
    "print(\"\\nV from SVD matches eigenvectors from covariance matrix:\")\n",
    "print(\"Close match?\", np.allclose(np.abs(Vt.T), np.abs(eigenvectors_pca[:, idx])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "822efe52",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# 4. Practical Applications in AI\n",
    "\n",
    "## 4.1 Data Representation\n",
    "\n",
    "Vectors and matrices are fundamental for representing data in machine learning:\n",
    "\n",
    "- **Vectors** — Each data point (image pixels, word embeddings, sensor readings)\n",
    "- **Matrices** — Datasets where rows = samples, columns = features\n",
    "- **Tensors** — Higher-dimensional arrays (images, video, time series)\n",
    "\n",
    "### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1faa82a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataset: 5 students with 3 test scores each\n",
    "grades = np.array(\n",
    "    [\n",
    "        [85, 90, 88],  # Student 1\n",
    "        [78, 85, 80],  # Student 2\n",
    "        [92, 95, 93],  # Student 3\n",
    "        [70, 75, 72],  # Student 4\n",
    "        [88, 87, 90],\n",
    "    ]\n",
    ")  # Student 5\n",
    "\n",
    "print(\"Dataset shape:\", grades.shape)  # (5, 3)\n",
    "\n",
    "# Calculate mean score for each student\n",
    "mean_per_student = np.mean(grades, axis=1)\n",
    "print(\"Mean per student:\", mean_per_student)\n",
    "\n",
    "# Calculate mean score for each test\n",
    "mean_per_test = np.mean(grades, axis=0)\n",
    "print(\"Mean per test:\", mean_per_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fd1e5a4",
   "metadata": {},
   "source": [
    "## 4.2 Linear Transformations\n",
    "\n",
    "Matrix multiplication applies geometric transformations to vectors:\n",
    "\n",
    "- **Rotation** — Rotate points/vectors\n",
    "- **Scaling** — Stretch/compress along axes\n",
    "- **Reflection** — Mirror across axes\n",
    "- **Shear** — Skew shapes\n",
    "\n",
    "### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb4d9711",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rotation matrix (90 degrees counterclockwise)\n",
    "theta = np.pi / 2  # 90 degrees in radians\n",
    "rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])\n",
    "\n",
    "# Original point\n",
    "point = np.array([1, 0])\n",
    "\n",
    "# Rotate the point\n",
    "rotated_point = rotation @ point\n",
    "print(\"Original point:\", point)\n",
    "print(\"Rotated point:\", rotated_point)  # Should be close to [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c00ad2d",
   "metadata": {},
   "source": [
    "## 4.3 Solving Linear Systems\n",
    "\n",
    "Systems of linear equations appear everywhere in ML/AI:\n",
    "\n",
    "- **Linear regression** — Fit line/plane to data\n",
    "- **Neural networks** — Each layer is a linear transformation\n",
    "- **Optimization** — Find parameters that minimize loss\n",
    "\n",
    "### Mathematical Form\n",
    "\n",
    "System of equations:\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "2x + 3y = 8 \\\\\n",
    "4x + y = 10\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Matrix form: $\\mathbf{Ax} = \\mathbf{b}$\n",
    "\n",
    "$$\\begin{bmatrix} 2 & 3 \\\\ 4 & 1 \\end{bmatrix} \\begin{bmatrix} x \\\\ y \\end{bmatrix} = \\begin{bmatrix} 8 \\\\ 10 \\end{bmatrix}$$\n",
    "\n",
    "### NumPy Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ce23818",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coefficient matrix\n",
    "A = np.array([[2, 3], [4, 1]])\n",
    "\n",
    "# Constants vector\n",
    "b = np.array([8, 10])\n",
    "\n",
    "# Solve for x\n",
    "x = np.linalg.solve(A, b)\n",
    "print(\"Solution:\", x)  # x = [2, 2] means x=2, y=2\n",
    "\n",
    "# Verify solution\n",
    "print(\"Verification A×x =\", A @ x)  # Should equal b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6760403f",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# 5. Quick Reference Cheat Sheet\n",
    "\n",
    "## Vector Operations\n",
    "\n",
    "| Operation             | Math Notation                            | NumPy Code                    | Note (Manual Calculation) |\n",
    "| --------------------- | ---------------------------------------- | ----------------------------- | ------------------------- |\n",
    "| Create vector         | $\\mathbf{v} = [1, 2, 3]$                 | `v = np.array([1, 2, 3])`     | List of numbers |\n",
    "| Vector addition       | $\\mathbf{a} + \\mathbf{b}$                | `a + b`                       | Add each component: $[a_1+b_1, a_2+b_2, \\ldots]$ |\n",
    "| Scalar multiplication | $c \\cdot \\mathbf{v}$                     | `c * v`                       | Multiply each element: $[c \\cdot v_1, c \\cdot v_2, \\ldots]$ |\n",
    "| Dot product           | $\\mathbf{a} \\cdot \\mathbf{b}$            | `np.dot(a, b)` or `a @ b`     | $\\sum_{i=1}^{n} a_i b_i$ or `np.sum(a * b)` |\n",
    "| Cross product (3D)    | $\\mathbf{a} \\times \\mathbf{b}$           | `np.cross(a, b)`              | $[a_2b_3-a_3b_2, a_3b_1-a_1b_3, a_1b_2-a_2b_1]$ |\n",
    "| Magnitude (norm)      | $\\|\\mathbf{v}\\|$                         | `np.linalg.norm(v)`           | $\\sqrt{\\sum_{i=1}^{n} v_i^2}$ or `np.sqrt(np.sum(v**2))` |\n",
    "| Normalize             | $\\hat{\\mathbf{v}} = \\frac{\\mathbf{v}}{\\|\\mathbf{v}\\|}$ | `v / np.linalg.norm(v)` | Divide by magnitude |\n",
    "| Projection            | $\\text{proj}_{\\mathbf{b}}\\mathbf{a}$     | `(a@b / b@b) * b`             | $\\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\mathbf{b} \\cdot \\mathbf{b}}\\mathbf{b}$ |\n",
    "\n",
    "## Matrix Operations\n",
    "\n",
    "| Operation             | Math Notation                                               | NumPy Code                    | Note (Manual Calculation) |\n",
    "| --------------------- | ----------------------------------------------------------- | ----------------------------- | ------------------------- |\n",
    "| Create matrix         | $\\mathbf{A} = \\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\end{bmatrix}$ | `A = np.array([[1,2],[3,4]])` | Nested lists (rows) |\n",
    "| Identity matrix       | $\\mathbf{I}_n$                                              | `np.eye(n)`                   | 1s on diagonal, 0s elsewhere |\n",
    "| Zero matrix           | $\\mathbf{0}_{m \\times n}$                                   | `np.zeros((m, n))`            | All elements = 0 |\n",
    "| Matrix addition       | $\\mathbf{A} + \\mathbf{B}$                                   | `A + B`                       | Add element-wise: $c_{ij} = a_{ij} + b_{ij}$ |\n",
    "| Matrix multiplication | $\\mathbf{AB}$                                               | `A @ B` or `np.dot(A, B)`     | $c_{ij} = \\sum_{k=1}^{n} a_{ik} b_{kj}$ (row × column) |\n",
    "| Element-wise mult.    | $\\mathbf{A} \\odot \\mathbf{B}$                               | `A * B`                       | Multiply element-wise: $c_{ij} = a_{ij} \\cdot b_{ij}$ |\n",
    "| Transpose             | $\\mathbf{A}^T$                                              | `A.T`                         | Flip over diagonal: $(A^T)_{ij} = A_{ji}$ |\n",
    "| Inverse               | $\\mathbf{A}^{-1}$                                           | `np.linalg.inv(A)`            | Satisfies $\\mathbf{AA}^{-1} = \\mathbf{I}$ |\n",
    "| Determinant (2×2)     | $\\text{det}(\\mathbf{A})$                                    | `np.linalg.det(A)`            | $ad - bc$ for $\\begin{bmatrix} a & b \\\\ c & d \\end{bmatrix}$ |\n",
    "| Determinant (3×3)     | $\\text{det}(\\mathbf{A})$                                    | `np.linalg.det(A)`            | $a(ei-fh) - b(di-fg) + c(dh-eg)$ |\n",
    "| Rank                  | $\\text{rank}(\\mathbf{A})$                                   | `np.linalg.matrix_rank(A)`    | Number of linearly independent rows/columns |\n",
    "\n",
    "## Advanced Operations\n",
    "\n",
    "| Operation             | Math Notation                            | NumPy Code                          | Note (Manual Calculation) |\n",
    "| --------------------- | ---------------------------------------- | ----------------------------------- | ------------------------- |\n",
    "| Eigendecomposition    | $\\mathbf{Av} = \\lambda\\mathbf{v}$        | `eigenval, eigenvec = np.linalg.eig(A)` | Solve $\\text{det}(\\mathbf{A} - \\lambda\\mathbf{I}) = 0$ |\n",
    "| SVD                   | $\\mathbf{A} = \\mathbf{U}\\boldsymbol{\\Sigma}\\mathbf{V}^T$ | `U, S, Vt = np.linalg.svd(A)` | Factorize any matrix into 3 matrices |\n",
    "| Solve linear system   | $\\mathbf{Ax} = \\mathbf{b}$               | `x = np.linalg.solve(A, b)`         | $\\mathbf{x} = \\mathbf{A}^{-1}\\mathbf{b}$ (but use solver, not inverse!) |\n",
    "| Covariance matrix     | $\\mathbf{C} = \\frac{1}{n-1}\\mathbf{X}^T\\mathbf{X}$ | `np.cov(X.T)`            | Measures feature correlations (center data first) |\n",
    "| Trace                 | $\\text{tr}(\\mathbf{A})$                  | `np.trace(A)`                       | Sum of diagonal elements: $\\sum_{i=1}^{n} a_{ii}$ |\n",
    "| Frobenius norm        | $\\|\\mathbf{A}\\|_F$                       | `np.linalg.norm(A, 'fro')`          | $\\sqrt{\\sum_{i,j} a_{ij}^2}$ |\n",
    "\n",
    "## Matrix Creation\n",
    "\n",
    "| Function              | Description                              | Example                           | Note |\n",
    "| --------------------- | ---------------------------------------- | --------------------------------- | ---- |\n",
    "| `np.array()`          | Create from list                         | `np.array([[1,2],[3,4]])`         | Most basic method |\n",
    "| `np.eye(n)`           | n×n identity matrix                      | `np.eye(3)`                       | Can also use `np.identity(n)` |\n",
    "| `np.zeros((m,n))`     | m×n matrix of zeros                      | `np.zeros((2,3))`                 | Useful for initialization |\n",
    "| `np.ones((m,n))`      | m×n matrix of ones                       | `np.ones((2,3))`                  | Scale with `* value` if needed |\n",
    "| `np.diag(v)`          | Diagonal matrix from vector              | `np.diag([1,2,3])`                | Creates square matrix |\n",
    "| `np.random.rand(m,n)` | Random values in [0,1)                   | `np.random.rand(2,3)`             | Uniform distribution |\n",
    "| `np.random.randn(m,n)`| Random values from standard normal       | `np.random.randn(2,3)`            | Mean=0, std=1 |\n",
    "| `np.arange(start, stop, step)` | Array with evenly spaced values | `np.arange(0, 10, 2)`             | Like Python's `range()` |\n",
    "| `np.linspace(start, stop, n)`  | n evenly spaced values      | `np.linspace(0, 1, 5)`            | Includes both endpoints |\n",
    "\n",
    "---\n",
    "\n",
    "## Additional Resources\n",
    "\n",
    "- **NumPy Documentation**: https://numpy.org/doc/\n",
    "- **Linear Algebra Visualized (3Blue1Brown)**: https://www.3blue1brown.com/topics/linear-algebra\n",
    "- **SciPy Lectures**: https://scipy-lectures.org/intro/numpy/index.html\n",
    "- **NumPy Linear Algebra**: https://numpy.org/doc/stable/reference/routines.linalg.html\n",
    "- **PCA Tutorial**: https://builtin.com/data-science/step-step-explanation-principal-component-analysis\n",
    "- **SVD Tutorial**: https://www.askpython.com/python/examples/singular-value-decomposition\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
