{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8ca45b7d",
   "metadata": {},
   "source": [
    "# Exercise 1- Determinant, EigenValues, EigenVectors and Diagonalization\n",
    "\n",
    "Consider the matrix:\n",
    "\n",
    "$$\n",
    "A = \\begin{pmatrix}\n",
    "3/2 & 1/2 & -1/2 \\\\\n",
    "-1/2 & 5/2 & 1/2 \\\\\n",
    "-1 & 1 & 2 \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "1. Compute the **determinant** of $A$ and verify if it is **invertible**.\n",
    "2. Find all **eigenvalues** of $A$ by solving the characteristic polynomial.\n",
    "3. For each **eigenvalue**, determine a corresponding **eigenvector**.\n",
    "4. Check whether A is **diagonalizable**, and if yes, find matrices $P$ and $D$ such that $A = P D P^{-1}$.\n",
    "\n",
    "## Solution\n",
    "\n",
    "1. To compute the determinant of $A$, we use the formula for the determinant of a 3x3 matrix:\n",
    "\n",
    "   $$\n",
    "   \\text{det}(A) = a(ei - fh) - b(di - fg) + c(dh - eg)\n",
    "   $$\n",
    "\n",
    "   where the matrix elements are:\n",
    "\n",
    "   $$\n",
    "   A = \\begin{pmatrix}\n",
    "   a & b & c \\\\\n",
    "   d & e & f \\\\\n",
    "   g & h & i \\\\\n",
    "   \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   Plugging in the values from matrix $A$:\n",
    "\n",
    "   $$\n",
    "   \\text{det}(A) = \\frac{3}{2} \\left( \\frac{5}{2} \\cdot 2 - \\frac{1}{2} \\cdot 1 \\right) - \\frac{1}{2} \\left( -\\frac{1}{2} \\cdot 2 - \\frac{1}{2} \\cdot (-1) \\right) + \\left( -\\frac{1}{2} \\right) \\left( -\\frac{1}{2} \\cdot 1 - \\frac{5}{2} \\cdot (-1) \\right)\n",
    "   $$\n",
    "\n",
    "   Calculating step by step:\n",
    "\n",
    "   - First term: $\\frac{3}{2}(5 - \\frac{1}{2}) = \\frac{3}{2} \\cdot \\frac{9}{2} = \\frac{27}{4}$\n",
    "   - Second term: $-\\frac{1}{2}(-1 + \\frac{1}{2}) = -\\frac{1}{2} \\cdot (-\\frac{1}{2}) = \\frac{1}{4}$\n",
    "   - Third term: $-\\frac{1}{2}(-\\frac{1}{2} + \\frac{5}{2}) = -\\frac{1}{2} \\cdot 2 = -1$\n",
    "\n",
    "   $$\n",
    "   \\text{det}(A) = \\frac{27}{4} + \\frac{1}{4} - 1 = \\frac{28}{4} - 1 = 7 - 1 = 6\n",
    "   $$\n",
    "\n",
    "   Since $\\text{det}(A) = 6 \\neq 0$, matrix $A$ is invertible.\n",
    "\n",
    "2. To find the eigenvalues, we solve the characteristic polynomial given by:\n",
    "   $$\\text{det}(A - \\lambda I) = 0$$\n",
    "   where $I$ is the identity matrix and $\\lambda$ is the eigenvalue. Thus:\n",
    "\n",
    "   $$\n",
    "   A - \\lambda I = \\begin{pmatrix}\n",
    "   \\frac{3}{2} - \\lambda & \\frac{1}{2} & -\\frac{1}{2} \\\\\n",
    "   -\\frac{1}{2} & \\frac{5}{2} - \\lambda & \\frac{1}{2} \\\\\n",
    "   -1 & 1 & 2 - \\lambda \\\\\n",
    "   \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   Computing the determinant:\n",
    "\n",
    "   $$\n",
    "   \\text{det}(A - \\lambda I) = \\left(\\frac{3}{2} - \\lambda\\right) \\left[\\left(\\frac{5}{2} - \\lambda\\right)(2 - \\lambda) - \\frac{1}{2} \\cdot 1\\right] - \\frac{1}{2}\\left[-\\frac{1}{2}(2 - \\lambda) - \\frac{1}{2} \\cdot (-1)\\right] + \\left(-\\frac{1}{2}\\right)\\left[-\\frac{1}{2} \\cdot 1 - \\left(\\frac{5}{2} - \\lambda\\right)(-1)\\right]\n",
    "   $$\n",
    "\n",
    "   Expanding the first bracket:\n",
    "   $$\\left(\\frac{5}{2} - \\lambda\\right)(2 - \\lambda) - \\frac{1}{2} = 5 - \\frac{5\\lambda}{2} - 2\\lambda + \\lambda^2 - \\frac{1}{2} = \\lambda^2 - \\frac{9\\lambda}{2} + \\frac{9}{2}$$\n",
    "\n",
    "   Expanding the second bracket:\n",
    "   $$-\\frac{1}{2}(2 - \\lambda) - \\frac{1}{2}(-1) = -1 + \\frac{\\lambda}{2} + \\frac{1}{2} = \\frac{\\lambda}{2} - \\frac{1}{2}$$\n",
    "\n",
    "   Expanding the third bracket:\n",
    "   $$-\\frac{1}{2} - (-1)\\left(\\frac{5}{2} - \\lambda\\right) = -\\frac{1}{2} + \\frac{5}{2} - \\lambda = 2 - \\lambda$$\n",
    "\n",
    "   Combining all terms and simplifying yields the characteristic polynomial:\n",
    "   $$-\\lambda^3 + 6\\lambda^2 - 11\\lambda + 6 = 0$$\n",
    "\n",
    "   Or equivalently:\n",
    "   $$\\lambda^3 - 6\\lambda^2 + 11\\lambda - 6 = 0$$\n",
    "\n",
    "   We can factor this as:\n",
    "   $$(\\lambda - 1)(\\lambda - 2)(\\lambda - 3) = 0$$\n",
    "\n",
    "   Therefore, the eigenvalues are:\n",
    "\n",
    "   - $\\lambda_1 = 1$\n",
    "   - $\\lambda_2 = 2$\n",
    "   - $\\lambda_3 = 3$\n",
    "\n",
    "3. For each eigenvalue, we find the corresponding eigenvector by solving $(A - \\lambda I)\\vec{v} = \\vec{0}$ using **Gauss-Jordan elimination**:\n",
    "\n",
    "   **For $\\lambda_1 = 1$:**\n",
    "\n",
    "   We need to solve:\n",
    "\n",
    "   $$\n",
    "   (A - I)\\vec{v} = \\begin{pmatrix}\n",
    "   1/2 & 1/2 & -1/2 \\\\\n",
    "   -1/2 & 3/2 & 1/2 \\\\\n",
    "   -1 & 1 & 1 \\\\\n",
    "   \\end{pmatrix} \\begin{pmatrix} v_1 \\\\ v_2 \\\\ v_3 \\end{pmatrix} = \\begin{pmatrix} 0 \\\\ 0 \\\\ 0 \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   Apply row reduction to the augmented matrix:\n",
    "\n",
    "   Starting matrix:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1/2 & 1/2 & -1/2 & 0 \\\\\n",
    "   -1/2 & 3/2 & 1/2 & 0 \\\\\n",
    "   -1 & 1 & 1 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_1 \\times 2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & 1 & -1 & 0 \\\\\n",
    "   -1/2 & 3/2 & 1/2 & 0 \\\\\n",
    "   -1 & 1 & 1 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_2 + \\frac{1}{2}R_1$ and $R_3 + R_1$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & 1 & -1 & 0 \\\\\n",
    "   0 & 2 & 0 & 0 \\\\\n",
    "   0 & 2 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_2 \\times \\frac{1}{2}$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & 1 & -1 & 0 \\\\\n",
    "   0 & 1 & 0 & 0 \\\\\n",
    "   0 & 2 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_3 - 2R_2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & 1 & -1 & 0 \\\\\n",
    "   0 & 1 & 0 & 0 \\\\\n",
    "   0 & 0 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_1 - R_2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & 0 & -1 & 0 \\\\\n",
    "   0 & 1 & 0 & 0 \\\\\n",
    "   0 & 0 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   From the reduced form:\n",
    "\n",
    "   - $v_1 - v_3 = 0 \\Rightarrow v_1 = v_3$\n",
    "   - $v_2 = 0$\n",
    "   - $v_3$ is free\n",
    "\n",
    "   Choosing $v_3 = 1$, we get: $\\vec{v}_1 = \\begin{pmatrix} 1 \\\\ 0 \\\\ 1 \\end{pmatrix}$\n",
    "\n",
    "   **For $\\lambda_2 = 2$:**\n",
    "\n",
    "   We need to solve:\n",
    "\n",
    "   $$\n",
    "   (A - 2I)\\vec{v} = \\begin{pmatrix}\n",
    "   -1/2 & 1/2 & -1/2 \\\\\n",
    "   -1/2 & 1/2 & 1/2 \\\\\n",
    "   -1 & 1 & 0 \\\\\n",
    "   \\end{pmatrix} \\begin{pmatrix} v_1 \\\\ v_2 \\\\ v_3 \\end{pmatrix} = \\begin{pmatrix} 0 \\\\ 0 \\\\ 0 \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   Apply row reduction:\n",
    "\n",
    "   Starting matrix:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   -1/2 & 1/2 & -1/2 & 0 \\\\\n",
    "   -1/2 & 1/2 & 1/2 & 0 \\\\\n",
    "   -1 & 1 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_1 \\times (-2)$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1 & 1 & 0 \\\\\n",
    "   -1/2 & 1/2 & 1/2 & 0 \\\\\n",
    "   -1 & 1 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_2 + \\frac{1}{2}R_1$ and $R_3 + R_1$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1 & 1 & 0 \\\\\n",
    "   0 & 0 & 1 & 0 \\\\\n",
    "   0 & 0 & 1 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_3 - R_2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1 & 1 & 0 \\\\\n",
    "   0 & 0 & 1 & 0 \\\\\n",
    "   0 & 0 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_1 - R_2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1 & 0 & 0 \\\\\n",
    "   0 & 0 & 1 & 0 \\\\\n",
    "   0 & 0 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   From the reduced form:\n",
    "\n",
    "   - $v_1 - v_2 = 0 \\Rightarrow v_1 = v_2$\n",
    "   - $v_3 = 0$\n",
    "   - $v_2$ is free\n",
    "\n",
    "   Choosing $v_2 = 1$, we get: $\\vec{v}_2 = \\begin{pmatrix} 1 \\\\ 1 \\\\ 0 \\end{pmatrix}$\n",
    "\n",
    "   **For $\\lambda_3 = 3$:**\n",
    "\n",
    "   We need to solve:\n",
    "\n",
    "   $$\n",
    "   (A - 3I)\\vec{v} = \\begin{pmatrix}\n",
    "   -3/2 & 1/2 & -1/2 \\\\\n",
    "   -1/2 & -1/2 & 1/2 \\\\\n",
    "   -1 & 1 & -1 \\\\\n",
    "   \\end{pmatrix} \\begin{pmatrix} v_1 \\\\ v_2 \\\\ v_3 \\end{pmatrix} = \\begin{pmatrix} 0 \\\\ 0 \\\\ 0 \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   Apply row reduction:\n",
    "\n",
    "   Starting matrix:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   -3/2 & 1/2 & -1/2 & 0 \\\\\n",
    "   -1/2 & -1/2 & 1/2 & 0 \\\\\n",
    "   -1 & 1 & -1 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_1 \\times (-2/3)$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1/3 & 1/3 & 0 \\\\\n",
    "   -1/2 & -1/2 & 1/2 & 0 \\\\\n",
    "   -1 & 1 & -1 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_2 + \\frac{1}{2}R_1$ and $R_3 + R_1$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1/3 & 1/3 & 0 \\\\\n",
    "   0 & -2/3 & 2/3 & 0 \\\\\n",
    "   0 & 2/3 & -2/3 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_2 \\times (-3/2)$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1/3 & 1/3 & 0 \\\\\n",
    "   0 & 1 & -1 & 0 \\\\\n",
    "   0 & 2/3 & -2/3 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_3 - \\frac{2}{3}R_2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & -1/3 & 1/3 & 0 \\\\\n",
    "   0 & 1 & -1 & 0 \\\\\n",
    "   0 & 0 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   $R_1 + \\frac{1}{3}R_2$:\n",
    "\n",
    "   $$\n",
    "   \\left[\\begin{array}{ccc|c}\n",
    "   1 & 0 & 0 & 0 \\\\\n",
    "   0 & 1 & -1 & 0 \\\\\n",
    "   0 & 0 & 0 & 0\n",
    "   \\end{array}\\right]\n",
    "   $$\n",
    "\n",
    "   From the reduced form:\n",
    "\n",
    "   - $v_1 = 0$\n",
    "   - $v_2 - v_3 = 0 \\Rightarrow v_2 = v_3$\n",
    "   - $v_3$ is free\n",
    "\n",
    "   Choosing $v_3 = 1$, we get: $\\vec{v}_3 = \\begin{pmatrix} 0 \\\\ 1 \\\\ 1 \\end{pmatrix}$\n",
    "\n",
    "4. **Diagonalizability Check:**\n",
    "\n",
    "   A matrix $A$ is diagonalizable if it has $n$ linearly independent eigenvectors, where $n$ is the dimension.\n",
    "\n",
    "   We have found three eigenvectors:\n",
    "\n",
    "   $$\n",
    "   \\vec{v}_1 = \\begin{pmatrix} 1 \\\\ 0 \\\\ 1 \\end{pmatrix}, \\quad \\vec{v}_2 = \\begin{pmatrix} 1 \\\\ 1 \\\\ 0 \\end{pmatrix}, \\quad \\vec{v}_3 = \\begin{pmatrix} 0 \\\\ 1 \\\\ 1 \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   To check linear independence, we form the matrix with these vectors as columns and check its determinant:\n",
    "\n",
    "   $$\n",
    "   P = \\begin{pmatrix}\n",
    "   1 & 1 & 0 \\\\\n",
    "   0 & 1 & 1 \\\\\n",
    "   1 & 0 & 1 \\\\\n",
    "   \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   $$\n",
    "   \\text{det}(P) = 1(1 \\cdot 1 - 1 \\cdot 0) - 1(0 \\cdot 1 - 1 \\cdot 1) + 0(0 \\cdot 0 - 1 \\cdot 1) = 1(1) - 1(-1) + 0 = 1 + 1 = 2 \\neq 0\n",
    "   $$\n",
    "\n",
    "   Since $\\text{det}(P) \\neq 0$, the eigenvectors are linearly independent, and **A is diagonalizable**.\n",
    "\n",
    "   The diagonal matrix $D$ contains the eigenvalues:\n",
    "\n",
    "   $$\n",
    "   D = \\begin{pmatrix}\n",
    "   1 & 0 & 0 \\\\\n",
    "   0 & 2 & 0 \\\\\n",
    "   0 & 0 & 3 \\\\\n",
    "   \\end{pmatrix}\n",
    "   $$\n",
    "\n",
    "   And the decomposition is:\n",
    "\n",
    "   $$\n",
    "   A = PDP^{-1}\n",
    "   $$\n",
    "\n",
    "   where:\n",
    "\n",
    "   $$\n",
    "   P = \\begin{pmatrix}\n",
    "   1 & 1 & 0 \\\\\n",
    "   0 & 1 & 1 \\\\\n",
    "   1 & 0 & 1 \\\\\n",
    "   \\end{pmatrix}, \\quad D = \\begin{pmatrix}\n",
    "   1 & 0 & 0 \\\\\n",
    "   0 & 2 & 0 \\\\\n",
    "   0 & 0 & 3 \\\\\n",
    "   \\end{pmatrix}\n",
    "   $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1d459b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define matrix A\n",
    "A = np.array([[3 / 2, 1 / 2, -1 / 2], [-1 / 2, 5 / 2, 1 / 2], [-1, 1, 2]])\n",
    "\n",
    "print(\"Matrix A:\")\n",
    "print(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e6d6989",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Compute the determinant\n",
    "det_A = np.linalg.det(A)\n",
    "print(f\"Determinant of A: {det_A:.4f}\")\n",
    "print(f\"Is A invertible? {det_A != 0}\")\n",
    "print(f\"Since det(A) ≠ 0, matrix A is invertible.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "190b6f87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. Find eigenvalues\n",
    "eigenvalues, eigenvectors = np.linalg.eig(A)\n",
    "\n",
    "print(\"Eigenvalues of A:\")\n",
    "for i, eigenvalue in enumerate(eigenvalues):\n",
    "    print(f\"  λ{i+1} = {eigenvalue:.6f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "484e04fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3. Find eigenvectors for each eigenvalue\n",
    "print(\"\\nEigenvectors of A:\")\n",
    "print(\"Matrix P (columns are eigenvectors):\")\n",
    "print(eigenvectors)\n",
    "print(\"\\nEach column corresponds to an eigenvalue:\")\n",
    "for i in range(len(eigenvalues)):\n",
    "    print(f\"\\nEigenvector for λ{i+1} = {eigenvalues[i]:.6f}:\")\n",
    "    print(eigenvectors[:, i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d6b809a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. Check if A is diagonalizable\n",
    "# A matrix is diagonalizable if it has n linearly independent eigenvectors\n",
    "# where n is the dimension of the matrix\n",
    "\n",
    "# Check if eigenvectors are linearly independent by checking rank of P\n",
    "P = eigenvectors\n",
    "rank_P = np.linalg.matrix_rank(P)\n",
    "\n",
    "print(f\"Rank of eigenvector matrix P: {rank_P}\")\n",
    "print(f\"Dimension of A: {A.shape[0]}\")\n",
    "\n",
    "if rank_P == A.shape[0]:\n",
    "    print(\n",
    "        \"\\nMatrix A is DIAGONALIZABLE because it has 3 linearly independent eigenvectors.\"\n",
    "    )\n",
    "else:\n",
    "    print(\"\\nMatrix A is NOT diagonalizable.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cddc2809",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct diagonal matrix D with eigenvalues on the diagonal\n",
    "D = np.diag(eigenvalues)\n",
    "\n",
    "print(\"\\nMatrix D (diagonal matrix of eigenvalues):\")\n",
    "print(D)\n",
    "\n",
    "print(\"\\nMatrix P (eigenvectors as columns):\")\n",
    "print(P)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49457c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify that A = P * D * P^(-1)\n",
    "P_inv = np.linalg.inv(P)\n",
    "A_reconstructed = P @ D @ P_inv\n",
    "\n",
    "print(\"\\nVerification: A = P * D * P^(-1)\")\n",
    "print(\"\\nOriginal matrix A:\")\n",
    "print(A)\n",
    "print(\"\\nReconstructed A from P * D * P^(-1):\")\n",
    "print(A_reconstructed)\n",
    "print(\"\\nDifference (should be close to zero):\")\n",
    "print(np.abs(A - A_reconstructed))\n",
    "print(f\"\\nMaximum difference: {np.max(np.abs(A - A_reconstructed)):.2e}\")\n",
    "\n",
    "if np.allclose(A, A_reconstructed):\n",
    "    print(\"\\n✓ Verification successful! A = P D P^(-1)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1127906",
   "metadata": {},
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14856001",
   "metadata": {},
   "source": [
    "# Exercise 2 - Singular Value Decomposition (SVD)\n",
    "\n",
    "Consider the matrix:\n",
    "\n",
    "$$\n",
    "S = \\begin{pmatrix}\n",
    "1 & 1 \\\\\n",
    "0 & 1 \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "1. Show that $S$ has a single eigenvalue $\\lambda = 1$ with only one independent eigenvector. Conclude that $S$ is not diagonalizable.\n",
    "2. Compute $S^T S$ and its eigenvalues $\\{\\lambda_1, \\lambda_2\\}$; set the singular values $\\sigma_i = \\sqrt{\\lambda_i}$.\n",
    "3. Find the right singular vectors $v_i$ as eigenvectors of $S^T S$ and the left singular vectors $u_i = \\frac{Sv_i}{\\sigma_i}$.\n",
    "4. Conclude the **SVD** $S = U \\Sigma V^T$ with $U = [u_1\\; u_2]$, $V = [v_1\\; v_2]$ and $\\Sigma = \\operatorname{diag}(\\sigma_1, \\sigma_2)$.\n",
    "\n",
    "## Solution\n",
    "\n",
    "### Part 1: Eigenvalues and Diagonalizability\n",
    "\n",
    "To find the eigenvalues, we solve the characteristic equation:\n",
    "$$\\text{det}(S - \\lambda I) = 0$$\n",
    "\n",
    "$$\n",
    "S - \\lambda I = \\begin{pmatrix}\n",
    "1 - \\lambda & 1 \\\\\n",
    "0 & 1 - \\lambda \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\text{det}(S - \\lambda I) = (1 - \\lambda)(1 - \\lambda) - 0 \\cdot 1 = (1 - \\lambda)^2 = 0\n",
    "$$\n",
    "\n",
    "Therefore, $\\lambda = 1$ is the only eigenvalue with **algebraic multiplicity 2**.\n",
    "\n",
    "Now, find the eigenvectors by solving $(S - I)\\vec{v} = \\vec{0}$:\n",
    "\n",
    "$$\n",
    "(S - I) = \\begin{pmatrix}\n",
    "0 & 1 \\\\\n",
    "0 & 0 \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Apply row reduction:\n",
    "\n",
    "$$\n",
    "\\left[\\begin{array}{cc|c}\n",
    "0 & 1 & 0 \\\\\n",
    "0 & 0 & 0\n",
    "\\end{array}\\right]\n",
    "$$\n",
    "\n",
    "From this: $v_2 = 0$ and $v_1$ is free.\n",
    "\n",
    "Choosing $v_1 = 1$, we get: $\\vec{v} = \\begin{pmatrix} 1 \\\\ 0 \\end{pmatrix}$\n",
    "\n",
    "**Conclusion:** There is only **one independent eigenvector** for eigenvalue $\\lambda = 1$. The geometric multiplicity (1) is less than the algebraic multiplicity (2), so **$S$ is NOT diagonalizable**.\n",
    "\n",
    "### Part 2: Compute $S^T S$ and Singular Values\n",
    "\n",
    "First, compute $S^T$:\n",
    "\n",
    "$$\n",
    "S^T = \\begin{pmatrix}\n",
    "1 & 0 \\\\\n",
    "1 & 1 \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Now compute $S^T S$:\n",
    "\n",
    "$$\n",
    "S^T S = \\begin{pmatrix}\n",
    "1 & 0 \\\\\n",
    "1 & 1 \\\\\n",
    "\\end{pmatrix} \\begin{pmatrix}\n",
    "1 & 1 \\\\\n",
    "0 & 1 \\\\\n",
    "\\end{pmatrix} = \\begin{pmatrix}\n",
    "1 & 1 \\\\\n",
    "1 & 2 \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "Find eigenvalues of $S^T S$ by solving $\\text{det}(S^T S - \\lambda I) = 0$:\n",
    "\n",
    "$$\n",
    "\\text{det}\\begin{pmatrix}\n",
    "1 - \\lambda & 1 \\\\\n",
    "1 & 2 - \\lambda \\\\\n",
    "\\end{pmatrix} = (1 - \\lambda)(2 - \\lambda) - 1 = 2 - \\lambda - 2\\lambda + \\lambda^2 - 1 = \\lambda^2 - 3\\lambda + 1 = 0\n",
    "$$\n",
    "\n",
    "Using the quadratic formula:\n",
    "\n",
    "$$\n",
    "\\lambda = \\frac{3 \\pm \\sqrt{9 - 4}}{2} = \\frac{3 \\pm \\sqrt{5}}{2}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "- $\\lambda_1 = \\frac{3 + \\sqrt{5}}{2} \\approx 2.618$\n",
    "- $\\lambda_2 = \\frac{3 - \\sqrt{5}}{2} \\approx 0.382$\n",
    "\n",
    "The singular values are:\n",
    "\n",
    "$$\n",
    "\\sigma_1 = \\sqrt{\\lambda_1} = \\sqrt{\\frac{3 + \\sqrt{5}}{2}} \\approx 1.618\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\sigma_2 = \\sqrt{\\lambda_2} = \\sqrt{\\frac{3 - \\sqrt{5}}{2}} \\approx 0.618\n",
    "$$\n",
    "\n",
    "### Part 3: Right and Left Singular Vectors\n",
    "\n",
    "**Right singular vectors** are eigenvectors of $S^T S$.\n",
    "\n",
    "**For $\\lambda_1 = \\frac{3 + \\sqrt{5}}{2}$:**\n",
    "\n",
    "Solve $(S^T S - \\lambda_1 I)\\vec{v}_1 = \\vec{0}$:\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "1 - \\frac{3 + \\sqrt{5}}{2} & 1 \\\\\n",
    "1 & 2 - \\frac{3 + \\sqrt{5}}{2} \\\\\n",
    "\\end{pmatrix} = \\begin{pmatrix}\n",
    "\\frac{-1 - \\sqrt{5}}{2} & 1 \\\\\n",
    "1 & \\frac{1 - \\sqrt{5}}{2} \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "From the first row: $\\frac{-1 - \\sqrt{5}}{2}v_1 + v_2 = 0$\n",
    "\n",
    "$$v_2 = \\frac{1 + \\sqrt{5}}{2}v_1$$\n",
    "\n",
    "Normalizing with $v_1 = 1$: $\\vec{v}_1 = \\begin{pmatrix} 1 \\\\ \\frac{1 + \\sqrt{5}}{2} \\end{pmatrix}$\n",
    "\n",
    "After normalization: $\\vec{v}_1 = \\frac{1}{\\sqrt{1 + \\left(\\frac{1+\\sqrt{5}}{2}\\right)^2}} \\begin{pmatrix} 1 \\\\ \\frac{1 + \\sqrt{5}}{2} \\end{pmatrix}$\n",
    "\n",
    "**For $\\lambda_2 = \\frac{3 - \\sqrt{5}}{2}$:**\n",
    "\n",
    "Similarly, solving $(S^T S - \\lambda_2 I)\\vec{v}_2 = \\vec{0}$ gives:\n",
    "\n",
    "$$v_2 = \\frac{1 - \\sqrt{5}}{2}v_1$$\n",
    "\n",
    "After normalization: $\\vec{v}_2 = \\frac{1}{\\sqrt{1 + \\left(\\frac{1-\\sqrt{5}}{2}\\right)^2}} \\begin{pmatrix} 1 \\\\ \\frac{1 - \\sqrt{5}}{2} \\end{pmatrix}$\n",
    "\n",
    "**Left singular vectors** are computed as $u_i = \\frac{Sv_i}{\\sigma_i}$:\n",
    "\n",
    "$$u_1 = \\frac{Sv_1}{\\sigma_1}, \\quad u_2 = \\frac{Sv_2}{\\sigma_2}$$\n",
    "\n",
    "### Part 4: SVD Decomposition\n",
    "\n",
    "The Singular Value Decomposition is:\n",
    "\n",
    "$$\n",
    "S = U \\Sigma V^T\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $U = [u_1\\; u_2]$ is a $2 \\times 2$ orthogonal matrix of left singular vectors\n",
    "- $\\Sigma = \\begin{pmatrix} \\sigma_1 & 0 \\\\ 0 & \\sigma_2 \\end{pmatrix} = \\begin{pmatrix} \\sqrt{\\frac{3+\\sqrt{5}}{2}} & 0 \\\\ 0 & \\sqrt{\\frac{3-\\sqrt{5}}{2}} \\end{pmatrix}$\n",
    "- $V = [v_1\\; v_2]$ is a $2 \\times 2$ orthogonal matrix of right singular vectors\n",
    "\n",
    "This decomposition always exists even when $S$ is not diagonalizable, making SVD more general than eigendecomposition.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e16e00f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define matrix S\n",
    "S = np.array([[1, 1], [0, 1]])\n",
    "\n",
    "print(\"Matrix S:\")\n",
    "print(S)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f040cd0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 1: Find eigenvalues and eigenvectors of S\n",
    "eigenvalues_S, eigenvectors_S = np.linalg.eig(S)\n",
    "\n",
    "print(\"Part 1: Eigenvalues and Eigenvectors of S\")\n",
    "print(\"=\" * 50)\n",
    "print(f\"\\nEigenvalues of S: {eigenvalues_S}\")\n",
    "print(f\"\\nEigenvectors of S:\")\n",
    "print(eigenvectors_S)\n",
    "\n",
    "# Check rank of eigenvector matrix\n",
    "rank_eig = np.linalg.matrix_rank(eigenvectors_S)\n",
    "print(f\"\\nRank of eigenvector matrix: {rank_eig}\")\n",
    "print(f\"Dimension of S: {S.shape[0]}\")\n",
    "\n",
    "if rank_eig < S.shape[0]:\n",
    "    print(\"\\n✗ S is NOT diagonalizable (rank < dimension)\")\n",
    "else:\n",
    "    print(\"\\n✓ S is diagonalizable\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "739b0ea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 2: Compute S^T S and its eigenvalues, then singular values\n",
    "S_T = S.T\n",
    "S_T_S = S_T @ S\n",
    "\n",
    "print(\"\\nPart 2: S^T S and Singular Values\")\n",
    "print(\"=\" * 50)\n",
    "print(\"\\nS^T (transpose of S):\")\n",
    "print(S_T)\n",
    "\n",
    "print(\"\\nS^T S:\")\n",
    "print(S_T_S)\n",
    "\n",
    "# Find eigenvalues of S^T S\n",
    "eigenvalues_STS, eigenvectors_STS = np.linalg.eig(S_T_S)\n",
    "\n",
    "# Sort eigenvalues in descending order\n",
    "idx = eigenvalues_STS.argsort()[::-1]\n",
    "eigenvalues_STS = eigenvalues_STS[idx]\n",
    "eigenvectors_STS = eigenvectors_STS[:, idx]\n",
    "\n",
    "print(f\"\\nEigenvalues of S^T S:\")\n",
    "print(f\"  λ1 = {eigenvalues_STS[0]:.6f}\")\n",
    "print(f\"  λ2 = {eigenvalues_STS[1]:.6f}\")\n",
    "\n",
    "# Compute singular values\n",
    "singular_values = np.sqrt(eigenvalues_STS)\n",
    "\n",
    "print(f\"\\nSingular values:\")\n",
    "print(f\"  σ1 = √λ1 = {singular_values[0]:.6f}\")\n",
    "print(f\"  σ2 = √λ2 = {singular_values[1]:.6f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5fea0c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 3: Compute right and left singular vectors\n",
    "print(\"\\nPart 3: Right and Left Singular Vectors\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "# Right singular vectors V (eigenvectors of S^T S)\n",
    "V = eigenvectors_STS\n",
    "\n",
    "print(\"\\nRight singular vectors V (columns are v1, v2):\")\n",
    "print(V)\n",
    "print(f\"\\nv1 = {V[:, 0]}\")\n",
    "print(f\"v2 = {V[:, 1]}\")\n",
    "\n",
    "# Left singular vectors U: u_i = S*v_i / σ_i\n",
    "U = np.zeros((2, 2))\n",
    "for i in range(2):\n",
    "    U[:, i] = (S @ V[:, i]) / singular_values[i]\n",
    "\n",
    "print(\"\\nLeft singular vectors U (columns are u1, u2):\")\n",
    "print(U)\n",
    "print(f\"\\nu1 = {U[:, 0]}\")\n",
    "print(f\"u2 = {U[:, 1]}\")\n",
    "\n",
    "# Verify orthonormality\n",
    "print(\"\\nVerification of orthonormality:\")\n",
    "print(f\"U^T U = \\n{U.T @ U}\")\n",
    "print(f\"V^T V = \\n{V.T @ V}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b2b6034",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 4: Construct and verify SVD\n",
    "print(\"\\nPart 4: Singular Value Decomposition (SVD)\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "# Construct diagonal matrix Σ\n",
    "Sigma = np.diag(singular_values)\n",
    "\n",
    "print(\"\\nMatrix Σ (diagonal matrix of singular values):\")\n",
    "print(Sigma)\n",
    "\n",
    "print(\"\\nMatrix U (left singular vectors):\")\n",
    "print(U)\n",
    "\n",
    "print(\"\\nMatrix V (right singular vectors):\")\n",
    "print(V)\n",
    "\n",
    "# Reconstruct S using SVD: S = U Σ V^T\n",
    "S_reconstructed = U @ Sigma @ V.T\n",
    "\n",
    "print(\"\\n\\nVerification: S = U Σ V^T\")\n",
    "print(\"\\nOriginal matrix S:\")\n",
    "print(S)\n",
    "\n",
    "print(\"\\nReconstructed S from U Σ V^T:\")\n",
    "print(S_reconstructed)\n",
    "\n",
    "print(\"\\nDifference (should be close to zero):\")\n",
    "print(np.abs(S - S_reconstructed))\n",
    "\n",
    "print(f\"\\nMaximum difference: {np.max(np.abs(S - S_reconstructed)):.2e}\")\n",
    "\n",
    "if np.allclose(S, S_reconstructed):\n",
    "    print(\"\\n✓ SVD verification successful! S = U Σ V^T\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b556ee3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bonus: Compare with NumPy's built-in SVD\n",
    "print(\"\\n\" + \"=\" * 50)\n",
    "print(\"Bonus: Comparison with NumPy's SVD function\")\n",
    "print(\"=\" * 50)\n",
    "\n",
    "U_numpy, Sigma_numpy, VT_numpy = np.linalg.svd(S)\n",
    "\n",
    "print(\"\\nNumPy's U:\")\n",
    "print(U_numpy)\n",
    "\n",
    "print(\"\\nNumPy's singular values:\")\n",
    "print(Sigma_numpy)\n",
    "\n",
    "print(\"\\nNumPy's V^T:\")\n",
    "print(VT_numpy)\n",
    "\n",
    "print(\"\\nNumPy's V:\")\n",
    "print(VT_numpy.T)\n",
    "\n",
    "print(\"\\n\\nComparison:\")\n",
    "print(f\"Our σ1 = {singular_values[0]:.6f}, NumPy's σ1 = {Sigma_numpy[0]:.6f}\")\n",
    "print(f\"Our σ2 = {singular_values[1]:.6f}, NumPy's σ2 = {Sigma_numpy[1]:.6f}\")\n",
    "print(\"\\n✓ Results match (note: signs of vectors may differ, which is normal)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a350444",
   "metadata": {},
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ad9ce0c",
   "metadata": {},
   "source": [
    "# Exercise 3- PCA on a tiny 2D dataset\n",
    "\n",
    "Given the four points (rows are samples):\n",
    "\n",
    "$$\n",
    "X = \\begin{pmatrix}\n",
    "2 & 0 \\\\\n",
    "0 & 2 \\\\\n",
    "3 & 1 \\\\\n",
    "1 & 3 \\\\\n",
    "\\end{pmatrix} \\in \\mathbb{R}^{4 \\times 2}\n",
    "$$\n",
    "\n",
    "1. Center the data: compute the column mean $\\mu$ and $Z = X - \\mathbf{1}\\,\\mu^T$.\n",
    "2. Compute the sample covariance matrix $C = \\frac{1}{n-1} Z^T Z$.\n",
    "3. Compute the eigenvalues and eigenvectors of $C$ (principal components).\n",
    "4. Order components by decreasing eigenvalues, and give the explained variance ratio for $k = 1$ and $k = 2$.\n",
    "5. Project the centered data on the first principal component $T_1 = Z v_1$.\n",
    "\n",
    "## Solution\n",
    "\n",
    "1. Centering the data\n",
    "\n",
    "- Column mean:\n",
    "  $$\\mu = \\frac{1}{4} \\sum_{i=1}^4 X_{i,:} = \\begin{pmatrix} \\tfrac{2+0+3+1}{4} \\\\ \\tfrac{0+2+1+3}{4} \\end{pmatrix} = \\begin{pmatrix} 1.5 \\\\ 1.5 \\end{pmatrix}$$\n",
    "- Centered matrix $Z = X - \\mathbf{1}\\,\\mu^T$:\n",
    "  $$\n",
    "  Z = \\begin{pmatrix}\n",
    "  2-1.5 & 0-1.5 \\\\\n",
    "  0-1.5 & 2-1.5 \\\\\n",
    "  3-1.5 & 1-1.5 \\\\\n",
    "  1-1.5 & 3-1.5 \\\\\n",
    "  \\end{pmatrix} = \\begin{pmatrix}\n",
    "  0.5 & -1.5 \\\\\n",
    "  -1.5 & 0.5 \\\\\n",
    "  1.5 & -0.5 \\\\\n",
    "  -0.5 & 1.5 \\\\\n",
    "  \\end{pmatrix}\n",
    "  $$\n",
    "\n",
    "2. Sample covariance matrix\n",
    "\n",
    "- Compute $Z^T Z$:\n",
    "  $$\n",
    "  Z^T Z = \\begin{pmatrix}\n",
    "  5 & -3 \\\\\n",
    "  -3 & 5 \\\\\n",
    "  \\end{pmatrix}\n",
    "  $$\n",
    "- With $n=4$, the unbiased sample covariance is:\n",
    "  $$\n",
    "  C = \\frac{1}{n-1} Z^T Z = \\frac{1}{3} \\begin{pmatrix} 5 & -3 \\\\ -3 & 5 \\end{pmatrix} = \\begin{pmatrix} \\tfrac{5}{3} & -1 \\\\ -1 & \\tfrac{5}{3} \\end{pmatrix}\n",
    "  $$\n",
    "\n",
    "3. Eigen-decomposition of $C$\n",
    "\n",
    "- For a matrix of the form $\\begin{pmatrix} a & b \\\\ b & a \\end{pmatrix}$, eigenvalues are $a\\pm b$ with eigenvectors proportional to $\\begin{pmatrix} 1 \\\\ 1 \\end{pmatrix}$ and $\\begin{pmatrix} 1 \\\\ -1 \\end{pmatrix}$.\n",
    "- Here $a=\\tfrac{5}{3}$, $b=-1$:\n",
    "  $$\n",
    "  \\lambda_1 = a - b = \\tfrac{5}{3} - (-1) = \\tfrac{8}{3} \\approx 2.6667, \\quad v_1 \\propto \\begin{pmatrix} 1 \\\\ -1 \\end{pmatrix}\n",
    "  $$\n",
    "  $$\n",
    "  \\lambda_2 = a + b = \\tfrac{5}{3} + (-1) = \\tfrac{2}{3} \\approx 0.6667, \\quad v_2 \\propto \\begin{pmatrix} 1 \\\\ 1 \\end{pmatrix}\n",
    "  $$\n",
    "- Unit-norm eigenvectors:\n",
    "  $$\n",
    "  v_1 = \\tfrac{1}{\\sqrt{2}} \\begin{pmatrix} 1 \\\\ -1 \\end{pmatrix},\\quad v_2 = \\tfrac{1}{\\sqrt{2}} \\begin{pmatrix} 1 \\\\ 1 \\end{pmatrix}\n",
    "  $$\n",
    "\n",
    "4. Explained variance ratio\n",
    "\n",
    "- Total variance $= \\lambda_1 + \\lambda_2 = \\tfrac{8}{3} + \\tfrac{2}{3} = \\tfrac{10}{3}$.\n",
    "- For $k=1$ (first PC only):\n",
    "  $$\n",
    "  \\text{EVR}_{k=1} = \\frac{\\lambda_1}{\\lambda_1+\\lambda_2} = \\frac{\\tfrac{8}{3}}{\\tfrac{10}{3}} = \\frac{8}{10} = 0.8\n",
    "  $$\n",
    "- For $k=2$ (both PCs): EVR $=1.0$.\n",
    "\n",
    "5. Projection onto the first principal component\n",
    "\n",
    "- Using $v_1 = \\tfrac{1}{\\sqrt{2}}(1, -1)^T$, the scores $T_1 = Z v_1$ are:\n",
    "  $$\n",
    "  T_1 = \\frac{1}{\\sqrt{2}} \\begin{pmatrix}\n",
    "  0.5 - (-1.5) \\\\\n",
    "  -1.5 - 0.5 \\\\\n",
    "  1.5 - (-0.5) \\\\\n",
    "  -0.5 - 1.5 \\\\\n",
    "  \\end{pmatrix} = \\frac{1}{\\sqrt{2}} \\begin{pmatrix}\n",
    "  2 \\\\\n",
    "  -2 \\\\\n",
    "  2 \\\\\n",
    "  -2 \\\\\n",
    "  \\end{pmatrix} = \\begin{pmatrix}\n",
    "  \\sqrt{2} \\\\\n",
    "  -\\sqrt{2} \\\\\n",
    "  \\sqrt{2} \\\\\n",
    "  -\\sqrt{2} \\\\\n",
    "  \\end{pmatrix}\n",
    "  $$\n",
    "\n",
    "Therefore, the principal directions are along $(1,-1)$ and $(1,1)$, with the first PC explaining 80% of the variance, and the projections alternate $\\pm\\sqrt{2}$ across the four centered points.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e67d960e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 3 - PCA (code verification)\n",
    "import numpy as np\n",
    "\n",
    "# Data matrix X (4x2)\n",
    "X = np.array([[2, 0], [0, 2], [3, 1], [1, 3]], dtype=float)\n",
    "\n",
    "print(\"X:\\n\", X)\n",
    "\n",
    "# 1) Center the data\n",
    "mu = X.mean(axis=0)\n",
    "Z = X - mu\n",
    "print(\"\\nMean mu:\", mu)\n",
    "print(\"\\nCentered Z:\\n\", Z)\n",
    "\n",
    "# 2) Sample covariance matrix (unbiased)\n",
    "C = (Z.T @ Z) / (len(X) - 1)\n",
    "print(\"\\nCovariance C = (1/(n-1)) Z^T Z:\\n\", C)\n",
    "\n",
    "# 3) Eigen-decomposition (symmetric)\n",
    "vals, vecs = np.linalg.eigh(C)  # eigh for symmetric matrices\n",
    "# Sort in descending order\n",
    "idx = np.argsort(vals)[::-1]\n",
    "vals = vals[idx]\n",
    "vecs = vecs[:, idx]\n",
    "print(\"\\nEigenvalues (desc):\", vals)\n",
    "print(\"Eigenvectors (columns, aligned with eigenvalues):\\n\", vecs)\n",
    "\n",
    "# 4) Explained variance ratios\n",
    "explained = vals / vals.sum()\n",
    "print(\"\\nExplained variance ratios:\", explained)\n",
    "print(f\"EVR k=1: {explained[0]:.4f}, EVR k=2: {explained.sum():.4f}\")\n",
    "\n",
    "# 5) Projection on first principal component\n",
    "v1 = vecs[:, 0]\n",
    "T1 = Z @ v1\n",
    "print(\"\\nFirst principal direction v1:\", v1)\n",
    "print(\"Projection T1 = Z v1:\\n\", T1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b8fbf76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bonus: Compare with scikit-learn PCA (optional)\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "pca = PCA(n_components=2, svd_solver=\"full\")\n",
    "Z_sklearn = X - X.mean(axis=0)\n",
    "pca.fit(Z_sklearn)\n",
    "\n",
    "print(\"\\n[sklearn] Components (rows):\\n\", pca.components_)\n",
    "print(\"[sklearn] Explained variance:\", pca.explained_variance_)\n",
    "print(\"[sklearn] Explained variance ratio:\", pca.explained_variance_ratio_)\n",
    "\n",
    "# Ensure direction consistency (signs may differ)\n",
    "print(\"\\nDirection alignment check (abs dot with our v1, v2):\")\n",
    "print(\"|v1·comp1|:\", abs(vecs[:, 0] @ pca.components_[0]))\n",
    "print(\"|v2·comp2|:\", abs(vecs[:, 1] @ pca.components_[1]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
