{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2d86cfee",
   "metadata": {},
   "source": [
    "# TP1: Linear Regression with Gradient Descent\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f122978f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5835855e",
   "metadata": {},
   "source": [
    "## Part A- Univariate Linear Regression\n",
    "\n",
    "### Step 1- Data Preparation\n",
    "\n",
    "We start with a simple dataset relating house size to price. The goal is to fit a line to predict price from size.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee3e297a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a simple Dataset (house size vs. price)\n",
    "x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])  # house size in 100 m²\n",
    "y = np.array([3, 4, 2, 5, 6, 7, 8, 9, 10])  # house price in $100k\n",
    "\n",
    "plt.scatter(x, y, color=\"blue\", marker=\"o\")\n",
    "plt.xlabel(\"House Size (100 m²)\")\n",
    "plt.ylabel(\"House Price ($100k)\")\n",
    "plt.title(\"Training Data: House Size vs. Price\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c501af1",
   "metadata": {},
   "source": [
    "### Step 2- Hypothesis Function\n",
    "\n",
    "Defines the model: a straight line with parameters $\\theta_0$ (intercept) and $\\theta_1$ (slope).\n",
    "\n",
    "$$\n",
    "h_\\theta(x) = \\theta_0 + \\theta_1 x\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f8d2bd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hypothesis(x, theta0, theta1):\n",
    "    return theta0 + theta1 * x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08a786b6",
   "metadata": {},
   "source": [
    "### Step 3- Cost Function (Mean Squared Error)\n",
    "\n",
    "Measures how well the model fits the data. Lower cost means better fit.\n",
    "\n",
    "$$\n",
    "J(\\theta_0, \\theta_1) = \\frac{1}{2m} \\sum_{i=1}^{m} (h_\\theta(x^{(i)}) - y^{(i)})^2\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68fb6483",
   "metadata": {},
   "outputs": [],
   "source": [
    "def computeCost(x, y, theta0, theta1):\n",
    "    m = len(y)\n",
    "    predictions = hypothesis(x, theta0, theta1)\n",
    "    cost = (1 / (2 * m)) * np.sum((predictions - y) ** 2)\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e59e7f2",
   "metadata": {},
   "source": [
    "### Step 4- Gradient Descent Update Rule\n",
    "\n",
    "Algorithm to minimize the cost function by updating $\\theta_0$ and $\\theta_1$ iteratively.\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j - \\alpha \\frac{\\partial J(\\theta_0, \\theta_1)}{\\partial \\theta_j}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40f81b57",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(x, y, theta0, theta1, alpha, iterations):\n",
    "    m = len(y)\n",
    "    cost_history = []\n",
    "\n",
    "    for _ in range(iterations):\n",
    "        predictions = hypothesis(x, theta0, theta1)\n",
    "        error = predictions - y\n",
    "\n",
    "        # Compute gradient using all data points\n",
    "        theta0 -= alpha * (1 / m) * np.sum(error)\n",
    "        theta1 -= alpha * (1 / m) * np.sum(error * x)\n",
    "\n",
    "        # Track cost after each iteration\n",
    "        cost_history.append(computeCost(x, y, theta0, theta1))\n",
    "\n",
    "    return theta0, theta1, cost_history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f839619a",
   "metadata": {},
   "source": [
    "### Step 5- Train the Model\n",
    "\n",
    "Run gradient descent to find the best parameters for the linear model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f02ea2ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "theta0 = 0.0\n",
    "theta1 = 0.0\n",
    "alpha = 0.01\n",
    "iterations = 1000\n",
    "\n",
    "theta0, theta1, cost_history = gradient_descent(x, y, theta0, theta1, alpha, iterations)\n",
    "print(\"Final theta0:\", theta0)\n",
    "print(\"Final theta1:\", theta1)\n",
    "\n",
    "# Plot cost convergence\n",
    "plt.plot(cost_history)\n",
    "plt.xlabel(\"Iterations\")\n",
    "plt.ylabel(\"Cost J(θ)\")\n",
    "plt.title(\"Cost Function Convergence\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6baa0cfd",
   "metadata": {},
   "source": [
    "### Step 6- Results & Visualization\n",
    "\n",
    "Visualize the fitted line and check how well it matches the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01183c89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot Regression Line\n",
    "plt.scatter(x, y, color=\"blue\", marker=\"o\", label=\"Training Data\")\n",
    "plt.plot(x, hypothesis(x, theta0, theta1), color=\"red\", label=\"Regression Line\")\n",
    "plt.xlabel(\"House Size (100 m²)\")\n",
    "plt.ylabel(\"House Price ($100k)\")\n",
    "plt.title(\"Linear Regression Fit\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "855b9390",
   "metadata": {},
   "source": [
    "## Part B- Multivariate Linear Regression\n",
    "\n",
    "Now we use more than one feature (e.g., size and number of rooms) to predict price.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb0d7923",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example dataset (size, rooms -> price)\n",
    "data = {\n",
    "    \"size\": [2100, 1600, 2400, 1416, 3000],\n",
    "    \"rooms\": [3, 2, 4, 2, 4],\n",
    "    \"price\": [400, 330, 369, 232, 540],\n",
    "}\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "x = df[[\"size\", \"rooms\"]].values\n",
    "y = df[\"price\"].values\n",
    "\n",
    "# Feature Normalization\n",
    "x = (x - np.mean(x, axis=0)) / np.std(x, axis=0)\n",
    "\n",
    "# Add intercept column\n",
    "x = np.c_[np.ones(x.shape[0]), x]\n",
    "\n",
    "print(\"X shape:\", x.shape)\n",
    "print(\"y shape:\", y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95318de9",
   "metadata": {},
   "source": [
    "### Step 2- Hypothesis (Vectorized)\n",
    "\n",
    "The model predicts using a vector of parameters and features: $h_\\theta(X) = X\\theta$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "492fa3cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hypothesis(x, theta):\n",
    "    return x.dot(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c06e358",
   "metadata": {},
   "source": [
    "### Step 3- Cost Function (Vectorized)\n",
    "\n",
    "Measures the fit for all features and parameters together.\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\frac{1}{2m} (X\\theta - y)^T (X\\theta - y)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e77c1fd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(x, y, theta):\n",
    "    m = len(y)\n",
    "    predictions = hypothesis(x, theta)\n",
    "    error = predictions - y\n",
    "    cost = (1 / (2 * m)) * np.dot(error.T, error)\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b198987",
   "metadata": {},
   "source": [
    "### Step 4- Gradient Descent Update Rule (Vectorized)\n",
    "\n",
    "Update all parameters at once using the gradient of the cost function.\n",
    "\n",
    "$$\n",
    "\\theta := \\theta - \\alpha \\frac{1}{m} X^T (X\\theta - y)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e336d11c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(x, y, theta, alpha, iterations):\n",
    "    m = len(y)\n",
    "    cost_history = []\n",
    "\n",
    "    for _ in range(iterations):\n",
    "        predictions = hypothesis(x, theta)\n",
    "        error = predictions - y\n",
    "        gradient = (1 / m) * np.dot(x.T, error)\n",
    "        theta = theta - alpha * gradient\n",
    "        cost_history.append(compute_cost(x, y, theta))\n",
    "\n",
    "    return theta, cost_history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afff1b17",
   "metadata": {},
   "source": [
    "### Step 5- Train and Compare\n",
    "\n",
    "Train the multivariate model and observe cost reduction over iterations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "463a07a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = np.zeros(x.shape[1])  # Initialize theta\n",
    "alpha = 0.01\n",
    "iterations = 1000\n",
    "theta, cost_history = gradient_descent(x, y, theta, alpha, iterations)\n",
    "\n",
    "print(\"Learned Parameters:\", theta)\n",
    "\n",
    "plt.plot(cost_history)\n",
    "plt.xlabel(\"Iterations\")\n",
    "plt.ylabel(\"Cost J(θ)\")\n",
    "plt.title(\"MultiVariate Cost Convergence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e1304be",
   "metadata": {},
   "source": [
    "### Step 6- Compare with Scikit-Learn\n",
    "\n",
    "Compare your implementation with a standard library to check correctness.\n",
    "\n",
    "_note: the values would not look close because we did not normalize the data for the Scikit-learn model_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56bfbd4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "model = LinearRegression()\n",
    "model.fit(df[[\"size\", \"rooms\"]], df[\"price\"])\n",
    "\n",
    "print(\"Scikit-Learn Coefficients:\", model.intercept_, model.coef_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8989513f",
   "metadata": {},
   "source": [
    "## Part C- Polynomial Regression\n",
    "\n",
    "Now we extend linear regression to polynomial regression to capture non-linear relationships. We'll use a dataset with features: size, number of bedrooms, number of floors, and age to predict house price.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3927048e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example dataset (size, bedrooms, floors, age -> price)\n",
    "data_poly = {\n",
    "    \"size\": [2104, 1416, 1534, 852],\n",
    "    \"bedrooms\": [5, 3, 3, 2],\n",
    "    \"floors\": [1, 2, 2, 1],\n",
    "    \"age\": [45, 40, 30, 36],\n",
    "    \"price\": [460, 232, 315, 178],\n",
    "}\n",
    "\n",
    "df_poly = pd.DataFrame(data_poly)\n",
    "\n",
    "X_poly = df_poly[[\"size\", \"bedrooms\", \"floors\", \"age\"]].values\n",
    "y_poly = df_poly[\"price\"].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "659359a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create polynomial features (degree 2)\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "poly = PolynomialFeatures(degree=2, include_bias=False)\n",
    "X_poly_expanded = poly.fit_transform(X_poly)\n",
    "\n",
    "print(\"Original shape:\", X_poly.shape)\n",
    "print(\"Expanded shape:\", X_poly_expanded.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbc25a14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit polynomial regression model\n",
    "model_poly = LinearRegression()\n",
    "model_poly.fit(X_poly_expanded, y_poly)\n",
    "\n",
    "print(\"Polynomial Regression Coefficients:\")\n",
    "print(model_poly.intercept_, model_poly.coef_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee17844f",
   "metadata": {},
   "source": [
    "### Visualization and Prediction Example\n",
    "\n",
    "You can visualize the fit for one feature (e.g., size) or show predictions for new data points.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "639a48ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict price for a new house (example)\n",
    "new_house = np.array([[2000, 3, 2, 10]])\n",
    "new_house_poly = poly.transform(new_house)\n",
    "predicted_price = model_poly.predict(new_house_poly)\n",
    "print(f\"Predicted price for house {new_house[0]}: ${predicted_price[0]:.2f}k\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
