{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fcac1f5e",
   "metadata": {},
   "source": [
    "# TP2: Logistic Regression with Gradient Descent\n",
    "\n",
    "By the end of this lab, students will be able to:\n",
    "\n",
    "- Understand the mathematical foundation of logistic regression.\n",
    "- Implement sigmoid function, cost function (cross-entropy), and gradient descent step by step in Python.\n",
    "- Apply logistic regression to univariate and multivariate classification tasks.\n",
    "- Visualize the decision boundary for 2D problems.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b129719",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4632eb15",
   "metadata": {},
   "source": [
    "## Part A- Univariate Logistic Regression\n",
    "\n",
    "### Step 1- Data Preparation\n",
    "\n",
    "- Provide a binary classification dataset (e.g., exam score vs pass/fail).\n",
    "- Students load data (NumPy/Pandas).\n",
    "- Task: Scatter plot the data, label classes differently.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef0372d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example dataset: exam score vs admitted (1) / not admitted (0)\n",
    "X = np.array([30, 40, 50, 60, 70, 80, 90])\n",
    "y = np.array([0, 0, 0, 1, 1, 1, 1])\n",
    "\n",
    "plt.scatter(X, y, c=y, cmap=\"bwr\", edgecolors=\"k\")\n",
    "plt.xlabel(\"Exam Score\")\n",
    "plt.ylabel(\"Admitted (1) / Not Admitted (0)\")\n",
    "plt.title(\"Training Data\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4bc5aa3",
   "metadata": {},
   "source": [
    "### Step 3- Sigmoid Function\n",
    "\n",
    "The sigmoid function is defined as:\n",
    "\n",
    "$$\n",
    "\\sigma(z) = \\frac{1}{1 + e^{-z}}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffab252d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "\n",
    "# Test\n",
    "print(sigmoid(0))  # expected 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99fab5d",
   "metadata": {},
   "source": [
    "### Step 3- Hypothesis Function\n",
    "\n",
    "The hypothesis function for logistic regression is given by:\n",
    "\n",
    "$$\n",
    "h_\\theta(x) = \\sigma(\\theta_0 + \\theta_1 x)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be500cfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hypothesis(theta0, theta1, x):\n",
    "    z = theta0 + theta1 * x\n",
    "    return sigmoid(z)\n",
    "\n",
    "\n",
    "# Test\n",
    "print(hypothesis(0, 1, 0))  # expected 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e14a68",
   "metadata": {},
   "source": [
    "### Step 4- Cost Function (Cross-Entropy)\n",
    "\n",
    "The cost function for logistic regression is defined as:\n",
    "\n",
    "$$\n",
    "J(\\theta_0, \\theta_1) = -\\frac{1}{m} \\sum_{i=1}^{m} \\left[ y^{(i)} \\log(h_\\theta(x^{(i)})) + (1 - y^{(i)}) \\log(1 - h_\\theta(x^{(i)})) \\right]\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c33e10db",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(theta0, theta1, X, y):\n",
    "    m = len(y)\n",
    "    predictions = hypothesis(theta0, theta1, X)\n",
    "    cost = -(1 / m) * np.sum(\n",
    "        y * np.log(predictions) + (1 - y) * np.log(1 - predictions)\n",
    "    )\n",
    "    return cost\n",
    "\n",
    "\n",
    "# Test\n",
    "print(compute_cost(0, 0, X, y))  # expected around 0.693"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f443447",
   "metadata": {},
   "source": [
    "### Step 5- Gradient Descent Update Rule\n",
    "\n",
    "The parameters are updated using the following rules:\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j - \\alpha \\frac{1}{m} \\sum_{i=1}^{m} \\left( h_\\theta(x^{(i)}) - y^{(i)} \\right) x_j^{(i)}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "868d061e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X, y, theta0, theta1, alpha, iterations):\n",
    "    m = len(y)\n",
    "    cost_history = []\n",
    "\n",
    "    for _ in range(iterations):\n",
    "        predictions = hypothesis(theta0, theta1, X)\n",
    "\n",
    "        error = predictions - y\n",
    "        theta0 -= alpha * (1 / m) * np.sum(error)\n",
    "        theta1 -= alpha * (1 / m) * np.sum(error * X)\n",
    "        cost = compute_cost(theta0, theta1, X, y)\n",
    "        cost_history.append(cost)\n",
    "\n",
    "    return theta0, theta1, cost_history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44878ad",
   "metadata": {},
   "source": [
    "### Step 6- Training and Visualization\n",
    "\n",
    "- Initialize θ.\n",
    "- Run gradient descent loop.\n",
    "- Plot cost over iterations.\n",
    "- Plot decision boundary on the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c8eac17",
   "metadata": {},
   "outputs": [],
   "source": [
    "theta0, theta1, cost_history = gradient_descent(\n",
    "    X, y, theta0=0, theta1=0, alpha=0.001, iterations=1000\n",
    ")\n",
    "\n",
    "print(\"Final theta0:\", theta0)\n",
    "print(\"Final theta1:\", theta1)\n",
    "\n",
    "# Plot cost convergence\n",
    "plt.plot(cost_history)\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.ylabel(\"Cost J\")\n",
    "plt.title(\"Cost Function Convergence\")\n",
    "plt.show()\n",
    "\n",
    "# Plot decision boundary\n",
    "plt.scatter(X, y, c=y, cmap=\"bwr\", edgecolors=\"k\")\n",
    "x_vals = np.linspace(min(X), max(X), 100)\n",
    "plt.plot(x_vals, hypothesis(theta0, theta1, x_vals), color=\"green\")\n",
    "plt.xlabel(\"Exam Score\")\n",
    "plt.ylabel(\"Admitted Probability\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ee96d0d",
   "metadata": {},
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89be06b0",
   "metadata": {},
   "source": [
    "## Part B- Multivariate Logistic Regression\n",
    "\n",
    "### Step 1- Data Preparation\n",
    "\n",
    "- Use dataset with multiple features (e.g., admission dataset: exam1, exam2 → admitted).\n",
    "- Normalize features.\n",
    "- Add bias column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc98d7f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example dataset (Exam1, Exam2 -> Admitted)\n",
    "data = {\n",
    "    \"exam1\": [34, 78, 50, 85, 60, 45, 82, 70],\n",
    "    \"exam2\": [78, 45, 60, 85, 75, 52, 43, 95],\n",
    "    \"admitted\": [0, 0, 0, 1, 1, 0, 1, 1],\n",
    "}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "X = df[[\"exam1\", \"exam2\"]].values\n",
    "y = df[\"admitted\"].values\n",
    "\n",
    "# Feature normalization\n",
    "X = (X - X.mean(axis=0)) / X.std(axis=0)\n",
    "\n",
    "# Add intercept column\n",
    "X = np.c_[np.ones(X.shape[0]), X]\n",
    "\n",
    "print(\"X shape:\", X.shape)\n",
    "print(\"y shape:\", y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aebeda2c",
   "metadata": {},
   "source": [
    "### Step 2- Sigmoid Function + Hypothesis Function\n",
    "\n",
    "The hypothesis function for multivariate logistic regression is given by:\n",
    "\n",
    "$$\n",
    "h_\\theta(x) = \\sigma(\\theta^T x)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d953c5d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "\n",
    "def hypothesis(theta, X):\n",
    "    z = np.dot(X, theta)\n",
    "    return sigmoid(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41a4191a",
   "metadata": {},
   "source": [
    "### Step 3- Vectorized Cost Function\n",
    "\n",
    "The cost function in vectorized form is:\n",
    "\n",
    "$$\n",
    "J(\\theta) = -\\frac{1}{m} \\left[ y^T \\log(h_\\theta(X)) + (1 - y)^T \\log(1 - h_\\theta(X)) \\right]\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f71cb165",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(theta, X, y):\n",
    "    m = len(y)\n",
    "    predictions = hypothesis(theta, X)\n",
    "    cost = -(1 / m) * np.sum(\n",
    "        y * np.log(predictions) + (1 - y) * np.log(1 - predictions)\n",
    "    )\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf63a86b",
   "metadata": {},
   "source": [
    "### Step 4- Vectorized Gradient Descent\n",
    "\n",
    "The gradient descent update rule in vectorized form is:\n",
    "\n",
    "$$\n",
    "\\theta := \\theta - \\alpha \\frac{1}{m} X^T (h_\\theta(X) - y)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afc7aa5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X, y, theta, alpha, iterations):\n",
    "    m = len(y)\n",
    "    cost_history = []\n",
    "\n",
    "    for _ in range(iterations):\n",
    "        predictions = hypothesis(theta, X)\n",
    "\n",
    "        error = predictions - y\n",
    "        theta -= alpha * (1 / m) * np.dot(X.T, error)\n",
    "        cost = compute_cost(theta, X, y)\n",
    "        cost_history.append(cost)\n",
    "\n",
    "    return theta, cost_history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4d012e0",
   "metadata": {},
   "source": [
    "### Step 5- Training and Convergence\n",
    "\n",
    "- Run gradient descent, track cost.\n",
    "- Plot cost vs iterations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a23d095",
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = np.zeros(X.shape[1])  # initialize\n",
    "theta, cost_history = gradient_descent(X, y, theta, alpha=0.1, iterations=1000)\n",
    "\n",
    "print(\"Learned parameters:\", theta)\n",
    "\n",
    "plt.plot(cost_history)\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.ylabel(\"Cost J\")\n",
    "plt.title(\"Multivariate Cost Convergence\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c958ec",
   "metadata": {},
   "source": [
    "### Step 6: Decision Boundary & Comparison\n",
    "\n",
    "- For 2D features, plot decision boundary (line separating classes).\n",
    "- Compare with Scikit-Learn’s `LogisticRegression`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a72c2bc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Fit using sklearn\n",
    "model = LogisticRegression()\n",
    "model.fit(df[[\"exam1\", \"exam2\"]], df[\"admitted\"])\n",
    "\n",
    "print(\"Sklearn intercept:\", model.intercept_)\n",
    "print(\"Sklearn coefficients:\", model.coef_)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
