{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Feature Engineering for NLP Tasks\n",
    "### Using Scikit-Learn and Gensim\n",
    "\n",
    "Feature engineering is a critical step in Natural Language Processing (NLP). It involves transforming raw text data into structured numerical features that machine learning algorithms can understand.\n",
    "\n",
    "This tutorial guides you through two fundamental approaches to NLP feature engineering:\n",
    "1. **Frequency-Based Methods** (using `scikit-learn`): Bag-of-Words, Stopword filtering, N-grams, and TF-IDF.\n",
    "2. **Prediction-Based Methods / Dense Word Embeddings** (using `gensim`): Word2Vec (Pre-trained and Custom) and FastText."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 1: Frequency-Based Vectorization (Scikit-Learn)\n",
    "\n",
    "### 1. Count Vectorization (Bag-of-Words)\n",
    "The Bag-of-Words (BoW) model represents text by counting the occurrences of each word within a document. It completely disregards grammar, word order, and context, focusing entirely on raw term frequency.\n",
    "\n",
    "We use scikit-learn's `CountVectorizer` to construct a vocabulary of all unique tokens across our corpus and map each sentence to its corresponding frequency vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "sents = [\n",
    "    'The dog barked in the park on another dog',\n",
    "    'The owner of the dog put him on the leash since he barked.',\n",
    "    'My dog is barking and chasing its tail.'\n",
    "]\n",
    "\n",
    "cv = CountVectorizer()\n",
    "X = cv.fit_transform(sents)\n",
    "X_array = X.toarray()\n",
    "\n",
    "print(\"Vocabulary Features:\")\n",
    "print(sorted(cv.vocabulary_.keys()))\n",
    "print(\"\\nDocument-Term Matrix Grid Layout:\")\n",
    "print(X_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Count Vectorization with Stopword Filtering\n",
    "Stopwords are highly frequent words (such as \"the\", \"is\", \"in\", \"on\") that provide grammatical structure but carry minimal semantic value for downstream tasks like classification or clustering. Filtering them out drastically reduces dimensionality and cleans up noisy features. We can pass `stop_words='english'` into the vectorizer to apply an internal English stopword list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "sents = [\n",
    "    'The dog barked in the park on another dog',\n",
    "    'The owner of the dog put him on the leash since he barked.',\n",
    "    'My dog is barking and chasing its tail.'\n",
    "]\n",
    "\n",
    "cv_stopwords = CountVectorizer(stop_words='english')\n",
    "X_stopwords = cv_stopwords.fit_transform(sents).toarray()\n",
    "\n",
    "print(\"Filtered Vocabulary (Stopwords Removed):\")\n",
    "print(sorted(cv_stopwords.vocabulary_.keys()))\n",
    "print(\"\\nFiltered Document-Term Matrix:\")\n",
    "print(X_stopwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. N-Gram Count Vectorization\n",
    "To counter the complete loss of local context inherent in a basic unigram model (1-gram), we introduce **N-grams**. N-grams group consecutive sequences of tokens together. \n",
    "* `ngram_range=(1,1)` extracts single words (unigrams).\n",
    "* `ngram_range=(1,2)` extracts both single words and word pairs (bigrams), preserving expressions like \"dog barked\" or \"chasing tail\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "sents = [\n",
    "    'The dog barked in the park on another dog',\n",
    "    'The owner of the dog put him on the leash since he barked.',\n",
    "    'My dog is barking and chasing its tail.'\n",
    "]\n",
    "\n",
    "cv_ngram = CountVectorizer(ngram_range=(1, 2), stop_words='english')\n",
    "X_ngram = cv_ngram.fit_transform(sents).toarray()\n",
    "\n",
    "print(\"Expanded N-Gram Vocabulary Space:\")\n",
    "print(sorted(cv_ngram.vocabulary_.keys()))\n",
    "print(\"\\nN-Gram Matrix Shape:\", X_ngram.shape)\n",
    "print(X_ngram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. TF-IDF Vectorization\n",
    "While count arrays track presence, they can over-emphasize words that are naturally common across an entire subject field. **TF-IDF (Term Frequency-Inverse Document Frequency)** fixes this by applying a balanced scaling calculation:\n",
    "$$\\text{TF-IDF}(t, d) = \\text{TF}(t, d) \\times \\text{IDF}(t)$$\n",
    "\n",
    "Where **TF** evaluates frequency inside a singular document, and **IDF** suppresses tokens that appear redundantly across the broader corpus. This accurately brings out structural signature terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import pandas as pd\n",
    "\n",
    "sents = [\n",
    "    'The dog barked in the park on another dog',\n",
    "    'The owner of the dog put him on the leash since he barked.',\n",
    "    'My dog is barking and chasing its tail.'\n",
    "]\n",
    "\n",
    "tfidf_vec = TfidfVectorizer(stop_words='english')\n",
    "X_tfidf = tfidf_vec.fit_transform(sents)\n",
    "\n",
    "print(\"Raw Output Feature Weight Metrics Grid:\")\n",
    "print(X_tfidf.toarray())\n",
    "\n",
    "feature_names = tfidf_vec.get_feature_names_out()\n",
    "df_tfidf = pd.DataFrame(X_tfidf[0].T.todense(), index=feature_names, columns=[\"TF-IDF\"])\n",
    "df_tfidf = df_tfidf.sort_values('TF-IDF', ascending=False)\n",
    "\n",
    "print(\"\\nRanked Terms within Document 1:\")\n",
    "print(df_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 2: Prediction-Based Continuous Dense Vectors (Gensim)\n",
    "\n",
    "Unlike sparse frequency configurations, vector embeddings capture semantic similarities using low-dimensional, continuous spatial real numbers.\n",
    "\n",
    "### 1. Utilizing Pre-Trained Word2Vec Models\n",
    "Gensim allows you to import heavy pre-trained models (such as the standard 300-dimension Google News corpus configuration) to map high-fidelity universal associations instantly without training overhead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim import models\n",
    "\n",
    "# Code template structure showcasing how to extract relationships using massive binaries:\n",
    "\"\"\"\n",
    "w2v_pretrained = models.KeyedVectors.load_word2vec_format(\n",
    "    './GoogleNews-vectors-negative300.bin', \n",
    "    binary=True\n",
    ")\n",
    "healthy_vector = w2v_pretrained['healthy']\n",
    "print(w2v_pretrained.most_similar('happy'))\n",
    "\"\"\"\n",
    "print(\"[Info] Pre-trained blueprint successfully structured for external usage.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Training a Custom Word2Vec Model\n",
    "When work covers hyper-specific vocabulary or unique domains, building specialized contexts becomes crucial. Gensim expects its input data pattern to follow a **list of tokenized sentence lists**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim import models\n",
    "\n",
    "sents = [\n",
    "    'The dog barked in the park on another dog',\n",
    "    'The owner of the dog put him on the leash since he barked.',\n",
    "    'My dog is barking and chasing its tail.'\n",
    "]\n",
    "\n",
    "tokenized_corpus = [sent.lower().replace('.', '').split() for sent in sents]\n",
    "\n",
    "custom_w2v = models.Word2Vec(\n",
    "    sentences=tokenized_corpus, \n",
    "    min_count=1, \n",
    "    vector_size=300, \n",
    "    workers=4\n",
    ")\n",
    "\n",
    "dog_vector = custom_w2v.wv['dog']\n",
    "print(\"Sample continuous coordinates slice for token 'dog':\")\n",
    "print(dog_vector[:5])\n",
    "\n",
    "print(\"\\nTop similar terms to 'barked':\")\n",
    "print(custom_w2v.wv.most_similar('barked'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. FastText (Subword-Level Handling)\n",
    "A main flaw of Word2Vec is its inability to manage completely unseen, unknown text (**Out-Of-Vocabulary / OOV** words). FastText resolves this issue by decomposing words into internal **Character N-Grams**.\n",
    "\n",
    "Because meaning is aggregated directly from smaller internal subword pieces, it can construct accurate fallback representations for modified or typo-ridden words based on their structural components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim import models\n",
    "\n",
    "sents = [\n",
    "    'The dog barked in the park on another dog',\n",
    "    'The owner of the dog put him on the leash since he barked.',\n",
    "    'My dog is barking and chasing its tail.'\n",
    "]\n",
    "\n",
    "tokenized_corpus = [sent.lower().replace('.', '').split() for sent in sents]\n",
    "\n",
    "fasttext_model = models.FastText(\n",
    "    sentences=tokenized_corpus, \n",
    "    vector_size=100, \n",
    "    window=5, \n",
    "    min_count=1, \n",
    "    workers=4, \n",
    "    sg=1\n",
    ")\n",
    "\n",
    "print(\"FastText vector for target 'dog' (first 5 elements):\")\n",
    "print(fasttext_model.wv['dog'][:5])\n",
    "\n",
    "print(\"\\nEvaluating semantics matching via subwords for 'barked':\")\n",
    "print(fasttext_model.wv.most_similar('barked'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
