{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ec69b768",
   "metadata": {},
   "source": [
    "# Introduction to NLTK (Natural Language Toolkit)\n",
    "\n",
    "**NLTK** is one of the most popular libraries for Natural Language Processing in Python. It provides easy-to-use interfaces to over 50 corpora and lexical resources, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and more.\n",
    "\n",
    "### Getting Started\n",
    "Before using NLTK's advanced features (like Lemmatization or Sentence Tokenization), you must ensure the library is installed and the necessary data packages are downloaded to your machine.\n",
    "\n",
    "1. **Installation:**\n",
    "   ```bash\n",
    "   pip install nltk\n",
    "   ```\n",
    "\n",
    "2. **Downloading Resources:**\n",
    "   NLTK requires specific datasets to function. Run the following command in a code cell to download the basics used in this notebook:\n",
    "   ```python\n",
    "   import nltk\n",
    "   nltk.download('punkt') # For tokenization\n",
    "   nltk.download('wordnet') # For lemmatization\n",
    "   ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee758969",
   "metadata": {},
   "source": [
    "# NLP Foundations: Text Preprocessing and Tokenization\n",
    "\n",
    "This notebook demonstrates essential techniques for preparing raw text for Natural Language Processing. We will cover:\n",
    "1. **Tokenization** (Word and Sentence level)\n",
    "2. **Normalization** (Lowercasing and Punctuation removal)\n",
    "3. **Vocabulary Building**\n",
    "4. **Morphological Processing** (Stemming and Lemmatization)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a80d642",
   "metadata": {},
   "source": [
    "## 1. Basic Word Tokenization\n",
    "The simplest way to tokenize text is using Python's built-in `.split()` method, which splits a string by all whitespace (tabs, new lines, and multiple spaces).\n",
    "Note how punctuation remains attached to the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d427942f-60ef-4bc2-b6c2-34b04fabd15c",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = \"They picnicked by the pool, then they lay back on the grass and looked at the stars.\"\n",
    "\n",
    "words = corpus.split()\n",
    "print(\"The count of corpus words:\", len(words))\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef72eabf",
   "metadata": {},
   "source": [
    "### Regex-based Tokenization\n",
    "To treat punctuation as separate tokens, we can use Regular Expressions (`re`). The pattern `\\w+|[^\\w\\s]` matches sequences of alphanumeric characters OR single non-word/non-space characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "940c5f57-6cab-4b33-aefb-7fb264caceee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "corpus = \"They picnicked by the pool, then they lay back on the grass and looked at the stars.\"\n",
    "\n",
    "tokens = re.findall(r\"\\w+|[^\\w\\s]\", corpus)\n",
    "print(\"The count of corpus words with punctuations:\", len(tokens))\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "897e90d7",
   "metadata": {},
   "source": [
    "## 2. Building a Vocabulary\n",
    "A **Vocabulary** is the set of unique tokens in a corpus. Using a Python `set` automatically removes duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfb797a2-f65f-44eb-9f5f-6d2b2669152e",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = \"They picnicked by the pool, then they lay back on the grass and looked at the stars.\"\n",
    "\n",
    "words = corpus.split()\n",
    "vocabulary = set(words)\n",
    "print(vocabulary)\n",
    "print(\"The size of vocabulary is:\", len(vocabulary))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "869012e8",
   "metadata": {},
   "source": [
    "### Case Normalization\n",
    "Notice that \"They\" and \"they\" are treated as different words above. Lowercasing the text before processing ensures that the model treats them as the same semantic unit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "086b45b7-98ee-4714-853a-9257470206e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = \"They picnicked by the pool, then they lay back on the grass and looked at the stars.\"\n",
    "\n",
    "words = corpus.lower().split()\n",
    "vocabulary = set(words)\n",
    "print(vocabulary)\n",
    "print(\"The size of vocabulary after using lowercasing is:\", len(vocabulary))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62342622",
   "metadata": {},
   "source": [
    "## 3. Punctuation Removal\n",
    "In some tasks (like simple sentiment analysis), punctuation isn't needed. We can use a translation table to strip all characters defined in `string.punctuation`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8dbb6fd-1f77-464d-a1b3-95073727557f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "\n",
    "def remove_punctuation(text):\n",
    "    return text.translate(str.maketrans(\"\", \"\", string.punctuation))\n",
    "\n",
    "\n",
    "corpus = \"They picnicked by the pool, then they lay back on the grass and looked at the stars.\"\n",
    "clean_corpus = remove_punctuation(corpus)\n",
    "print(\"Before:\", corpus)\n",
    "print(\"After:\", clean_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61bf5f34",
   "metadata": {},
   "source": [
    "## 4. Stemming vs. Lemmatization\n",
    "* **Stemming:** A heuristic process that chops off the ends of words (e.g., \"reporting\" becomes \"report\"). It is fast but can result in non-dictionary words.\n",
    "* **Lemmatization:** Uses vocabulary and morphological analysis to return the base or dictionary form of a word (the lemma)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faced2b3-e241-4d6e-99a8-5804df2ab7d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "words = [\"emailing\", \"replying\", \"reporting\", \"presentations\", \"meeting\", \"scheduling\"]\n",
    "porter = nltk.PorterStemmer()\n",
    "stems = [porter.stem(word) for word in words]\n",
    "print(\"Words after stemming\")\n",
    "print(stems)\n",
    "\n",
    "wnl = nltk.WordNetLemmatizer()\n",
    "lemmas = [wnl.lemmatize(word) for word in words]\n",
    "print(\"Words after lemmatizations\")\n",
    "print(lemmas)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ff46ce4",
   "metadata": {},
   "source": [
    "## 5. Advanced NLTK Tokenizers\n",
    "The `TreebankWordTokenizer` uses standard conventions (like splitting contractions) used in the Penn Treebank corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f4377f8-a6c3-42ce-8857-f8c5a62e2491",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize.treebank import TreebankWordTokenizer\n",
    "\n",
    "tokenizer = TreebankWordTokenizer()\n",
    "corpus = \"They picnicked by the pool, then they lay back on the grass and looked at the stars.\"\n",
    "tokens = tokenizer.tokenize(corpus)\n",
    "print(\"tokens\")\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "641a4c68",
   "metadata": {},
   "source": [
    "## 6. Sentence Tokenization\n",
    "Breaking a paragraph into individual sentences is crucial for many NLP pipelines. NLTK's `sent_tokenize` is pre-trained to handle abbreviations (like \"p.m.\" and \"U.S.A.\") so they aren't mistaken for sentence endings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2ffb32b-0779-427d-9228-a4f457b5d650",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import PunktTokenizer\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "corpus = \"\"\"He plays well. She said \"Peter is back!\". I'm going at 5 p.m. to U.S.A.\"\"\"\n",
    "sent_tokenizer = PunktTokenizer()\n",
    "sentences = sent_tokenizer.tokenize(corpus)\n",
    "print(sentences)\n",
    "\n",
    "sentences = sent_tokenize(corpus)\n",
    "print(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "333c6284",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5cdebd7",
   "metadata": {},
   "source": [
    "## 🧪 Student Exercise: The Preprocessing Pipeline\n",
    "\n",
    "**Objective:** Create a pipeline that processes a multi-sentence corpus into a clean list of lemmas.\n",
    "\n",
    "**Tasks:**\n",
    "1. **Sentence Tokenization:** Split the provided `exercise_corpus` into individual sentences.\n",
    "2. **Normalization:** For each sentence, remove punctuation and convert all text to lowercase.\n",
    "3. **Word Tokenization:** Split the cleaned sentences into individual words (tokens).\n",
    "4. **Lemmatization:** Reduce each word to its base form (lemma) using the `WordNetLemmatizer`.\n",
    "5. **Vocabulary:** Print the final size of your unique vocabulary.\n",
    "\n",
    "**Challenge:** Compare your lemmatized results with a stemmed version. Which one makes more sense for a human reader?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exercise-cell",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import string\n",
    "from nltk.tokenize import sent_tokenize, TreebankWordTokenizer\n",
    "from nltk.stem import WordNetLemmatizer, PorterStemmer\n",
    "\n",
    "# Sample Corpus for the exercise\n",
    "exercise_corpus = \"\"\"\n",
    "The students are studying hard for their NLP exams. \n",
    "They are practicing tokenization, stemming, and lemmatization techniques! \n",
    "Does practicing these exercises help them understand the foundations? \n",
    "Yes, it definitely helps.\n",
    "\"\"\"\n",
    "\n",
    "# Tools\n",
    "tokenizer = TreebankWordTokenizer()\n",
    "wnl = WordNetLemmatizer()\n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "tokens = tokenizer.tokenize(exercise_corpus)  # we tokenize the corpus\n",
    "vocabulary = set(tokens)  # we create a vocabulary set from the tokens\n",
    "\n",
    "print(\"Vocabulary size before cleaning: \", len(vocabulary))\n",
    "print(\"Vocabulary before cleaning:\", vocabulary)\n",
    "\n",
    "lemmatized_vocabulary = set()\n",
    "stemmed_vocabulary = set()\n",
    "\n",
    "for sent in sent_tokenize(exercise_corpus):  # we iterate over each sentence\n",
    "    clean_sent = remove_punctuation(\n",
    "        sent.lower()\n",
    "    )  # we clean the sentence (lowercase + remove punctuation)\n",
    "    lemma_tokens = [\n",
    "        wnl.lemmatize(token) for token in tokenizer.tokenize(clean_sent)\n",
    "    ]  # we lemmatize the tokens\n",
    "    for lemma in lemma_tokens:\n",
    "        lemmatized_vocabulary.add(lemma)  # we add each lemma to the vocabulary\n",
    "        stemmed_vocabulary.add(\n",
    "            stemmer.stem(lemma)\n",
    "        )  # we stem each lemma and add to the vocabulary\n",
    "\n",
    "\n",
    "print(\"Lemmatized Vocabulary size: \", len(lemmatized_vocabulary))\n",
    "print(\"Lemmatized Vocabulary:\", lemmatized_vocabulary)\n",
    "print(\"Stemmed Vocabulary size: \", len(stemmed_vocabulary))\n",
    "print(\"Stemmed Vocabulary:\", stemmed_vocabulary)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
