{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLTK vs SpaCy: POS Tagging and NER Comparison Tutorial\n",
    "\n",
    "This notebook provides a hands-on comparison between NLTK and SpaCy for:\n",
    "- **Part-of-Speech (POS) Tagging**\n",
    "- **Named Entity Recognition (NER)**\n",
    "\n",
    "We'll explore different approaches including NLTK's HMM tagger and compare performance, accuracy, and ease of use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Installation and Setup\n",
    "\n",
    "First, let's install the required libraries and download necessary data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Installation commands (uncomment if needed)\n",
    "# !pip install nltk spacy pandas tabulate\n",
    "# !python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import spacy\n",
    "import pandas as pd\n",
    "from tabulate import tabulate\n",
    "import time\n",
    "from collections import Counter\n",
    "\n",
    "# Download required NLTK data\n",
    "nltk.download('punkt')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('maxent_ne_chunker_tab')\n",
    "nltk.download('words')\n",
    "nltk.download('treebank')\n",
    "nltk.download('universal_tagset')\n",
    "\n",
    "# Load SpaCy model\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "print(\"✓ Setup complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Sample Text for Analysis\n",
    "\n",
    "We'll use a sample text containing various entities and grammatical structures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text = \"\"\"\n",
    "Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in April 1976. \n",
    "The company is headquartered in Cupertino, California. In 2024, Apple released the iPhone 15 \n",
    "which quickly became popular worldwide. Microsoft and Google are major competitors in the \n",
    "technology industry. Tim Cook became CEO in 2011 and has led the company to unprecedented growth.\n",
    "\"\"\"\n",
    "\n",
    "print(\"Sample Text:\")\n",
    "print(sample_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Part-of-Speech (POS) Tagging\n",
    "\n",
    "### 3.1 NLTK POS Tagging (Default Tagger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenize and tag with NLTK's default tagger\n",
    "tokens_nltk = nltk.word_tokenize(sample_text)\n",
    "\n",
    "start_time = time.time()\n",
    "pos_tags_nltk = nltk.pos_tag(tokens_nltk)\n",
    "nltk_default_time = time.time() - start_time\n",
    "\n",
    "print(\"NLTK POS Tagging (Default Tagger):\")\n",
    "print(\"=\"*60)\n",
    "for i in range(0, min(20, len(pos_tags_nltk))):\n",
    "    word, tag = pos_tags_nltk[i]\n",
    "    print(f\"{word:15} -> {tag}\")\n",
    "    \n",
    "print(f\"\\nProcessing time: {nltk_default_time:.4f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 NLTK POS Tagging with HMM Tagger\n",
    "\n",
    "Let's train an HMM (Hidden Markov Model) tagger using the Treebank corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tag import hmm\n",
    "from nltk.corpus import treebank\n",
    "\n",
    "# Load training data from Treebank corpus\n",
    "print(\"Training HMM tagger on Treebank corpus...\")\n",
    "train_data = treebank.tagged_sents()[:3000]\n",
    "\n",
    "# Train HMM tagger\n",
    "trainer = hmm.HiddenMarkovModelTrainer()\n",
    "hmm_tagger = trainer.train_supervised(train_data)\n",
    "\n",
    "print(\"✓ HMM tagger trained successfully!\")\n",
    "\n",
    "# Tag with HMM tagger\n",
    "start_time = time.time()\n",
    "pos_tags_hmm = hmm_tagger.tag(tokens_nltk)\n",
    "nltk_hmm_time = time.time() - start_time\n",
    "\n",
    "print(\"\\nNLTK POS Tagging (HMM Tagger):\")\n",
    "print(\"=\"*60)\n",
    "for i in range(0, min(20, len(pos_tags_hmm))):\n",
    "    word, tag = pos_tags_hmm[i]\n",
    "    print(f\"{word:15} -> {tag}\")\n",
    "    \n",
    "print(f\"\\nProcessing time: {nltk_hmm_time:.4f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 SpaCy POS Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Process with SpaCy\n",
    "start_time = time.time()\n",
    "doc_spacy = nlp(sample_text)\n",
    "spacy_time = time.time() - start_time\n",
    "\n",
    "print(\"SpaCy POS Tagging:\")\n",
    "print(\"=\"*60)\n",
    "count = 0\n",
    "for token in doc_spacy:\n",
    "    if not token.is_space and count < 20:\n",
    "        print(f\"{token.text:15} -> {token.pos_:8} (Fine: {token.tag_})\")\n",
    "        count += 1\n",
    "    if count >= 20:\n",
    "        break\n",
    "\n",
    "print(f\"\\nProcessing time: {spacy_time:.4f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 POS Tagging Comparison Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create comparison for first 15 tokens\n",
    "comparison_data = []\n",
    "\n",
    "for i in range(min(15, len(tokens_nltk))):\n",
    "    token = tokens_nltk[i]\n",
    "    \n",
    "    # Get tags from each method\n",
    "    nltk_default_tag = pos_tags_nltk[i][1] if i < len(pos_tags_nltk) else \"N/A\"\n",
    "    nltk_hmm_tag = pos_tags_hmm[i][1] if i < len(pos_tags_hmm) else \"N/A\"\n",
    "    \n",
    "    # Find corresponding SpaCy token\n",
    "    spacy_tag = \"N/A\"\n",
    "    for spacy_token in doc_spacy:\n",
    "        if spacy_token.text == token and not spacy_token.is_space:\n",
    "            spacy_tag = spacy_token.pos_\n",
    "            break\n",
    "    \n",
    "    comparison_data.append({\n",
    "        'Token': token,\n",
    "        'NLTK Default': nltk_default_tag,\n",
    "        'NLTK HMM': nltk_hmm_tag,\n",
    "        'SpaCy': spacy_tag\n",
    "    })\n",
    "\n",
    "df_pos_comparison = pd.DataFrame(comparison_data)\n",
    "print(\"\\nPOS Tagging Comparison:\")\n",
    "print(tabulate(df_pos_comparison, headers='keys', tablefmt='grid', showindex=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Named Entity Recognition (NER)\n",
    "\n",
    "### 4.1 NLTK NER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import ne_chunk\n",
    "\n",
    "# Perform NER with NLTK\n",
    "start_time = time.time()\n",
    "ne_tree = ne_chunk(pos_tags_nltk)\n",
    "nltk_ner_time = time.time() - start_time\n",
    "\n",
    "print(\"NLTK Named Entities:\")\n",
    "print(\"=\"*60)\n",
    "\n",
    "nltk_entities = []\n",
    "for subtree in ne_tree:\n",
    "    if hasattr(subtree, 'label'):\n",
    "        entity_name = ' '.join([word for word, tag in subtree.leaves()])\n",
    "        entity_type = subtree.label()\n",
    "        nltk_entities.append((entity_name, entity_type))\n",
    "        print(f\"{entity_name:30} -> {entity_type}\")\n",
    "\n",
    "print(f\"\\nTotal entities found: {len(nltk_entities)}\")\n",
    "print(f\"Processing time: {nltk_ner_time:.4f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 SpaCy NER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract entities from SpaCy\n",
    "print(\"SpaCy Named Entities:\")\n",
    "print(\"=\"*60)\n",
    "\n",
    "spacy_entities = []\n",
    "for ent in doc_spacy.ents:\n",
    "    spacy_entities.append((ent.text, ent.label_))\n",
    "    print(f\"{ent.text:30} -> {ent.label_:12} ({spacy.explain(ent.label_)})\")\n",
    "\n",
    "print(f\"\\nTotal entities found: {len(spacy_entities)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Visualize SpaCy Entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "# Visualize entities (works in Jupyter)\n",
    "print(\"SpaCy Entity Visualization:\")\n",
    "displacy.render(doc_spacy, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 NER Comparison Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create NER comparison\n",
    "ner_comparison_data = []\n",
    "\n",
    "# Add NLTK entities\n",
    "for entity, entity_type in nltk_entities:\n",
    "    ner_comparison_data.append({\n",
    "        'Entity': entity,\n",
    "        'NLTK Type': entity_type,\n",
    "        'SpaCy Type': 'Not Found'\n",
    "    })\n",
    "\n",
    "# Match with SpaCy entities\n",
    "for entity, entity_type in spacy_entities:\n",
    "    found = False\n",
    "    for item in ner_comparison_data:\n",
    "        if entity.lower() in item['Entity'].lower() or item['Entity'].lower() in entity.lower():\n",
    "            item['SpaCy Type'] = entity_type\n",
    "            found = True\n",
    "            break\n",
    "    \n",
    "    if not found:\n",
    "        ner_comparison_data.append({\n",
    "            'Entity': entity,\n",
    "            'NLTK Type': 'Not Found',\n",
    "            'SpaCy Type': entity_type\n",
    "        })\n",
    "\n",
    "df_ner_comparison = pd.DataFrame(ner_comparison_data)\n",
    "print(\"\\nNER Comparison:\")\n",
    "print(tabulate(df_ner_comparison, headers='keys', tablefmt='grid', showindex=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Performance and Feature Comparison\n",
    "\n",
    "### 5.1 Processing Speed Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speed_comparison = [\n",
    "    ['Task', 'NLTK Default', 'NLTK HMM', 'SpaCy'],\n",
    "    ['POS Tagging', f'{nltk_default_time:.4f}s', f'{nltk_hmm_time:.4f}s', f'{spacy_time:.4f}s'],\n",
    "    ['NER', f'{nltk_ner_time:.4f}s', 'N/A', 'Included in POS']\n",
    "]\n",
    "\n",
    "print(\"\\nProcessing Speed Comparison:\")\n",
    "print(tabulate(speed_comparison, headers='firstrow', tablefmt='grid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Comprehensive Feature Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_comparison = [\n",
    "    ['Feature', 'NLTK', 'SpaCy'],\n",
    "    ['POS Tagging Approach', 'Statistical (Perceptron, HMM)', 'Neural Network (CNN)'],\n",
    "    ['Tagset', 'Penn Treebank', 'Universal Dependencies'],\n",
    "    ['NER Approach', 'Rule-based + ML', 'Deep Learning (CNN)'],\n",
    "    ['Entity Types (Default)', '3 types (PERSON, ORGANIZATION, GPE)', '18+ types'],\n",
    "    ['Processing Speed', 'Moderate (HMM slower)', 'Fast (batch processing)'],\n",
    "    ['Setup Complexity', 'Multiple downloads required', 'Single model download'],\n",
    "    ['Dependency Parsing', 'Limited support', 'Built-in, robust'],\n",
    "    ['Lemmatization', 'Basic (WordNet)', 'Advanced (context-aware)'],\n",
    "    ['Pipeline Architecture', 'Manual chaining', 'Integrated pipeline'],\n",
    "    ['Customization', 'High (train own models)', 'Medium (update existing)'],\n",
    "    ['Memory Usage', 'Low', 'Higher (neural models)'],\n",
    "    ['Best For', 'Learning, research, custom models', 'Production, accuracy, speed']\n",
    "]\n",
    "\n",
    "print(\"\\nComprehensive Feature Comparison:\")\n",
    "print(tabulate(feature_comparison, headers='firstrow', tablefmt='grid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Key Observations and Recommendations\n",
    "\n",
    "### NLTK Strengths:\n",
    "- **Educational**: Excellent for learning NLP concepts\n",
    "- **Customizable**: Easy to train custom models (like HMM tagger)\n",
    "- **Lightweight**: Lower memory footprint\n",
    "- **Flexible**: More control over individual components\n",
    "\n",
    "### NLTK Weaknesses:\n",
    "- **Accuracy**: Generally lower accuracy on complex texts\n",
    "- **Speed**: HMM tagger can be slower\n",
    "- **Limited NER**: Fewer entity types, less sophisticated\n",
    "- **Manual Pipeline**: Requires manual setup of processing steps\n",
    "\n",
    "### SpaCy Strengths:\n",
    "- **Accuracy**: State-of-the-art neural models\n",
    "- **Speed**: Highly optimized for production\n",
    "- **Complete Pipeline**: All-in-one processing\n",
    "- **Rich Entities**: More entity types and better recognition\n",
    "- **Visualization**: Built-in visualization tools\n",
    "\n",
    "### SpaCy Weaknesses:\n",
    "- **Memory**: Higher memory usage\n",
    "- **Black Box**: Less transparent model internals\n",
    "- **Less Flexible**: Harder to modify core algorithms\n",
    "\n",
    "### Recommendations:\n",
    "- **Use NLTK when**: Learning NLP, building custom models, working with limited resources, need maximum control\n",
    "- **Use SpaCy when**: Building production systems, need high accuracy, processing large volumes, want quick setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Conclusion\n",
    "\n",
    "This notebook demonstrated the practical differences between NLTK and SpaCy for POS tagging and NER:\n",
    "\n",
    "1. **NLTK** offers multiple tagging approaches including trainable HMM models, making it ideal for educational purposes and custom solutions.\n",
    "\n",
    "2. **SpaCy** provides superior accuracy and speed through modern neural networks, making it better suited for production applications.\n",
    "\n",
    "3. **HMM Tagger** in NLTK showcases statistical NLP approaches and offers a good balance between complexity and customization.\n",
    "\n",
    "The choice between NLTK and SpaCy depends on your specific needs:\n",
    "- **Learning and experimentation**: NLTK\n",
    "- **Production systems**: SpaCy\n",
    "- **Custom models**: NLTK\n",
    "- **Quick deployment**: SpaCy\n",
    "\n",
    "Both libraries have their place in the NLP toolkit, and understanding both makes you a more versatile NLP practitioner!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
