{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a4a9b8d",
   "metadata": {},
   "source": [
    "\n",
    "# Lab: Rule-Based Sentence Tokenization\n",
    "\n",
    "## Objective\n",
    "In this lab, you will **design and implement the simplest rule-based sentence tokenizer (Decision Tree)**.\n",
    "\n",
    "You will write a Python function that:\n",
    "- Takes a **raw text corpus** as input\n",
    "- Returns a **list of sentences**\n",
    "\n",
    "This lab focuses on **reasoning about rules and edge cases**, not using pre-built NLP libraries.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb43e98e",
   "metadata": {},
   "source": [
    "\n",
    "## Background\n",
    "\n",
    "Sentence tokenization (sentence segmentation) is the task of identifying sentence boundaries.\n",
    "A naive approach is to split on punctuation such as:\n",
    "- `.`\n",
    "- `!`\n",
    "- `?`\n",
    "\n",
    "However, this approach fails in many real-world cases:\n",
    "- Abbreviations (e.g., *Dr.*, *Mr.*)\n",
    "- Decimal numbers (e.g., *3.14*)\n",
    "- Titles and acronyms\n",
    "- Quoted text\n",
    "\n",
    "Your goal is to design **reasonable heuristic rules** to handle these cases.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449bf6f9",
   "metadata": {},
   "source": [
    "\n",
    "## Task Description\n",
    "\n",
    "You must implement the function:\n",
    "\n",
    "```python\n",
    "def rule_based_sentence_tokenizer(text):\n",
    "    pass\n",
    "```\n",
    "\n",
    "### Requirements\n",
    "- Input: a string containing multiple sentences\n",
    "- Output: a list of sentence strings\n",
    "- You should catch candidate sentence boundaries (\".\", \"!\", \"?\")\n",
    "- Decide for each if it is a real sentence boundary or not using rules (if else statements)\n",
    "- Do not use NLP libraries (NLTK, spaCy, CoreNLP, etc.) to solve the problem\n",
    "- Compare your method result to the result of Punkt Tokenizer from NLTK on the given testing corpus.\n",
    "\n",
    "Your solution does **not need to be perfect**, but it should handle common cases reasonably well.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ade062b7",
   "metadata": {},
   "source": [
    "\n",
    "## Testing Corpus\n",
    "\n",
    "Use the following corpus to test your tokenizer.\n",
    "It intentionally contains **edge cases**.\n",
    "\n",
    "Pay attention to:\n",
    "- Abbreviations\n",
    "- Numbers\n",
    "- Capitalization\n",
    "- Quotation marks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b5dc1a27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dr. Smith arrived at 5 p.m. He said, \"This is unexpected.\".\n",
      "Apple released a new product today. It costs $999.99!\n",
      "Is this the best option? Many people think so.\n",
      "Mr. Johnson lives in the U.S. He works at Apple Inc.\n",
      "The value of pi is approximately 3.14. It is used in math.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_corpus = \"\"\"\n",
    "Dr. Smith arrived at 5 p.m. He said, \"This is unexpected.\".\n",
    "Apple released a new product today. It costs $999.99!\n",
    "Is this the best option? Many people think so.\n",
    "Mr. Johnson lives in the U.S. He works at Apple Inc.\n",
    "The value of pi is approximately 3.14. It is used in math.\n",
    "\"\"\"\n",
    "\n",
    "print(test_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3143a91",
   "metadata": {},
   "source": [
    "\n",
    "## Your Task\n",
    "\n",
    "1. Write a rule-based sentence tokenizer.\n",
    "2. Apply it to the testing corpus.\n",
    "3. Print each detected sentence on a new line.\n",
    "\n",
    "💡 *Hint:* Start simple, then refine your rules.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a17e45c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "ABREVATION = [\"p.m.\", \"U.S.\", \"Inc.\"]\n",
    "\n",
    "PROUNOUNS = [\"Dr.\", \"Mr.\"]\n",
    "\n",
    "import re\n",
    "\n",
    "\n",
    "def is_EOS(token, next_token):\n",
    "\n",
    "    # iza ken ra2em decimal\n",
    "    if re.match(r\"\\d+\\.\\d+$\", token):\n",
    "        return False\n",
    "\n",
    "    # iza kenit abr\n",
    "    if token in PROUNOUNS and next_token[0].isupper():\n",
    "        return False\n",
    "\n",
    "    if token in ABREVATION:\n",
    "        if token in ABREVATION and next_token[0].isupper():\n",
    "            return True\n",
    "        return False\n",
    "\n",
    "    # iza 5lst b punctuation\n",
    "    if re.search(r\"[.!?]$\", token):\n",
    "        return True\n",
    "\n",
    "    return False\n",
    "\n",
    "\n",
    "def rule_based_sentence_tokenizer(text):\n",
    "    tokens = re.findall(r\"\\S+\", text)\n",
    "    print(\"Tokens: \", tokens)\n",
    "\n",
    "    sentences = []\n",
    "    current_sentence = []\n",
    "    for i, token in enumerate(tokens):\n",
    "        current_sentence.append(token)\n",
    "        if token == \"\\n\":\n",
    "            sentences.append(\" \".join(current_sentence))\n",
    "            continue\n",
    "        if i < len(tokens) - 1 and is_EOS(token, tokens[i + 1]):\n",
    "            sentences.append(\" \".join(current_sentence))\n",
    "            current_sentence = []\n",
    "\n",
    "    if current_sentence:\n",
    "        sentences.append(\" \".join(current_sentence))\n",
    "\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5220cd2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens:  ['Dr.', 'Smith', 'arrived', 'at', '5', 'p.m.', 'He', 'said,', '\"This', 'is', 'unexpected.\".', 'Apple', 'released', 'a', 'new', 'product', 'today.', 'It', 'costs', '$999.99!', 'Is', 'this', 'the', 'best', 'option?', 'Many', 'people', 'think', 'so.', 'Mr.', 'Johnson', 'lives', 'in', 'the', 'U.S.', 'He', 'works', 'at', 'Apple', 'Inc.', 'The', 'value', 'of', 'pi', 'is', 'approximately', '3.14.', 'It', 'is', 'used', 'in', 'math.']\n",
      "['Dr. Smith arrived at 5 p.m.', 'He said, \"This is unexpected.\".', 'Apple released a new product today.', 'It costs $999.99!', 'Is this the best option?', 'Many people think so.', 'Mr. Johnson lives in the U.S.', 'He works at Apple Inc.', 'The value of pi is approximately 3.14.', 'It is used in math.']\n",
      "Sentence 1: Dr. Smith arrived at 5 p.m.\n",
      "Sentence 2: He said, \"This is unexpected.\".\n",
      "Sentence 3: Apple released a new product today.\n",
      "Sentence 4: It costs $999.99!\n",
      "Sentence 5: Is this the best option?\n",
      "Sentence 6: Many people think so.\n",
      "Sentence 7: Mr. Johnson lives in the U.S.\n",
      "Sentence 8: He works at Apple Inc.\n",
      "Sentence 9: The value of pi is approximately 3.14.\n",
      "Sentence 10: It is used in math.\n"
     ]
    }
   ],
   "source": [
    "# Test your implementation\n",
    "\n",
    "sentences = rule_based_sentence_tokenizer(test_corpus)\n",
    "print(sentences)\n",
    "\n",
    "for i, s in enumerate(sentences, 1):\n",
    "    print(f\"Sentence {i}: {s}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8c81d0fc-7c78-4e0a-846d-d0d9d705f0eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "NLTK Sentences:\n",
      "Sentence 1: \n",
      "Dr. Smith arrived at 5 p.m.\n",
      "Sentence 2: He said, \"This is unexpected.\".\n",
      "Sentence 3: Apple released a new product today.\n",
      "Sentence 4: It costs $999.99!\n",
      "Sentence 5: Is this the best option?\n",
      "Sentence 6: Many people think so.\n",
      "Sentence 7: Mr. Johnson lives in the U.S.\n",
      "Sentence 8: He works at Apple Inc.\n",
      "Sentence 9: The value of pi is approximately 3.14.\n",
      "Sentence 10: It is used in math.\n"
     ]
    }
   ],
   "source": [
    "# TODO: Compare to Punkt Tokenizer of NLTK\n",
    "\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "nltk_sentences = sent_tokenize(test_corpus)\n",
    "print(\"\\nNLTK Sentences:\")\n",
    "for i, s in enumerate(nltk_sentences, 1):\n",
    "    print(f\"Sentence {i}: {s}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
