UNIT 6: Natural Language Processing (NLP) – Class X

1. Natural Language

Natural Language refers to any system of communication that has evolved naturally in humans through use and repetition without conscious planning (e.g., English, Hindi, Spanish, Mandarin).

Unlike computer programming languages (like Python, C++, or Java), which are strictly structured, rule-bound, and unambiguous, human natural languages are flexible, fluid, and heavily context-dependent.

2. Features of Natural Language

• Ambiguity: A single word or sentence can have multiple meanings depending on context.
  • Lexical Ambiguity: “I saw a bat.” (Cricket bat vs. nocturnal mammal)
  • Syntactic Ambiguity: “I saw the man with the telescope.” (Did I use the telescope, or was the man holding it?)
• Context Dependability: The true meaning of a phrase relies heavily on surrounding sentences, tone, speaker identity, and background knowledge.
• Syntax vs. Semantics:
  • Syntax: Grammatical arrangement of words in a sentence.
  • Semantics: Actual logical meaning conveyed by those words.
• Evolution & Nuance: Natural languages constantly absorb new words, slang, idioms, metaphors, and regional dialects over time.

3. Importance and Real-Life Applications of NLP

Importance:

Human language makes up the vast majority of unstructured data generated daily (emails, reviews, social media posts, chats). Natural Language Processing (NLP) bridges the gap between human communication and computer understanding, allowing machines to read, analyze, and synthesize human language efficiently.

Real-Life Applications:

• Voice Assistants: Google Assistant, Apple Siri, and Amazon Alexa process spoken speech and convert it into machine action.
• Language Translation: Google Translate uses neural machine translation to convert text across languages instantly.
• Auto-generated Captions: YouTube automatically transcribes spoken audio into synchronized text subtitles.
• Email Filtering: Gmail automatically classifies incoming messages into “Primary”, “Social”, “Promotions”, or “Spam”.
• Autocorrect & Predictive Text: Keyboard tools like Grammarly or Gboard suggest spelling corrections and next-word predictions.
• Keyword Extraction: Summarization algorithms scan research papers or news articles to pull out core topics.

4. Stages of Natural Language Processing

When an NLP system processes input text, it passes through five distinct hierarchical stages:

1. Lexical (Morphological) Analysis: Breaks down text into single words or tokens and analyzes word structures, prefixes, and suffixes.
2. Syntactic Analysis (Parsing): Checks grammar, sentence structure, and word relationships using formal grammatical rules.
3. Semantic Analysis: Checks whether the literal meaning of the grammatically correct sentence makes logical sense in the real world.
4. Discourse Integration: Connects the meaning of a sentence to preceding and succeeding sentences to ensure narrative continuity. It checks how sentences or phrases relate to each other in a larger context.
5. Pragmatic Analysis: Interprets real-world context, hidden intent, non-literal meanings, and sarcasm.

5. Chatbots

A Chatbot is an AI-powered software application designed to simulate human conversation through text or voice interfaces. Chatbots serve as interactive conversational agents used widely in customer support, automated inquiries, and virtual assistance.

6. Script Bots vs. Smart Bots

Parameter	Script Bot (Rule-Based)	Smart Bot (AI-Based)
Operating Principle	Operates strictly on pre-programmed scripts, decision trees, and keyword triggers.	Operates on Machine Learning (ML), NLP models, and statistical patterns.
Flexibility	Fails or throws error messages when input strays outside predefined rules.	Understands unpredictable, natural language variations and typos.
Learning Ability	Static; cannot learn from interactions unless manually updated by a developer.	Dynamic; continuously learns and improves over time from ongoing conversation data.
Complexity	Simple, cheap, and fast to build.	Complex, requires larger datasets, higher computing power, and AI models.
Examples	Bank IVR menus, basic website FAQ pop-ups.	Kuki (Mitsuku), ChatGPT, Eliza, Google Assistant.

7. Text Processing & Text Normalization

Text Processing involves cleaning, converting, and standardizing unstructured human text into a clean format that computer models can process mathematically.

Text Normalization is the series of preprocessing steps used to reduce text complexity, eliminate noise, and transform all words into a uniform, base representation.

8. Example of Text Normalization (6 Steps)

Raw Input Sentence:

“The 2 fast dogs were barking loudly at the cats! They jumped over fences.”

• Step 1: Sentence Segmentation
  • Divides the raw paragraph into individual sentence units.
  • Output:
    1. “The 2 fast dogs were barking loudly at the cats!”
    2. “They jumped over fences.”
• Step 2: Tokenization
  • Breaks sentences into smaller individual items called tokens (words, numbers, punctuation).
  • Output: ['The', '2', 'fast', 'dogs', 'were', 'barking', 'loudly', 'at', 'the', 'cats', '!', 'They', 'jumped', 'over', 'fences', '.']
• Step 3: Lowercasing (Converting Case)
  • Converts all characters to lowercase so identical words with different cases (e.g., “The” and “the”) match.
  • Output: ['the', '2', 'fast', 'dogs', 'were', 'barking', 'loudly', 'at', 'the', 'cats', '!', 'they', 'jumped', 'over', 'fences', '.']
• Step 4: Removing Punctuation & Special Characters
  • Strips out symbols, punctuation marks, and unwanted characters that offer no semantic value.
  • Output: ['the', '2', 'fast', 'dogs', 'were', 'barking', 'loudly', 'at', 'the', 'cats', 'they', 'jumped', 'over', 'fences']
• Step 5: Removing Stop Words
  • Removes structural grammar words (e.g., “the”, “were”, “at”, “they”, “over”) that appear frequently but carry little informative meaning.
  • Output: ['2', 'fast', 'dogs', 'barking', 'loudly', 'cats', 'jumped', 'fences']
• Step 6: Stemming & Lemmatization
  • Reduces remaining words to their base root form.
    • Stemming: Chopping off affixes (can produce non-dictionary roots, e.g., “barking” → “bark”).
    • Lemmatization: Mapping words to actual dictionary roots using grammar (e.g., “dogs” → “dog”, “cats” → “cat”).
  • Final Normalized Output: ['2', 'fast', 'dog', 'bark', 'loudly', 'cat', 'jump', 'fence']

9. NLP Models (BoW and TF-IDF) with Examples

Machine learning algorithms cannot process plain text strings directly; text must be converted into numerical vector representations. Following are the two popular techniques to extract information from the text data.

A. Bag-of-Words (BoW)

The Bag-of-Words model counts the frequency of occurrence of each word across documents while disregarding sentence structure, context, and word order.

Steps involved in BoW algorithm are as follows:

Step 1: Preprocessing & Tokenization

Convert text to lowercase, remove punctuation, and split sentences into individual word tokens:

Step 2: Vocabulary Creation

Extract all unique words across both documents to build the master vocabulary list:

Step 3: Vectorization (Frequency Count)

Count the frequency of each vocabulary word within each individual document:

Step 4: Document Vector Table Generation

Map the counts into a structured Document-Term Matrix (Document Vector Table):

Example:

Step 1: Preprocessing & Tokenization

Convert text to lowercase, remove punctuation, and split sentences into individual word tokens:

• Document 1:["ai", "helps", "humans", "daily"]
• Document 2:["humans", "use", "ai", "technology"]

Step 2: Vocabulary Creation

Extract all unique words across both documents to build the master vocabulary list:

• Vocabulary ($V$):["ai", "daily", "helps", "humans", "technology", "use"](Total size = 6)

Step 3: Vectorization (Frequency Count)

Count the frequency of each vocabulary word within each individual document:

• Document 1:ai: 1, daily: 1, helps: 1, humans: 1, technology: 0, use: 0
• Document 2:ai: 1, daily: 0, helps: 0, humans: 1, technology: 1, use: 1

Step 4: Document Vector Table Generation

Map the counts into a structured Document-Term Matrix (Document Vector Table):

Document	ai	daily	helps	humans	technology	use
Document 1	1	1	1	1	0	0
Document 2	1	0	0	1	1	1

B. TF-IDF (Term Frequency – Inverse Document Frequency)

While BoW counts word frequency, TF-IDF assigns mathematical weight to words based on how informative they are across an entire collection of documents (corpus).

TF-IDF = TF X IDF

1. Term Frequency (TF): Measures how often a word appears in a specific single document.
   TF(t, d) = Count of term t in document d / Total number of words in document d
2. Inverse Document Frequency (IDF): Assigns higher weights to rare words and penalizes words that appear everywhere across all documents.
   IDF(t, D) = log(Total number of documents in corpus D / Number of documents containing term t)

Worked Example:

• Corpus: 2 documents (D = 2).
• Document 1 (3 words): “data science data”
• Word under test: “science”
• TF Calculation: TF(“science”, D1) = 1/3 = 0.33
• IDF Calculation (Assuming “science” appears only in 1 out of 2 documents):
  IDF({“science”, D) = log(2/1) = log(2) ≈ 0.301
• Final TF-IDF Weight:
  TF-IDF = 0.33 X 0.301 ≈ 0.099

10. Code-Based NLP Tools

Code-based tools offer high customization, programmatic automation, and detailed control over text pipeline creation through programming languages like Python.

• NLTK (Natural Language Toolkit):
  • One of the most popular open-source Python libraries created for academic teaching and research in NLP.
  • Provides built-in functions for tokenization, stemming, lemmatization, stop-word filtering, and part-of-speech (POS) tagging.
• SpaCy:
  • An industrial-grade, high-performance Python NLP library built for production environments.
  • Optimized for fast execution, neural network models, entity recognition (NER), and large-scale deep learning pipelines.

11. No-Code Based NLP Tools

No-Code tools allow non-programmers to perform complex NLP tasks, data visualization, and model training using graphical user interfaces (GUIs) or drag-and-drop workflows.

• Orange Data Mining:
  • An open-source, component-based visual data science suite with a dedicated Text Mining add-on.
  • Users build pipelines visually by connecting widgets for text pre-processing, word clouds, sentiment scoring, and topic modeling without writing code.
• MonkeyLearn:
  • A cloud-based commercial platform featuring simple GUI-driven AI modules.
  • Enables businesses to create custom text classifiers, sentiment analyzers, and keyword extraction workflows effortlessly.

12. Introduction to Sentiment Analysis

Sentiment Analysis (also called Opinion Mining) is an NLP technique used to determine the emotional tone, sentiment, or attitude expressed within a piece of text.

• Classification Categories:
  • Positive: Expresses satisfaction or praise (e.g., “This phone has an incredible battery!”).
  • Negative: Expresses dissatisfaction or criticism (e.g., “The service was slow and terrible.”).
  • Neutral: Expresses objective facts without emotion (e.g., “The package arrived on Tuesday.”).

13. Applications of Sentiment Analysis

• Brand & Reputation Monitoring: Companies analyze social media posts (X/Twitter, Instagram) to gauge public perception during product launches.
• Customer Feedback & Review Analysis: E-commerce platforms (Amazon, Flipkart) automatically sort thousands of product reviews to flag complaints.
• Financial Market Forecasting: Investors analyze financial news articles and earnings calls to predict stock price movements based on market sentiment.
• Political Campaigning: Political analysts evaluate voter reactions to speeches, debates, and policy decisions across digital platforms.
• Automated Support Ticket Prioritization: Helpdesk tools detect angry or urgent tone in customer emails and route high-priority tickets directly to human agents.

Some important questions on NLP:

Q1. An AI model has been developed to predict whether electric vehicle batteries need replacement based on performance data. The model was tested on a dataset of 700 vehicles and the resulting confusion matrix is as follows :

Confusion Matrix		Reality
		Yes	No
Prediction	Yes	590	20
	No	10	80

The above Confusion Matrix can also be represented as follows :

Actual Values	Predicted Values
		1	0
	1	590	10
	0	20	80

(a) How many total cases are False Positives in the above scenario ?
(b) Calculate Precision, Recall and F1 score

(a) From the given confusion matrix (where positive target 1 = “Yes” and negative target 0 = “No”):

• True Positive (TP) = 590 (Actual 1, Predicted 1)
• False Positive (FP) = 20 (Actual 0, Predicted 1)
• False Negative (FN) = 10 (Actual 1, Predicted 0)
• True Negative (TN) = 80 (Actual 0, Predicted 0)

Total False Positives = 20

(b) Precision, Recall, and F1 Score:

Precision:

Precision = TP/(TP + FP) = 590/(590 + 20) = 590/610 ≈0.9672 (96.72%)

Recall:

Recall = TP/(TP + FN) = 590/(590 + 10) = 590/600 ≈0.9833 (98.33%)

F1 Score:

F1 Score = 2 x (Precision x Recall) / (Precision + Recall) = 2 x (0.9672 x 0.9833) / (0.9672 + 0.9833) = 1.9018/1.9505 ≈0.9750 (97.50%)

Q2. Consider the following documents :
Document 1 : AI helps humans daily.
Document 2 : Humans use AI technology.
Implement all the four steps of Bag of Words (BoW) model to create a document vector table.

Bag of Words (BoW) Implementation

Step 1: Preprocessing & Tokenization

Convert text to lowercase, remove punctuation, and split sentences into individual word tokens:

• Document 1:["ai", "helps", "humans", "daily"]
• Document 2:["humans", "use", "ai", "technology"]

Step 2: Vocabulary Creation

Extract all unique words across all documents to build the master vocabulary list:

• Vocabulary (V):["ai", "daily", "helps", "humans", "technology", "use"](Total size = 6)

Step 3: Vectorization (Frequency Count)

Count the frequency of each vocabulary word within each individual document:

• Document 1:ai: 1, daily: 1, helps: 1, humans: 1, technology: 0, use: 0
• Document 2:ai: 1, daily: 0, helps: 0, humans: 1, technology: 1, use: 1

Step 4: Document Vector Table Generation

Map the counts into a structured Document-Term Matrix (Document Vector Table):

Document	ai	daily	helps	humans	technology	use
Document 1	1	1	1	1	0	0
Document 2	1	0	0	1	1	1

Q3.