Unit 1: Foundations of NLP and Text Processing - Subjective Questions

Natural Language Processing (NLP) originated from the intersection of linguistics, computer science, artificial intelligence, and mathematics.

• 1940s-1950s: Early work was influenced by information theory and Alan Turing's ideas about machine intelligence. The Turing Test proposed evaluating intelligence through language-based conversation.
• 1950s-1960s: Research concentrated on rule-based machine translation. Systems used dictionaries and manually written grammar rules.
• 1960s-1980s: Symbolic NLP became prominent. Programs such as ELIZA demonstrated pattern-based conversation, while formal grammars were used for parsing.
• 1980s-2000s: Statistical NLP emerged because larger digital corpora and greater computing power became available. Probabilistic models such as n-grams, Hidden Markov Models, and statistical parsers replaced many hand-written rules.
• 2000s-2010s: Machine-learning methods using features such as Bag-of-Words and TF-IDF became standard.
• 2010s-present: Deep learning introduced word embeddings, recurrent neural networks, attention, and transformers. Modern language models learn representations from massive text collections.

Thus, NLP has evolved from rule-based systems to statistical learning, and subsequently to deep neural language models.

Language is a structured system of symbols, sounds, or written forms used to communicate meaning. Grammar is the set of principles that governs how linguistic units are formed and combined.

Major levels of linguistic analysis include:

• Phonetics and phonology: Study speech sounds and their organization. They are important in speech recognition and synthesis.
• Morphology: Examines how words are formed from smaller meaningful units called morphemes.
• Lexical analysis: Studies words, their categories, and their dictionary properties.
• Syntax: Examines how words combine to form grammatically valid phrases and sentences.
• Semantics: Determines the literal meaning of words and sentences.
• Pragmatics: Interprets meaning using speaker intention and situational context.
• Discourse: Studies relationships between multiple sentences, including coherence and reference.

An NLP system often needs several of these levels because grammatical structure alone does not completely determine the intended meaning.

Morphology is the study of word structure and the processes through which words are formed. Its smallest meaning-bearing unit is a morpheme.

For example, the word unhappiness can be analyzed as un + happy + ness.

Inflectional morphology:

• Changes the grammatical form of a word without creating a new lexical category.
• Usually expresses tense, number, person, or comparison.
• Examples: walk → walked, cat → cats, and small → smaller.

Derivational morphology:

• Creates a new word or changes the grammatical category or meaning of the base word.
• Examples: happy → happiness, teach → teacher, and possible → impossible.

Morphological analysis helps NLP systems perform stemming, lemmatization, part-of-speech tagging, machine translation, information retrieval, and handling of morphologically rich languages. It also reduces data sparsity by connecting related word forms.

Syntax is the study of rules governing the arrangement of words into phrases, clauses, and sentences.

Two common representations are:

• Constituency parsing: Divides a sentence into nested phrases such as noun phrases and verb phrases. For example, in The student reads a book, The student is a noun phrase and reads a book is a verb phrase.
• Dependency parsing: Represents grammatical relationships between individual words. In the same sentence, reads is the main verb, student is its subject, and book is its object.

Syntactic analysis is useful because it:

• Identifies subjects, objects, modifiers, and predicates.
• Helps resolve some forms of structural ambiguity.
• Supports information extraction and relation extraction.
• Improves machine translation and question answering.
• Provides sentence structure for semantic interpretation.

However, syntactic correctness does not guarantee meaningfulness. A sentence can be grammatically valid while being semantically unusual.

Semantics is the study of meaning in words, phrases, and sentences.

• Lexical semantics concerns the meanings and relationships of words. It includes synonymy, antonymy, polysemy, and hyponymy. For example, bank may mean a financial institution or the side of a river.
• Compositional semantics determines sentence meaning from the meanings of individual words and the way they are combined. The sentences The dog chased the cat and The cat chased the dog contain similar words but express different meanings because of syntax.
• Contextual semantics uses surrounding text or situational information. In She deposited cash at the bank, the context selects the financial sense of bank.

Semantic processing is difficult because natural language contains ambiguity, figurative expressions, implicit knowledge, and context-dependent references. It is essential for question answering, search, translation, summarization, and conversational systems.

Major NLP challenges include:

• Lexical ambiguity: A word can have several meanings, such as bat referring to an animal or sporting equipment.
• Syntactic ambiguity: A sentence can have multiple grammatical interpretations. For example, I saw the man with a telescope does not clearly identify who has the telescope.
• Semantic and pragmatic ambiguity: Intended meaning may depend on real-world knowledge, speaker intention, irony, or sarcasm.
• Coreference: A system must determine what pronouns and noun phrases refer to.
• Language variation: Dialects, slang, spelling variations, and code-switching increase complexity.
• Data sparsity and OOV words: Rare or unseen words may not be represented adequately.
• Long-range context: Meaning may depend on information appearing many sentences earlier.
• Multilingual complexity: Languages differ in scripts, word order, and morphology.
• Noisy text: Social-media text may contain abbreviations, emojis, and grammatical errors.
• Bias and domain shift: Models may learn social biases or perform poorly outside their training domain.

These challenges make complete language understanding substantially harder than simple keyword matching.

Important applications of NLP include:

• Machine translation: Converts text from one language to another while preserving meaning and fluency.
• Sentiment analysis: Classifies opinions as positive, negative, or neutral and may identify emotions or aspects.
• Information extraction: Detects entities, relations, events, dates, and facts in unstructured documents.
• Question answering and chatbots: Interpret user queries, retrieve relevant knowledge, and generate appropriate responses.
• Text summarization: Produces a shorter version of a document while retaining its central information.
• Search engines: Analyze queries and documents to rank relevant results.
• Spam and content classification: Categorize messages or documents according to topic, intent, or safety.
• Speech interfaces: Combine automatic speech recognition with language understanding and response generation.

Most real applications combine several tasks. For example, a chatbot may require tokenization, intent classification, entity recognition, dialogue-state tracking, retrieval, and language generation.

Tokenization is the process of dividing text into smaller units called tokens.

• Sentence tokenization: Divides a document into sentences. It is useful for parsing and summarization, but abbreviations such as Dr. can make sentence boundaries ambiguous.
• Word tokenization: Splits sentences into words and punctuation. It is intuitive and efficient, but contractions, compounds, and languages without spaces are difficult to process.
• Character tokenization: Treats every character as a token. It eliminates word-level OOV problems, but sequences become long and individual tokens carry limited meaning.
• Subword tokenization: Breaks words into reusable pieces using methods such as Byte Pair Encoding or WordPiece. For example, unhappiness might become un, happi, and ness. It balances vocabulary size and sequence length, although resulting pieces may not correspond to linguistic morphemes.

The appropriate tokenizer depends on the language, model architecture, vocabulary size, and application requirements.

Stemming and lemmatization reduce inflected words to a common form, but they use different methods.

Stemming:

• Applies heuristic rules to remove prefixes or suffixes.
• May produce a non-dictionary form.
• Example: studies, studying, and studied may be reduced to studi.
• It is fast and useful in search or large-scale indexing where approximate matching is sufficient.
• It may over-stem unrelated words or under-stem related forms.

Lemmatization:

• Uses vocabulary, morphological analysis, and often part-of-speech information.
• Produces a valid dictionary base form called a lemma.
• Examples: studies → study and better → good, depending on context and part of speech.
• It gives more linguistically accurate output but is slower and language-dependent.

Stemming is appropriate when speed and recall are priorities. Lemmatization is preferable when grammatical correctness and precise semantic analysis matter.

Stop words are frequent function words such as the, is, of, and to. Removing them can reduce vocabulary size, storage, and noise in traditional retrieval or topic-classification systems.

However, stop words may carry important information:

• not changes sentiment polarity.
• Pronouns may be required for coreference resolution.
• Function words can help identify writing style or authorship.
• Modern sequence models use word order and context, so removal may damage sentence meaning.

Punctuation handling can include removal, separation into tokens, or normalization. Punctuation may be discarded in simple Bag-of-Words models, but it is useful for:

• Detecting sentence boundaries.
• Identifying questions and exclamations.
• Interpreting contractions and abbreviations.
• Capturing sentiment or emphasis, as in Great! versus Great?

Therefore, stop-word and punctuation policies must follow the target task, language, model, and expected input rather than being treated as universal cleaning rules.

An out-of-vocabulary (OOV) word is a word encountered during inference that is absent from the system's fixed vocabulary. OOV words arise from names, spelling errors, new terminology, inflections, slang, and domain-specific expressions.

Common strategies include:

• Unknown token: Map all unseen words to a special token such as [UNK]. This is simple but loses the identity and structure of the word.
• Normalization and spelling correction: Standardize predictable variants before vocabulary lookup.
• Stemming or lemmatization: Map related inflected forms to a known root or lemma.
• Character-level models: Construct word representations from characters and therefore process unseen spellings.
• Subword tokenization: Divide rare words into known pieces. For example, an unseen word may still be represented through a known prefix, root, and suffix.
• Byte-level tokenization: Represents any input through bytes, nearly eliminating OOV cases.
• Vocabulary expansion or retraining: Add frequent domain terms when adapting a model.

Modern neural systems generally prefer subword or byte-level methods because they retain more information than a single unknown token.

Text normalization transforms text into a more consistent representation while attempting to preserve task-relevant meaning.

A possible pipeline is:

1. Convert text to a consistent Unicode representation.
2. Standardize whitespace and line breaks.
3. Apply case folding when capitalization is not required.
4. Normalize dates, numbers, URLs, email addresses, or user mentions using task-specific placeholders.
5. Expand contractions or abbreviations when beneficial.
6. Correct selected spelling variants.
7. Tokenize the normalized text.
8. Optionally apply stop-word removal, stemming, or lemmatization.

Excessive normalization can destroy useful distinctions:

• Lowercasing may remove information from proper nouns and acronyms.
• Replacing every number may hide quantities important to financial or medical tasks.
• Removing accents can merge distinct words.
• Expanding contractions may be ambiguous.
• Deleting emojis or punctuation may remove sentiment.

Normalization should therefore be reproducible, language-aware, and guided by the downstream objective.

The Bag-of-Words (BoW) model represents a document by the occurrence counts of vocabulary terms while ignoring grammar and word order.

For the documents:

• : cats chase mice
• : dogs chase cats

Using the ordered vocabulary [cats, chase, mice, dogs], the vectors are:

A binary BoW representation records only presence or absence, whereas a count representation records term frequency.

Advantages:

• Simple to construct and interpret.
• Efficient with sparse data structures.
• Often effective for document classification and retrieval baselines.

Limitations:

• Ignores word order and syntax.
• Does not directly represent semantic similarity.
• Produces high-dimensional sparse vectors.
• Gives common and rare terms inappropriate importance unless weighting is used.
• Cannot distinguish sentences such as dog bites man and man bites dog when their word counts are identical.

An n-gram is a contiguous sequence of tokens extracted from text.

For the sentence deep learning processes language:

• Unigrams: deep, learning, processes, language
• Bigrams: deep learning, learning processes, processes language
• Trigrams: deep learning processes, learning processes language

Larger n-grams preserve more local word order and can capture expressions such as not good, which a unigram representation cannot model directly.

The choice of involves a trade-off:

• Small values of produce fewer features, require less data, and generalize more easily, but capture little context.
• Large values of preserve more phrase structure, but produce a very large sparse vocabulary.
• As increases, most possible sequences occur rarely or never, creating data sparsity.
• Large n-grams also require more memory and may fail to match slightly different expressions.

Practical systems often combine unigrams and bigrams, apply frequency thresholds, or use smoothing and feature selection.

Term Frequency-Inverse Document Frequency (TF-IDF) assigns a high weight to a term when it is frequent in one document but uncommon across the corpus.

Normalized term frequency is:

where is the count of term in document , and is the number of terms in the document.

Inverse document frequency is:

where is the total number of documents and is the number containing .

For the given values:

Therefore:

The term's TF-IDF weight is approximately 0.1382. Exact values vary with alternative TF scaling, logarithm bases, and smoothing conventions.

Bag-of-Words, n-grams, and TF-IDF are sparse text-representation techniques with different emphasis.

Bag-of-Words:

• Represents documents using token counts or binary indicators.
• Captures term occurrence but ignores word order.
• Is simple, interpretable, and suitable as a baseline.
• Produces sparse, high-dimensional vectors.

N-grams:

• Represent contiguous sequences of tokens.
• Capture limited local order and phrases such as not useful.
• Improve performance when phrase patterns are informative.
• Greatly increase dimensionality and data sparsity as grows.

TF-IDF:

• Reweights BoW or n-gram features using local frequency and corpus rarity.
• Reduces the influence of terms common across many documents.
• Is effective in retrieval, similarity measurement, and classical text classification.
• Still ignores meaning and, when applied to unigrams, word order.

In summary, BoW is the simplest count representation, n-grams add local context, and TF-IDF improves feature weighting. All three depend on a fixed vocabulary and do not inherently model deep semantic relationships.

A suitable social-media sentiment pipeline could contain the following stages:

1. Preserve the raw text: Retain an unmodified version for auditing and error analysis.
2. Unicode normalization: Standardize visually equivalent character sequences.
3. Replace selected metadata: Map URLs and user mentions to placeholders such as [URL] and [USER], since their presence may matter even when their exact value does not.
4. Handle hashtags: Preserve meaningful hashtag words and optionally segment forms such as #BestMovieEver.
5. Preserve emojis and emoticons: Convert them to stable tokens because they often express sentiment directly.
6. Normalize elongated forms carefully: Convert soooo good to a controlled representation while retaining an emphasis marker.
7. Tokenize with a social-media-aware tokenizer: Correctly process contractions, hashtags, emojis, and punctuation.
8. Retain negation: Words such as not, never, and hardly are essential for polarity.
9. Handle OOV items: Use subword tokenization or character features for slang, misspellings, and new words.
10. Build features: Use TF-IDF unigrams and bigrams for a classical model, or subword token IDs for a transformer.
11. Apply identical transformations during training and inference: Store the fitted vocabulary and normalization rules to prevent inconsistency and data leakage.

Stop-word removal, stemming, and punctuation deletion should be evaluated experimentally because they may remove sentiment cues.

Language interpretation depends on multiple linguistic levels working together.

Morphological analysis:

• students contains the plural suffix -s.
• books also carries plural number.
• reading contains the progressive suffix -ing.
• were is an inflected past-tense plural form of be.

Syntactic analysis:

• The students forms the subject noun phrase.
• were reading the books forms the verb phrase.
• the books is the direct object of reading.
• Agreement between plural students and were helps establish the grammatical structure.

Semantic analysis:

• The students are interpreted as the agents performing the activity.
• The books are the objects or themes involved in the activity.
• The construction were reading indicates an ongoing activity situated in the past.

Morphology provides grammatical features, syntax establishes structural relationships, and semantics assigns roles and meaning. An error at one level can affect later analysis; for example, incorrect part-of-speech or number detection may lead to an incorrect parse and interpretation.

Tokenization is difficult because visible spaces do not always correspond to linguistic word boundaries.

• Contractions: don't may be treated as one token or split into do and n't. The best choice depends on the model and task.
• Hyphenation: Expressions such as state-of-the-art may be one lexical unit, several words, or both for indexing purposes.
• Punctuation: Periods can indicate sentence boundaries, abbreviations, decimals, or domain names. Apostrophes may mark possession or contraction.
• Languages without spaces: Chinese, Japanese, and Thai require word segmentation based on dictionaries, statistical models, or learned subword units.
• Morphologically rich languages: A single written word may encode information that English expresses using several words.
• Domain-specific text: Biomedical names, legal references, source code, chemical formulas, and product identifiers require specialized rules.
• Social-media text: Hashtags, emojis, mentions, URLs, and creative spellings complicate standard tokenizers.

A robust tokenizer must be language-aware, domain-aware, consistent between training and inference, and evaluated by its effect on the downstream task. Subword tokenization reduces many vocabulary problems but does not eliminate boundary and normalization decisions.

The statement is incorrect because preprocessing can remove signal as well as noise.

• Normalization: Lowercasing reduces vocabulary size, but it removes distinctions such as US versus us. Number replacement may help topic classification but harm financial prediction.
• Stop-word removal: Removing frequent words can improve some retrieval systems, yet deleting not can reverse sentiment and deleting pronouns can harm coreference analysis.
• Stemming: It groups related forms efficiently, but aggressive stemming may merge words with different meanings or produce unreadable stems. Lemmatization may be more precise, although it also depends on correct context and part-of-speech analysis.
• Punctuation handling: Punctuation may be unnecessary in a simple topic model, but question marks, exclamation marks, apostrophes, and sentence boundaries can be important for intent, sentiment, and parsing.
• Modern language models: Pretrained transformers expect text resembling their pretraining input. Heavy cleaning may create a distribution mismatch and degrade performance.

Preprocessing should therefore be treated as a set of testable modeling decisions. Each transformation should have a task-specific justification, be fitted without leaking test data, and be evaluated through controlled experiments and error analysis.