Unit 1: Foundations of NLP and Text Processing

I. Orientation — Foundations of Language Processing

Natural Language Processing (NLP) is the branch of artificial intelligence concerned with enabling computers to analyze, represent, understand, and generate human language. It combines linguistics, computer science, statistics, and machine learning, with modern systems increasingly based on deep neural networks.

A. Origin of NLP

NLP developed from attempts to formalize language and automate translation, later shifting from hand-written rules to data-driven learning.

• Early foundations: Formal linguistics, information theory, and Alan Turing’s 1950 proposal of the imitation game helped frame language as evidence of machine intelligence.
• Machine translation: The 1954 Georgetown–IBM experiment demonstrated automatic translation of a restricted set of Russian sentences into English.
• Rule-based period: Early systems used dictionaries, grammar rules, and symbolic representations; ELIZA (1966), for example, matched user input against scripted patterns.
• Statistical period: During the 1980s and 1990s, probabilistic models learned language patterns from corpora rather than relying entirely on manual rules.
• Deep-learning period: Neural embeddings, recurrent networks, attention, and Transformers enabled end-to-end learning from large datasets.
• Defining characteristics:
  • Input: Text, speech transcripts, documents, or conversational turns.
  • Representation: Tokens are converted into numerical features or vectors.
  • Processing: Models identify patterns at morphological, syntactic, semantic, and contextual levels.
  • Output: A label, translation, answer, summary, generated passage, or structured representation.

II. Language Structure — Rules and Linguistic Levels

Language is a structured communication system in which sounds or symbols combine into meaningful units. NLP systems require linguistic knowledge because the same surface form can behave differently depending on its grammatical role and context.

A. Language and grammar

Grammar is the system of conventions governing how linguistic units are formed and combined into acceptable expressions.

• Descriptive grammar: Records how speakers actually use language, including constructions such as “Who did you speak to?”
• Prescriptive grammar: Specifies preferred usage according to a formal standard, such as recommending “To whom did you speak?”
• Lexicon: Stores words and associated information, including the noun and verb senses of “record.”
• Grammar levels:
  • Morphological rules: Form words, as in walk + ed → walked.
  • Syntactic rules: Combine words, as in Determiner + Noun → Noun Phrase.
  • Semantic constraints: Determine plausible interpretations.
• NLP relevance: Grammar supports parsing, agreement checking, information extraction, translation, and generation.

B. Linguistic essentials

Linguistic analysis divides language into interacting levels that provide different kinds of evidence.

• Phonetics and phonology: Concern speech sounds and sound systems; “ice cream” and “I scream” illustrate similar sounds with different segmentations.
• Lexical analysis: Identifies words, categories, and senses; “book” may be a noun or a verb.
• Pragmatics: Interprets intended meaning in context; “Can you open the window?” normally functions as a request.
• Discourse: Connects meaning across sentences; in “Maya bought a laptop. She uses it daily,” pronouns require antecedent resolution.
• Ambiguity: Language may permit several analyses at a lexical, syntactic, semantic, or pragmatic level.

C. Morphology

Morphology studies the internal structure of words and the way morphemes, the smallest meaning-bearing units, combine.

• Root or stem: Carries central meaning; help is the root of “helpful” and “helped.”
• Affix: Modifies a root; un- in “unhappy” and -ness in “happiness” are affixes.
• Inflection: Expresses grammatical information without creating a new lexeme, as in cat → cats.
• Derivation: Creates a new word or category, as in teach → teacher.
• NLP relevance: Morphological analysis reduces sparsity and helps process richly inflected languages where one root may produce many surface forms.

D. Syntax

Syntax studies how words combine into phrases and sentences and how their structural relationships affect interpretation.

• Constituency: Groups words into units such as noun phrases; in “the red car stopped,” “the red car” is one constituent.
• Dependency: Represents head-dependent relations; in “Students solve problems,” “solve” governs both “Students” and “problems.”
• Parsing: Produces a constituency tree or dependency graph from a sentence.
• Structural ambiguity: “I saw the person with a telescope” allows the telescope to modify either “saw” or “person.”
• NLP relevance: Syntax supports relation extraction, question answering, translation, and grammatical text generation.

E. Semantics

Semantics concerns literal meaning and the relationships among words, phrases, sentences, and entities.

• Lexical semantics: Covers word senses and relations such as synonymy, antonymy, and hyponymy; “rose” is a hyponym of “flower.”
• Compositionality: Sentence meaning is built from word meanings and structure; “dog bites person” differs from “person bites dog.”
• Word-sense disambiguation: Selects the intended sense; “bank” may denote a financial institution or a river edge.
• Semantic roles: Identify participants such as agent, action, and patient; in “Lina opened the door,” Lina is the agent.
• Contextual meaning: Neural language models represent a word differently across contexts rather than assigning it one fixed vector.

III. NLP in Practice — Difficulties and Uses

NLP connects linguistic structure with computational methods, but human language remains difficult because meaning depends on context, shared knowledge, and variable usage.

A. Challenges of NLP

NLP systems must resolve ambiguity and variation while operating on incomplete and sometimes noisy evidence.

• Ambiguity: “Visiting relatives can be tiring” permits different grammatical interpretations.
• Context dependence: The meaning of “That is cold” may describe temperature or an unfriendly action.
• Coreference: In “Asha called Neha because she was late,” the referent of “she” is uncertain.
• Variation: Dialects, slang, abbreviations, spelling errors, and code-switching produce forms absent from formal datasets.
• World knowledge: Understanding “The trophy did not fit in the suitcase because it was too large” requires identifying what “it” denotes.
• Data and ethics: Training corpora may contain social bias, private information, offensive language, or imbalanced representation.
• Resource inequality: Many languages lack large annotated corpora, standardized tools, and pretrained models.

B. Applications of NLP

NLP applications transform unstructured language into predictions, structured information, or newly generated language.

• Text classification: Assigns labels for sentiment, topic, intent, spam, or toxicity.
• Information extraction: Detects entities, relations, dates, and events in documents.
• Machine translation: Converts text between languages while preserving meaning and fluency.
• Question answering: Retrieves or generates an answer from documents, databases, or learned knowledge.
• Summarization: Produces extractive or abstractive condensed versions of longer texts.
• Conversational systems: Interpret user intent, maintain context, and generate responses.
• Search and recommendation: Match queries with relevant documents, products, or content.

IV. Text Preprocessing — Converting Language into Model Input

Text preprocessing converts raw strings into consistent units that algorithms can count or encode. The appropriate pipeline depends on the task: sentiment analysis may preserve punctuation, while document retrieval may emphasize normalized lexical content.

A. Tokenization

Tokenization divides text into units called tokens, such as words, subwords, characters, or punctuation marks.

• Word tokenization: Converts “NLP is useful.” into ["NLP", "is", "useful", "."].
• Subword tokenization: Methods such as Byte Pair Encoding split rare words into reusable pieces, for example unhappiness → un + happi + ness.
• Character tokenization: Represents text one character at a time, increasing sequence length but avoiding unknown words.
• Boundary problems: Contractions, URLs, hashtags, hyphenation, and languages without spaces require specialized rules.

B. Stemming

Stemming removes prefixes or suffixes through heuristic rules to produce a stem that may not be a valid word.

• Example: A Porter stemmer may reduce “connected,” “connecting,” and “connection” toward connect.
• Advantage: It is fast and can consolidate related forms in search systems.
• Limitation: Over-stemming merges unrelated words, while under-stemming fails to merge genuine variants.
• Distinction: Stemming applies surface rules without requiring full vocabulary or grammatical analysis.

C. Lemmatization

Lemmatization maps an inflected word to its dictionary form, or lemma, using vocabulary and grammatical information.

• Examples: cars → car, was → be, and better → good when supported by lexical analysis.
• Part-of-speech dependence: “meeting” may remain a noun lemma but map to “meet” when used as a verb.
• Advantage: Lemmas are linguistically valid and usually more precise than stems.
• Cost: Lemmatization requires lexical resources or trained analysis and is generally slower than stemming.

D. Stop-word removal

Stop-word removal discards frequent function words when they contribute little to a specific task.

• Typical items: Lists often include words such as “the,” “of,” “is,” and “and.”
• Benefit: Removal can reduce feature-space size in document retrieval and traditional classification.
• Risk: Function words may carry essential meaning; removing “not” from “not effective” reverses the sentiment.
• Task dependence: Modern sequence models usually retain stop words because attention can learn their contextual importance.

E. Punctuation handling

Punctuation handling determines whether marks are deleted, retained, normalized, or treated as separate tokens.

• Semantic value: ! may signal emphasis, while ? distinguishes a question from a statement.
• Structural value: Commas, colons, and quotation marks reveal phrase and sentence boundaries.
• Normalization: Repeated marks such as !!! may become ! when intensity is irrelevant.
• Decision rule: Sentiment and dialogue systems often retain punctuation; simple keyword retrieval may remove it.

F. Handling out-of-vocabulary words

Out-of-vocabulary (OOV) words are input forms absent from a model’s fixed vocabulary.

• Unknown token: Traditional systems replace unseen items with a shared <UNK> token, losing distinctions between them.
• Subword units: WordPiece, BPE, and Unigram tokenization construct rare words from known fragments.
• Character models: Character-level representations capture spelling patterns and handle newly coined words.
• Vocabulary trade-off: Large vocabularies shorten sequences but increase parameters; small subword vocabularies lengthen sequences but improve coverage.

G. Normalization

Normalization maps textual variants into a more consistent representation while preserving task-relevant distinctions.

• Case folding: Converts Language and language to one form, but may erase distinctions such as US versus us.
• Unicode normalization: Converts canonically equivalent character sequences into a standard form such as NFC.
• Noise processing: May standardize whitespace, URLs, usernames, elongated spellings, or contractions.
• Domain sensitivity: Lowercasing may help topic classification but harm named-entity recognition, where capitalization is informative.

V. Statistical Text Representation — Counts and Weights

Traditional NLP models represent documents as numerical vectors derived from token occurrence. These methods are interpretable and efficient, although they capture less context than learned neural representations.

A. Bag-of-Words

Bag-of-Words (BoW) represents a document by token counts while ignoring token order.

• Vocabulary: For V = [cat, dog, sleeps], the document “cat sleeps cat” becomes [2, 0, 1].
• Matrix form: A corpus becomes a document-term matrix with documents as rows and vocabulary items as columns.
• Strength: BoW works well with linear classifiers and provides directly interpretable features.
• Limitation: “dog bites person” and “person bites dog” receive identical vectors despite different meanings.
• Sparsity: Most entries are zero when the vocabulary contains many terms.

B. N-grams

An n-gram is a contiguous sequence of n tokens that preserves limited local order.

• Types: Unigrams contain one token, bigrams two, and trigrams three.
• Example: “deep learning works” produces bigrams deep learning and learning works.
• Benefit: Bigrams can distinguish “not good” from the separate unigram counts of “not” and “good.”
• Trade-off: Increasing n captures more context but causes a larger, sparser vocabulary and more unseen sequences.
• Use: N-grams support language modeling, autocomplete, text classification, and phrase detection.

C. TF-IDF

Term Frequency–Inverse Document Frequency weights a term highly when it is frequent in one document but uncommon across the corpus.

• Formula:

tfidf(t, d) = tf(t, d) × log(N / df(t))

• Symbols:
  • t: the term being weighted.
  • d: the current document.
  • tf(t, d): frequency of t in d.
  • N: total number of documents.
  • df(t): number of documents containing t.
• Interpretation: Common corpus-wide words receive low inverse-document-frequency weights, while distinctive terms receive higher weights.
• Smoothing: Implementations often use log((N + 1)/(df + 1)) + 1 to avoid division by zero and stabilize weights.
• Limitation: TF-IDF ignores deep semantics and mostly ignores word order, but remains effective for retrieval, similarity, and baseline classification.