Session 06💬

NLP & Word Embeddings

NLP tasks and linguistic levels, sparse text representations, distributional semantics, Word2Vec, evaluation, and contextual meaning.

Learning Objectives

✓Recognize the major linguistic levels and downstream NLP tasks
✓Compare one-hot, bag-of-words, TF-IDF, and dense word vectors
✓Explain distributional semantics and the Word2Vec training signal
✓Distinguish Skip-gram from CBOW and intrinsic from extrinsic evaluation
✓Explain why one fixed vector cannot represent every meaning of an ambiguous word

What NLP Systems Must Represent

Natural Language Processing studies how computers represent and act on human language. Its history moves from hand-written rules, through statistical models, to neural systems. The stack ranges from sounds and word form through syntax and sentence structure to context and meaning. Tasks include tagging, parsing, word segmentation, coreference, sentiment analysis, summarization, question answering, translation, and language modeling. A model can succeed on a benchmark while failing at another linguistic level, so name the exact task and output.

Sparse Baselines: One-Hot & Bag-of-Words

A one-hot vector assigns one vocabulary dimension to each word. A bag-of-words document vector counts those dimensions and ignores order. These representations are simple, transparent, and often strong baselines, but two related words remain orthogonal and the vector grows with the vocabulary. Every occurrence of a word has the same representation, so syntax, local context, and polysemy are lost. Sparse baselines remain useful because they reveal whether a more complex model adds real value.

TF-IDF: Local Importance, Global Rarity

Term frequency measures how often term $i$ occurs in document $j$ . Inverse document frequency downweights terms that appear in many documents. Their product highlights words that are frequent in one document but rare across the collection. TF-IDF is easy to compute and excellent for retrieval or linear text classifiers, yet it still represents statistics rather than the contextual meaning or word order of a sentence. The log and smoothing conventions vary by library, so record the exact definition used.

\operatorname{tfidf}(i,j)=\operatorname{tf}(i,j)\cdot\log\frac{|D|}{\operatorname{df}(i)}

A term is informative when it is locally frequent and globally uncommon.

Distributional Semantics

The distributional hypothesis says that a word's meaning is reflected by the words that occur nearby. One family of methods constructs a word-context co-occurrence matrix and factorizes it with SVD or a related technique. Another family learns vectors by predicting a center word from context or context words from a center. Both turn words into dense distributed representations, where geometry can encode similarity instead of assigning each word an unrelated axis.

Word2Vec: Skip-gram & CBOW

Word2Vec scans a large corpus with a context window. Skip-gram uses the center word to predict surrounding words and can learn rare-word relations well but creates more training pairs. Continuous Bag-of-Words averages or combines context to predict the center and is usually faster. A softmax based on vector dot products defines the probability; training raises scores for observed pairs and lowers them for alternatives. Practical implementations use approximations such as negative sampling because a full vocabulary softmax is expensive.

P(o\mid c)=\frac{\exp(u_o^{\top}v_c)}{\sum_{w\in V}\exp(u_w^{\top}v_c)}

Skip-gram learns center vector

v_c

and outside vectors

u_o

from context prediction.

Pretrained Embeddings & Similarity

Word2Vec, GloVe, and fastText provide pretrained vectors learned from large corpora. Word2Vec is predictive, GloVe uses global co-occurrence statistics, and fastText represents subword units, which helps with morphology and uncommon words. The best source corpus is one close to the target domain. Cosine similarity compares vector direction rather than magnitude; analogies and nearest neighbors are useful diagnostics, not proof that the embeddings are fair or sufficient for a downstream task.

\cos(a,b)=\frac{a^{\top}b}{\|a\|_2\,\|b\|_2}

Cosine similarity is 1 for the same direction, 0 for orthogonal vectors, and -1 for opposite directions.

Cosine similarity without a libraryPython

import numpy as np

def cosine(a, b):
    a = np.asarray(a, dtype=float)
    b = np.asarray(b, dtype=float)
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

similarity = cosine(embedding['doctor'], embedding['physician'])

Intrinsic vs. Extrinsic Evaluation

Intrinsic evaluation probes the representation directly with word similarity, analogy, or another intermediate task. It is fast and helps diagnose geometry, but it matters only when it correlates with the actual application. Extrinsic evaluation replaces the embedding inside a downstream task such as named-entity recognition and measures task performance. It is slower and can hide whether the embedding or another subsystem caused the change. A convincing study uses both: intrinsic tests to understand behavior and extrinsic tests to establish utility.

Ambiguity Requires Contextual Embeddings

A static embedding assigns one vector to a word type, so "bank" must average its financial and river meanings. A contextual model instead creates a representation for each token occurrence based on surrounding text. This is the bridge to Transformers and BERT: the vocabulary embedding is only the starting point, then self-attention changes the token vector according to context. Contextualization improves sense separation but does not remove dataset bias, domain shift, or the need for downstream evaluation.

←PreviousRecurrent Neural Networks →NextAttention & Transformers