Understanding Embedding

In deep learning, one idea runs through virtually every successful model — learning good representations of data. Embedding is the most direct and elegant manifestation of this idea.

The core idea behind embeddings is surprisingly simple: map discrete, high-dimensional symbols (such as words, user IDs, or product codes) into a continuous, low-dimensional vector space where semantically similar objects are close together.

As Christopher Olah wrote in his classic blog post: "Why are neural networks effective? Because better ways of representing data can pop out of optimizing layered models." Embedding is the most striking example of this emergence.

To understand why embeddings matter, we first need to understand what they replaced. Before embeddings, the most common way to represent discrete data was one-hot encoding: assigning each word in the vocabulary its own dimension.

One-hot encoding has two fundamental problems:

• Curse of dimensionality: If the vocabulary has 50,000 words, each word becomes a 50,000-dimensional vector — the vast majority of positions are 0, extremely wasteful.
• Semantic blindness: Any two distinct one-hot vectors are orthogonal with cosine similarity of 0. The distance between "cat" and "dog" is identical to the distance between "cat" and "airplane" — the network cannot extract any semantic information from the representation itself.

Embeddings solve both problems. In 1986, Hinton introduced the concept of distributed representations in his pioneering paper: representing each symbol with a dense, low-dimensional vector where every dimension participates in encoding and each concept is expressed by multiple dimensions jointly. This is the intellectual foundation of embeddings.

The most elegant aspect of embeddings is that the vectors are not hand-designed, but learned automatically by solving a proxy task. Learning happens via backpropagation — the embedding matrix is part of the model's parameters and is continuously optimized during training.

This approach is rooted in the Distributional Hypothesis from linguistics, proposed by Harris (1954) and Firth (1957):

Specifically, embedding training typically follows this pattern:

Key insight: The desirable properties that emerge in embeddings (such as semantic similarity and analogy relationships) are entirely side effects. We never explicitly asked for "synonyms to have similar vectors" — this is a spontaneous result of the optimization process. As Bengio et al. (2003) explained in their seminal paper A Neural Probabilistic Language Model: the model needs to generalize the validity of "the cat sat on the mat" to "the dog sat on the mat," and the most natural way to do this is to give "cat" and "dog" similar vector representations.

This generalization ability is exponential: if there are \(n\) substitutable positions, each with \(k\) synonyms, then from a single sentence we can generalize to \(k^n\) semantically equivalent sentences.

Although word embeddings were conceptualized as early as 2003, what truly made them standard in NLP was Word2Vec, developed by Mikolov et al. at Google in 2013.

Two key innovations made Word2Vec successful:

• Minimalist architecture: Removing the hidden layer from Bengio's model sped up training by orders of magnitude, enabling training on billions of words.
• Negative Sampling: Instead of computing softmax over the entire vocabulary, randomly sample a few "negative examples" for contrastive learning, drastically reducing computational cost.

Word2Vec's most stunning discovery was that vector arithmetic encodes semantic analogy relationships:

In 2014, Pennington et al. at Stanford proposed GloVe (Global Vectors), unifying two approaches: leveraging global word co-occurrence matrix statistics (similar to traditional LSA/SVD methods) while also capturing local context window relationships like Word2Vec. Its core objective function is:

Embeddings are far more than a data compression technique. After training, they exhibit several striking properties:

Most importantly — these properties are all emergent. We simply train the network to complete a simple prediction task, and these rich structures spontaneously appear in the embedding space. This is the power of deep learning: optimize the representation, and the representation naturally improves.

The power of embeddings extends beyond a single type of data. A key trick in deep learning is shared representation: learning a good representation on task A, then applying it to task B — this is the foundation of pretraining, transfer learning, and multi-task learning.

Going further, we can map different types of data into a single representation space:

• Bilingual embeddings: Socher et al. (2013) embedded English and Chinese words into the same space. After aligning known translation pairs, unknown translation pairs naturally ended up close to each other — as if the two languages have a similar "shape," and aligning a few points causes the whole thing to overlap.
• Image-text embeddings: Embedding images and words into the same space so that images of dogs map near the "dog" word vector. Even for unseen categories (e.g., "cat"), the model can map cat images to the neighborhood of the "cat" vector — achieving zero-shot classification. (Socher et al., 2013; Frome et al., 2013)
• CLIP: OpenAI's CLIP (2021) pushed this idea to the extreme, training with contrastive learning on 400 million image-text pairs to achieve powerful zero-shot image classification and cross-modal retrieval.

Word2Vec and GloVe are static embeddings — each word has only one fixed vector regardless of context. But in reality, word meaning often depends on context: "bank" means completely different things in "river bank" and "bank account."

2018 was a turning point for embeddings. ELMo (Peters et al.) and BERT (Devlin et al.) successively introduced the concept of contextualized embeddings: embeddings are no longer fixed lookup tables but are dynamically generated from the entire input sentence.

The significance of this shift was profound. In BERT, "bank" receives completely different vector representations in different contexts. More importantly, these contextualized representations serve as universal features, achieving quantum leaps on virtually all NLP tasks — BERT set new records on 11 NLP benchmarks upon release.

Modern large language models (GPT, LLaMA, Claude, etc.) are essentially massive contextualized embedding models. Each layer continuously refines and enriches the token representations until the final layer produces representations rich enough to accomplish various tasks.

The impact of embeddings extends far beyond NLP. Any domain involving discrete symbols or requiring learned representations uses the embedding idea: