Tokenization and Text Embedding

Ever wondered how modern AI systems understand human language? At their core, Large Language Models (LLMs) never actually see or read words as humans do. Instead, they operate exclusively on numerical representations. Because LLMs are based on artificial neural networks (ANNs), they can only process numerical data. To accommodate textual inputs, two steps are necessary: tokenization and text embeddings. Tokenization involves breaking down input text into smaller chunks, such as words or sentences. Text embeddings represent these words with numerical vectors that convey their meaning and relationship to other words, allowing the model to understand the context and nuances of language.

Tokenization

Before text can be turned into mathematical vectors, it must first undergo tokenization. Tokenization is the process of chopping continuous raw text into smaller, discrete building blocks called tokens. These tokens can represent whole words, subword pieces, or individual characters and punctuation marks.

At a fundamental level, tokenization functions much like a dictionary lookup: the system maintains a fixed vocabulary table where each unique token is assigned a specific numerical ID (for example, "apple" might be ID 4821, while "ing" might be ID 312). When an AI model receives raw text, it segments the input into these known tokens and looks up their corresponding integer IDs in its vocabulary table. These discrete integer IDs serve as the initial bridge between raw human characters and machine-readable data, transforming unstructured string sequences into orderly arrays of numbers ready to be passed to the embedding layer.

Note that these initial token IDs carry no semantic understanding on their own: ID 4821 ("apple") is mathematically no closer to ID 4822 ("banana") than to any random word. The actual understanding of meaning happens in the next stage, where the embedding layer translates these simple index numbers into rich geometric vectors.

How Text Embeddings Work

Note: This article uses a simplified geometric approach for teaching purposes. In this guide, we use neat axes like age or gender to keep things easy to visualize. Real embedding dimensions aren’t clear-cut like this. Real models use hundreds or thousands of abstract mathematical dimensions.

Let’s take a closer look at how we can embed words related to gender and age using 2D numerical vectors. As illustrated in the figure, similar age categories and genders tend to cluster together, demonstrating a clear pattern of semantic relationships.

Text	Text Embedding
boy	[1,1]
man	[1,3]
child	[3,1]
adult	[3,3]
girl	[5,1]
woman	[5,3]

Let’s enhance our word embeddings by adding an additional dimension to capture how words like prince, king, and queen relate to the rest of the dataset. This new dimension, which we’ll call royalty, allows these terms to cluster together in a meaningful way. Take a look at the numerical vectors that result from this addition: they now have three dimensions: two for capturing word meanings within the existing space, and one for representing their membership in the royalty category.

Text	Text Embedding
boy	[1,1,0]
man	[1,3,0]
child	[3,1,0]
adult	[3,3,0]
girl	[5,1,0]
woman	[5,3,0]
prince	[1,1,2]
king	[1,3,2]
queen	[5,3,2]

Just like individual words, entire sentences can be embedded, resulting in a single vector that captures their semantic meaning and context. This embedding enables meaningful relationships between sentences to emerge, allowing for the identification of clusters with similar meaning. For instance, consider the following illustration, which demonstrates how sentences with analogous meanings are grouped together.

How Real Embeddings Differ

While our 2D and 3D visual models are great for building intuition, real-world embeddings used in modern AI systems work a bit differently behind the scenes:

• High Dimensionality: To map out complex relationships between words and sentences, we need far more than three dimensions. Real-world LLMs use vectors with hundreds or thousands of dimensions (typically 768, 1,536, or more) to capture language nuances.
• Abstract, Hidden Axes: Unlike our toy examples where we assigned simple labels like “age” or “gender,” real embedding dimensions do not have clear human names. They are learned automatically by the neural network during training, where a single dimension might encode a blend of context, tone, and grammar all at once.
• Contextual Meaning: In simple static models, a word always gets the exact same list of numbers. Modern LLMs create contextual embeddings, meaning the word “bank” gets a different set of numbers in “river bank” than it does in “money bank.”
• Normalized Continuous Values: Production embeddings typically consist of decimal floating-point numbers bounded between -1.0 and 1.0 (often normalized to unit length).
• Distance Calculations: Rather than checking simple coordinates, AI models calculate geometric proximity using mathematical distance metrics (like Cosine Similarity) to figure out how close two meanings are in vector space.

Final Thoughts

In essence, modern LLMs bridge the gap between human language and machine understanding through a sophisticated two-step process. By first tokenizing text into discrete numerical IDs and then transforming these IDs into rich, multi-dimensional embeddings, LLMs can grasp the intricate semantic relationships and contextual nuances of language. While simplified models offer an intuitive glimpse, real-world embeddings leverage high-dimensional, abstract vectors and continuous values to enable LLMs to process and interpret human communication with remarkable accuracy.