Character Tokenizer & Vocabulary Mapping

The Big Idea

Following the architectural plan introduced in Lesson 1.1, our first task is to bridge human language and tensor math. Computers and neural networks are high-speed numerical calculators: they understand additions and matrix multiplications, but they have zero concept of the letter 'H' or the word 'Apple'. Before a neural network can process a single word, human text must be converted into numbers. A tokenizer is the secret decoder ring that translates raw characters into discrete integer IDs and back again.

Commercial LLMs like GPT-4 use Byte-Pair Encoding (BPE), which compresses text into 50,000 subwords. While efficient for training on trillions of web pages, BPE introduces confusing tokenization artifacts and hides the math behind complex heuristics. By building a character-level tokenizer on Shakespeare's plays, our entire vocabulary is exactly 65 characters: every letter, punctuation mark, and newline.

We build two simple lookup dictionaries: stoi (string to integer) and itos (integer to string). Encoding transforms a string like 'Dropout' into [16, 43, 50, 51, 50, 56, 55]. Decoding reconstructs those integer IDs back into readable text. Once converted, the entire corpus becomes a 1D PyTorch tensor of integers ready for training.

Architecture Blueprint & Tensor Mechanics

The tokenizer extracts all unique characters, sorts them deterministically, and builds bidirectional mapping dictionaries before converting strings into PyTorch LongTensors.

import torch

# Raw training text (e.g. Tiny Shakespeare)
text = "First Citizen:\nBefore we proceed any further, hear me speak."

# 1. Extract sorted unique characters across the corpus
chars = sorted(list(set(text)))
vocab_size = len(chars)
print(f"Vocabulary Size: {vocab_size}")

# 2. Build bidirectional lookup dictionaries
stoi = {ch: i for i, ch in enumerate(chars)} # char -> integer
itos = {i: ch for i, ch in enumerate(chars)} # integer -> char

# 3. Encoder: takes a string and outputs a list of integer IDs
encode = lambda s: [stoi[c] for c in s]

# 4. Decoder: takes a list of integers and outputs the decoded string
decode = lambda l: ''.join([itos[i] for i in l])

# Verification test
sample_text = "Dropout Developer"
encoded_tokens = encode(sample_text)
decoded_text = decode(encoded_tokens)
print(f"Original: '{sample_text}'")
print(f"Encoded:  {encoded_tokens}")
print(f"Decoded:  '{decoded_text}'")

# 5. Convert the entire dataset into a 1D PyTorch integer tensor
data = torch.tensor(encode(text), dtype=torch.long)
print(f"Dataset tensor shape: {data.shape}, dtype: {data.dtype}")