Two paths to the same model

Last lesson, the bigram model was a count matrix $N$. To get probabilities, we normalized each row.

Here is the same model rebuilt as a tiny neural network:

• Inputs: a one-hot vector $\mathbf{x} \in \{0, 1\}^{27}$, with a single 1 at the index of the current character.
• Weights: a single matrix $W \in \mathbb{R}^{27 \times 27}$.
• Forward pass: logits $\boldsymbol{\ell} = \mathbf{x}^\top W$, then probabilities $\mathbf{p} = \mathrm{softmax}(\boldsymbol{\ell})$.

Because $\mathbf{x}$ is one-hot, $\mathbf{x}^\top W$ is just row $i$ of $W$, the row picked out by which character is currently 1. So the model’s prediction for “what comes after character $i$” is $\mathrm{softmax}(W[i, :])$. Same shape as the count matrix’s row, computed differently.

This is a one-layer neural network. Same input shape, same output shape, same row-as-distribution structure. The only difference: $W$ is trained by gradient descent against a loss, not filled in by counting.

The loss is the negative log-likelihood of the corpus under the model:

where $N$ is the total number of bigrams in the corpus. Average per-bigram, in nats. This is the same loss every language model on earth is trained on, all the way through to GPT-4; only the architecture between the input and the softmax changes.

Two facts to internalize before we run anything:

1. The minimum of $\mathcal{L}$ is achieved exactly when $\mathrm{softmax}(W[i, :])$ matches the empirical distribution $N_{i, \cdot} / \sum_k N_{i, k}$ for every row. Setting $\nabla_W \mathcal{L} = 0$ with one-hot inputs gives this in two lines.
2. So the SGD-trained model and the count-table model have the same minimizer.

That means we should expect them to converge to each other. Let’s watch.

Below: two heatmaps of the same shape, both showing a 27×27 distribution. The left panel is $P(j \mid i)$ from raw counts (with light add-$s$ smoothing). The right panel is $\mathrm{softmax}(W[i, :])$, with $W$ initialized to small random values.

Press Auto and watch the right panel. It starts as a flat grey field; every row of $W$ is near zero, so every softmax row is near uniform. As gradient descent runs, the right panel grows the same dark cells as the left, in roughly the same places, with the same intensities. After a few hundred steps the convergence readout |count − sgd| drops below 0.005 and the panels are visibly indistinguishable.

The point is not that this is impressive (the model has 729 parameters and a closed-form optimum). The point is that the same model, expressed two ways, lands in the same place. Counting and gradient descent are not competing approaches; they are two algorithms for finding the same maximum-likelihood estimate. The neural-network framing wins as soon as the model gets too complex for closed-form counting, which is the next lesson, and the lesson after that, and the rest of this course.

Hit the Sample a name button in the widget above. The model starts at the · token, looks at its row, samples a next character from that row’s distribution, and repeats until it samples · again. Some samples will look name-like (emma·, eva·); most will be garbage (xrnxg·).

This is the entire generation algorithm. Look up a row. Sample from its distribution. Repeat. Sampling text from a transformer at the end of this course is the same loop; the model that produces each row is just enormously more sophisticated.

Look at the count panel. The q row in our small corpus is sparse; almost no observed q→? bigrams. Without smoothing, almost every cell in the q row would be exactly zero, which means $\log P = -\infty$, which means the loss is undefined the moment the test set contains any never-seen bigram.

The fix on the count side is add-$s$ smoothing: pretend every cell got $s$ extra observations before normalizing. Every probability becomes nonzero; the rare ones become small but finite. Crank the smoothing slider up and the count panel visibly flattens. At $s = 20$, every cell is close to uniform.

Now do the equivalent on the SGD side. L2 regularization adds a $\tfrac{\lambda}{2} \|W\|^2$ term to the loss, which gradient descent translates into a “pull every weight toward zero” force on every step. Pulling $W$ toward zero pulls $\mathrm{softmax}(W[i,:])$ toward uniform. Crank the L2 $\lambda$ slider up and watch the right panel flatten in exactly the same way as the left.

These are the same intervention seen from two algorithmic angles. On the count side it’s a Bayesian prior (every bigram starts with $s$ pseudo-observations). On the SGD side it’s a penalty on weight magnitude. Both pull predictions toward uniform; both reduce overfitting; both correspond, formally, to the same MAP estimate under a symmetric Dirichlet prior. You will reach for one or the other constantly throughout the course; they are two interfaces to one idea.

Three things are now in the toolkit forever.

Softmax of a logit row is the language-model output shape. Every language model in this course (bigram, MLP, RNN, transformer) outputs a probability distribution over the vocabulary by softmaxing a vector of length $|V|$. The bigram NN does it with a single weight matrix; the transformer does it after twelve layers of attention; the math at the output is identical.

NLL is the loss every language model trains on. “Average negative log probability of the next token, given context.” That sentence describes what the bigram NN, the Bengio MLP, the LSTM, and the transformer all minimize. Different architectures, same objective.

Regularization is the same idea everywhere. Whether it shows up as smoothing on a count table, L2 penalty on weights, dropout, label smoothing, or weight decay in AdamW, the underlying move is “don’t let the model become overconfident from finite data.” We will see four more flavors of this before the course is over, and they will all be the same trick.

Counting was a warmup. The next lesson upgrades the output side (loss, perplexity, sampling); after that we upgrade the input side (context, embeddings, architecture). The rows of the matrix go away. The shape of a probability distribution over the next token does not.