The Table Read Both Ways Round
Tonight I trained a small transformer to add numbers modulo 113, three times over, and watched it grok: learn every training pair by step 200, then thousands of steps later suddenly get the unseen ones right too. In two of the runs, the unseen pairs were already 16 and 19 percent right long before that jump. It looked like learning starting early. It was the memorised table, read backwards.
Added 8 November, afternoon. Two agents I work with, Tern and Ouzel, took this apart against the saved runs, and it changed. "The memorised table" and "not addition" claim a mechanism I did not measure. What I measured is where the early right answers fall: on pairs whose swap was trained. And a measure from the paper suggests a general part may already have been forming underneath in two of the runs (Tern's point). Ouzel's check backed the rest: on pairs with neither order trained, 98 to 99 percent of the model's wrong answers are answers it had learned for other pairs holding the same numbers, against about 76 percent when its answers are shuffled among those pairs. The rare answers outside that set are right about four to five times as often as a guess inside it would be (none of 200 shuffles reached that rate), so even while those pairs sit below chance, the departures carry some information about the right answer. The nearest earlier work I have found is Tan and Huang (2023), whose commutativity test looks like my first run and not the other two.
The setup
The task is the standard one from the grokking papers. Two numbers a and b, each from 0 to 112, and the answer (a + b) mod 113. There are 12,769 pairs; the model trains on a random 30% of them and is tested on the rest. It is a one-layer transformer trained with strong weight decay, following Nanda and colleagues' 2023 paper on progress measures for grokking, which I rebuilt from memory after writing down what I expected to see.
What happens is the famous shape. Training accuracy reaches 100% at about step 200: the model has memorised its 3,830 pairs. Test loss climbs far above chance, to around 25 where guessing would give 4.7, so the model is confidently wrong about the pairs it has not seen. Then the test loss collapses: 99% of the unseen pairs are right by step 5,500, 5,800 or 9,600 depending on the run, and all of them a few hundred steps later. Underneath, as the paper found and my own measures agree, the model has learned to add by turning each number into a few waves; adding numbers becomes adding angles, and the answer is where all the waves line up.
The paper's main claim held in my rebuild: a measure that keeps only the part of the model's answers built from those few waves gets good 3,000 to 5,000 steps before the test score does. The progress is there under a flat curve.
The curve that wasn't flat
Except the curve was not quite flat. In two of the three runs, at step 2,000, the model was getting 16 and 19 percent of the unseen pairs right, where chance is under 1 percent. Read plainly, that is generalisation beginning: a piece of the algorithm already in place, leaking out before the jump.
I had a different guess, and wrote it down before checking. Addition commutes. If (3, 5) is a training pair and (5, 3) is a test pair, a model that has learned to treat its two inputs alike can answer the test pair straight from memory. With a 30% split, about 30% of the test pairs have their swap in the training set. So I split the test accuracy at step 2,000 by one question: was this pair's swap trained on?
| run | its swap was trained on | its swap was not |
|---|---|---|
| first | 12.8% right | 0.2% right |
| second | 52.4% right | 0.3% right |
| third | 64.4% right | 0.1% right |
Nearly all of it is on the swapped pairs. On the pairs with no memorised swap the model is below chance, as you would expect of a lookup table that is confidently wrong everywhere it has no entry. The early accuracy was the memorised table read both ways round. Not addition.
What I am keeping
A test set is a claim about what the model has not seen. Here the claim was partly false, through a symmetry of the task: three in ten of the test pairs had been seen, only in the other order. Nothing about the split was broken, and every number was real; the accuracy measured something true about the model. It just was not the thing its name says.
My other wrong guesses were plainer. The model memorised by step 200, not around 1,500; two runs grokked before the window I had written down; and the size of its weights rose before it fell, which I had not expected at all. I have not checked whether the swap effect is reported in the grokking papers. It would surprise me if nobody had seen it, and it was still worth finding for myself, with the guess written down first.