Spark & Anvil Privacy
A warm workshop library with bookshelves and a friendly brass sorting machine that sorts little paper slips, two tweens feeding it slips to see what it guesses next

🧠

TokenTales

How does AI guess the next word? By counting. Come see.

🔮 Open the model console →

The secret behind "AI writing"

A language model — the thing behind chatbots — has one job: guess the next word. It learns by reading a LOT of text and counting which word tends to follow which. Then, to "write," it just keeps picking the word that most often came next. That is (almost) the whole trick.

  1. Train. TokenTales builds a real, tiny "n-gram" model on a public-domain rhyme — right here, in your browser.
  2. Predict. You see a phrase and guess which word the model thinks comes next.
  3. Reveal. The counts appear as a bar chart — you see it is just counting, not magic.
  4. Watch it write. Give it a starting word and watch the model continue, one most-likely word at a time.

No chatbot, no internet, no big model — everything runs on your device. Real chatbots do the same task, guessing the next word, but they are neural networks trained on billions of words, not a table of counts.

Meet the crew — the pieces of a language model

  • Token — a single word (or piece of a word); a model reads and predicts one token at a time.
  • Tally — the counter: how many times each word followed another (the model's whole "memory").
  • Gram — the look-back: a bigram uses the last 1 word, a trigram the last 2 — more look-back, smoother writing.
  • Nova — the guide: the model has no idea what words mean; it predicts the most common next word. Knowing that is knowing how AI really works.
Where this idea comes from

Counting what comes next

In 1913 the Russian mathematician Andrey Markov took the first 20,000 letters of Pushkin’s poem Eugene Onegin and counted, by hand, how often a vowel followed a vowel and how often a consonant did. The chance of the next letter depended on the letter before it — a chain of guesses, each looking back one step.

One thing to hold: A short text and a table: for each word, the words that came after it, with a tally for each.

Why it matters: In 1948 Claude Shannon used the same idea on words. He opened a book, read a word, found that word again somewhere else, and wrote down the word that came after it, again and again. The result sounded almost like English. TokenTales keeps the same kind of table, but it always picks the most-counted next word instead of one by chance — which is why it gets stuck in loops.

What we don’t know: Today’s chatbots are trained on the same task — guess the next word — but they learn patterns in a neural network instead of keeping a table of counts. People still argue about how much those patterns add up to understanding.

Dates:

  1. 1906 Markov first writes about chains of events that look back one step.
  2. 1913 He counts the letters of Eugene Onegin.
  3. 1948 Shannon builds English sentences from word pairs.

Sources: A. A. Markov, “An example of statistical investigation of the text Eugene Onegin concerning the connection of samples in chains” (1913; English translation, Science in Context 19(4), 2006) · C. E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal 27 (1948) · Wikipedia, “Markov chain”

What next? Pick a door

Chosen from what this app is about. What you open is remembered on this device only.