Attention Is All
A thought experiment about a whale, a sentence, and the one budget you cannot refill.

You came straight here, so you never saw the front page. There is a whale crossing it right now. It will keep crossing whether anyone looks or not.
Did you see it move? If you did not, here is the strange part. The whale was on your screen the entire time, and for you it was not there.
William James put it plainly in 1890: "My experience is what I agree to attend to." I want to take that sentence more literally than he probably meant it, and I want a machine to help.
A word reaches back
Read this sentence, and resolve the word it without trying.
The whale did not cross the mountain because it was too …
One word changes at the far end of the sentence, and it becomes a different animal, or a different continent's worth of rock. You did not reread anything. Something in you reached back across the sentence and took the part it needed.
That reaching is the whole subject of this essay.
The line and the room
Imagine a line of nine people. Each holds one word and can only speak to the next person along. To understand the sentence, the last person has to rely on what was passed down the line.
For years, this was how machines read. Recurrent networks passed a summary from word to word. Watch what happens to whale by the time the message arrives. Long sentences were a game of telephone.
Now remove the line. Put all nine people in one room, and let anyone look directly at anyone. Nobody has to remember what was whispered, because nothing was whispered. It is all still there to look at.
Now only one question is left: when it looks around the room, how much of its looking goes to each person? That distribution is attention, and it does the work the line used to do.
Attention was not a new idea in 2017. Bahdanau, Cho and Bengio had bolted it onto recurrent translators three years earlier. What the eight authors of Attention Is All You Need proposed was to keep the attention and throw away the rest, "dispensing with recurrence and convolutions entirely."
The new word in that title was all.
The budget
Here is the mechanism, in the form they published it.
Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V
Skip everything except softmax. It takes any set of scores and turns them into shares that add up to exactly one. This means attention is zero-sum by construction: nothing can get more of it unless something else gets less.
So here is the experiment. This plate has four things in it. Each one is printed only as darkly as the attention it receives. Move your pointer across it. The numbers are a real softmax, computed as you move.
Nothing on that plate is erased when you look away. The file is the same. But turn the temperature down and you get a mind that sees one thing and nothing else. Turn it up and every element sits at the same pale grey. You can have everything in view, as long as you are not really looking at any of it.
You cannot attend harder. You can only attend elsewhere.
Look up
Look at the paragraphs above this one. They have faded.
That is not a metaphor. A few lines of code on this page are taking the ink out of whatever leaves the middle of your screen. Those paragraphs still exist. You could scroll back and they would darken again. I am only showing you what was already happening.

Where the analogy breaks
A model's attention is nothing like a scarce resource in practice. The original paper runs eight heads in parallel, stacked in six layers. The next model will run more. It spends attention the way water spends downhill, and it does not get tired, and no one has ever made it look away by sending it a notification.
You get one head, about sixteen waking hours, and no way to buy more. The softmax that was a design choice for them is a hard physical limit for you. When people talk about the attention economy they mean this quite exactly: something that sums to one, with many parties bidding on the shares.
So I think the title was accidentally truer about us than about machines. A transformer needs attention plus a good deal else. You, in the only sense that you can check from the inside, are made of it. What you attended to today is what today was.
Ten seconds
There is one more experiment, and you are the only one who can run it. Hold the button down. For ten seconds everything else on this page goes away, and there is only the whale that you might have missed on your way here.
That was ten seconds, and you will not get them back. You gave them to a drawing of a whale, and for a little while it was all there was.