Saved on this device

Follow your curiosity

What discovery has been shared with you?

FACTREE gives you something interesting to discover every day — and something worth talking about together.

Start with one fact. Explore it, go deeper, then follow whichever branch catches your imagination.

Choose subjects for a surprise

Exploring any topic

Begin your discovery

Your next discovery is one click away.

Choose one or more subjects above, or leave Any Topic selected and let curiosity decide.

Technology

Understanding Attention Mechanisms in Transformer Language Models

Quick fact

The attention mechanism was introduced in the 2017 paper 'Attention Is All You Need', which led to the Transformer architecture that powers modern models like GPT-4 and BERT.

Why this is interesting

You've probably used a chatbot or translator, but have you ever wondered how the model knows which words are most important in a sentence? The secret lies in a concept called 'attention'—and it's not the kind you pay in class.