Attention is All You Need: Introduction to the Transformer Architecture

After learning about the evolution of Conversational AI, I was intrigued to learn about the 3 most influential words in the AI space. Yes, I’m talking about LLMs aka the Large Language Models. But to get to that we will first go through the significant breakthrough that happened in 2017 - The introduction of the Transformer architecture by Vaswani et al. in the paper "Attention is All You Need.”

I wanted to know how the title of the research paper came into place which helped me understand the transformer architecture. The research paper introduced a new deep-learning architecture called the Transformer. This architecture solely relied on an attention mechanism – a technique that lets the model focus on specific parts of the input sequence when processing it (wait a minute, we’ll get to this). Vaswani and other researchers explained that by ditching recurrent neural networks (RNNs) and convolutional layers, the Transformer achieved impressive results in machine translation tasks.

Now, what are Recurrent Neural Networks (RNNs) and why did the researchers discard them?

Imagine you're reading a sentence. A regular computer program might process each word one by one, without any memory of what came before. An RNN is different. It’s like you reading the sentence - you remember the previous words to understand the current one.

Unlike traditional neural networks where each input is independent, RNNs have an internal memory that allows them to process information based on what they've seen before.

Now, RNNs must have certain drawbacks because of which the Transformer architecture came into place, right? Let’s understand the drawbacks that the Transformer aimed to address:

1. Long-term Dependencies: RNNs struggle to capture relationships between words that are far apart in a sequence. Imagine a very long sentence where the meaning of a word at the beginning depends on something mentioned much later in the sentence. RNNs might have trouble remembering that far back.
2. Sequential Processing: RNNs process information step-by-step, which can be slow for long sequences.

This article by the Financial Times explained this beautifully via Visual Storytelling.

The Transformer architecture addressed these issues by focusing on two things -

1. Attention Mechanism: This allows the model to focus on specific parts of the input sequence that are most relevant to the current word being processed. It's like being able to highlight important parts of a sentence while reading, instead of having to read everything in order.
2. Parallel Processing: Unlike RNNs, the Transformer can analyze all parts of the sequence simultaneously, making it faster for longer sequences.

By ditching RNNs and relying on the attention mechanism and parallel processing, the Transformer architecture achieved great results in machine translation tasks. It showed that you don’t necessarily need complex RNNs if you have a powerful attention mechanism to capture relationships between words.

The Transformer Architecture laid the foundation for today's sophisticated LLMs. People quickly recognized its potential, leading to the creation of increasingly powerful models such as GPT (Generative Pretrained Transformer) by OpenAI.

So, now we know why the T in GPT stands for Transformer.

In the next post, I’ll try to decode LLMs and how they work.