In this Article, We are gonna look into the various aspects of exploring and understanding the rapidly evolve of the Generative Artificial Intelligence (AI).
Historical Context: Seq2Seq Paper and NMT by Joint Learning to Align & Translate Paper
Seq2Seq Paper :
The Seq2Seq Paper, officially titled "Sequence to Sequence Learning with Neural Networks" by Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, was published in 2014. This paper introduced the Encoder-Decoder Architecture to solve a seq2seq task, like Language translation, text summarization, Question and Answering system. Here, we are gonna use Recurrent Neural networks to transform input sequences to output sequences.
Problem with this Approach :
The major issue with this encoder–decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector. It may be difficult for the neural network to compress the long sentences into fixed-length as it may lead to the decrease of the performances.
Drawbacks :
Long - term dependency.
Slow processing.
NMT by Joint Learning to Align & Translate Paper
The "NMT by Joint Learning to Align & Translate Paper" by Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio, was published in 2015. This was like an Upgradation of the previous Seq2Seq model. This paper introduced the Attention Mechanism to overcome the drawbacks of Long-term dependencies.
Attention Mechanism :
The model can give attention more on important words based on the weights, leading to more accurate translations.
The introduction of the Transformer model, was published "Attention is All You Need" by Google in 2017, which solves the sequential training problem of earlier architecture by removing the need of RNN cells completely.
The Transformers Consists of both Encoder-Decoder Architecture and Attention Mechanism which was introduced in 2015.
Advantages of Transformers
Transformers have emerged as a dominant architecture in the field of natural language processing (NLP) for several compelling reasons:
Scalable and Parallel training: Unlike recurrent neural networks (RNNs), Transformers are highly scalable and can handle input sequences of variable length, which is essential for processing long sequences. Transformers can process tokens in parallel. It accelerates training and inference, making transformers more efficient for large-scale NLP tasks.
Capturing Long-Range Dependencies: Transformers leverage self-attention mechanisms to capture global dependencies between tokens in a sequence. This allows them to effectively model long-range relationships, which is crucial for tasks like language translation and text summarization.
Multi Model : Transformers can take text data, image data, audio, video data at a time and process the output which makes it as a multi- model.
Revolutionized NLP with LLM: Transformers have consistently demonstrated state-of-the-art performance on a wide range of NLP benchmarks and tasks including Language translation, Text generation, Understanding context, Question answering, and Sentiment analysis. Their superior performance has made them the go-to choice for many NLP applications.
Accelerated GenAI : Before Transformers, there are various problems of Long-term dependency , slow processing and more but This approach has significantly boosted performance on various NLP tasks and facilitated rapid development in the Generative Artificial Intelligence field.
Explain the working of each transformer component.
Input Embeddings: The input embeddings are the initial representations of the input tokens (words or subwords) in a sequence. These embeddings are typically learned during training using techniques like word embeddings (e.g., Word2Vec, GloVe) or subword embeddings (e.g., Byte Pair Encoding, WordPiece). Each token is represented as a high-dimensional vector in an embedding space.
Positional Encodings: Since transformer models lack recurrence or convolution to capture sequential order, positional encodings are added to the input embeddings to convey the position of each token in the sequence. These positional encodings can be learned or predefined and are added element-wise to the input embeddings.
Encoder: The encoder is responsible for processing the input sequence and covert into numerical representations. It typically consists of multiple identical layers, each containing two main sub-components: 1. Self-attention with multi-head 2. Feed Forward ANN Encoder - Great at Understanding text.
Decoder: The decoder generates an output sequence based on the representations learned by the encoder. Like the encoder, it consists of multiple identical layers, each containing three main sub-components: 1. Masked self-attention 2. Cross attention with multi-head 3. Feed Forward ANN Decoder - Great at Generating text.
Output Layer: The output layer of the transformer model computes the probability distribution over the vocabulary for each token in the output sequence. Typically, this involves applying a softmax function to the final representations generated by the decoder, producing the likelihood of each token in the vocabulary.
How is GPT-1 trained from Scratch?
GPT-1, or "Generative Pre-trained Transformer 1," is a variant of the transformer architecture specifically designed for generating natural language text. It was introduced by OpenAI in 2018. Here's how GPT-1 is trained from scratch:
Pre-training Objective: Like BERT (Bidirectional Encoder Representations from Transformers), GPT-1 follows a pre-training and fine-tuning paradigm. However, unlike BERT, which employs a masked language modeling (MLM) objective to learn bidirectional representations, GPT-1 uses an autoregressive language modeling objective. This means that during pre-training, the model is trained to predict the next token in a sequence given the preceding context.
GPT-1 is pre-trained on a large corpus of text data, typically consisting of a diverse range of sources such as books, articles, and websites. The dataset is tokenized into subword units (e.g., Byte Pair Encoding) to handle out-of-vocabulary words and improve generalization.
Architecture: GPT-1 consists of a stack of transformer decoder layers. Each layer contains self-attention mechanisms, cross attention mechanism and feedforward neural networks, similar to the decoder in the standard transformer architecture.
Training Procedure: GPT-1 is trained using a variant of the stochastic gradient descent (SGD) algorithm called Adam. The model is trained iteratively over multiple epochs, with each epoch consisting of batches of input sequences sampled from the pre-training dataset. The parameters of the model are updated based on the gradients of the loss function computed with respect to the model's predictions.
Evaluation: During training, the model's performance is monitored using metrics such as perplexity, which measures how well the model predicts the next token in a sequence. Lower perplexity values indicate better performance.
