Gemma 3 Technical Report

Today's paper introduces Gemma 3, the latest addition to Google DeepMind's family of open language models. This multimodal model ranges from 1 to 27 billion parameters and introduces vision understanding capabilities, wider language coverage, and longer context handling of at least 128K tokens.

Method Overview

Gemma 3 builds upon the decoder-only transformer architecture used in previous Gemma versions, with several key architectural improvements. The most significant change is the integration of vision capabilities through a SigLIP vision encoder that converts images into a sequence of 256 "soft tokens" that the language model can process. To handle images of different resolutions and aspect ratios, the model implements a "Pan and Scan" method that segments images into non-overlapping crops, processes them separately, and then combines the results.

To enable longer context handling (up to 128K tokens) without excessive memory usage, the architecture introduces a 5:1 ratio of local to global attention layers. Local attention layers use a sliding window of 1024 tokens, while global layers attend to the entire context. This design significantly reduces the memory footprint of the KV cache during inference, which typically grows linearly with context length in traditional transformer models.

The training process follows a two-stage approach. First, the models are pre-trained on a mixture of text and image data (14 trillion tokens for the 27B model), with an increased proportion of multilingual content compared to previous versions. Knowledge distillation is used during pre-training, where the model learns from a larger teacher model. The second stage involves instruction tuning through a combination of supervised fine-tuning and reinforcement learning with various reward functions to improve helpfulness, math, coding, reasoning, and multilingual abilities while minimizing harmful outputs.

For quantization, the team employs Quantization Aware Training to create more efficient versions of the models in different formats (int4, block int4, and switched fp8), making them more accessible for deployment on resource-constrained devices.

Results

Gemma 3 demonstrates significant improvements over its predecessors across various benchmarks. The 27B instruction-tuned model achieves an Elo score of 1338 in the LMSYS Chatbot Arena, placing it among the top 10 models and outperforming many larger models like DeepSeek-V3 (1318), LLaMA 3 405B (1257), and Qwen2.5-70B (1257).

On standard benchmarks, Gemma 3 shows substantial gains in mathematics, reasoning, and multilingual capabilities. The 27B model achieves 67.5% on MMLU-Pro, 89.0% on MATH, and 60.3% on HiddenMath, making it competitive with Gemini-1.5-Pro across many metrics. The 4B instruction-tuned model performs comparably to the previous 27B model, demonstrating the effectiveness of the new architecture and training approach.

The paper also reports significantly lower memorization rates compared to previous models, with no detected personal information in outputs classified as memorization across all Gemma 3 models, indicating improved privacy protection.

The vision capabilities show strong performance, particularly with the Pan & Scan method, which improves results on DocVQA by 4.8 percentage points and on InfoVQA by 17.0 percentage points for the 27B model.

Conclusion

Gemma 3 represents a significant advancement in open language models by successfully integrating vision understanding, extending context length, and improving multilingual capabilities while maintaining efficiency for consumer hardware. For more information please consult the full paper.

Congrats to the authors for their work!

Gemma Team, Google DeepMind. "Gemma 3 Technical Report." 12 Mar. 2025.

View draft history