Exploring Innovative Training Techniques for Small Language Models: The Case of Phi-3

In the ever-evolving landscape of artificial intelligence (AI), the development of efficient and high-performing Small Language Models (SLMs) is being touted as a game changer. The recent introduction of the Phi-3 models by Microsoft represents a significant leap forward in this domain, showcasing the ability of SLMs to deliver robust performance with considerably less data and a compact footprint. This article explores the innovative training techniques employed in the Phi-3 models, their performance benchmarks, and potential use cases.

 Before diving into Phi-3 let's explore a few terms for better understanding:

• Synthetic Data - Data created artificially through computer simulation or machine learning techniques to supplement or complement real-world data. It replicates the statistical characteristics and patterns of real-world data. This data can be used as an alternative to real-world data when real-world data is in short supply. Data produced this way has a positive impact on data privacy.
• Instruction Tuning - It is a technique for fine-tuning large/small language models on a labeled dataset of instructional prompts and corresponding outputs. It is like giving special training to your SLMs to follow instructions better by showing it examples of tasks and the expected outputs. Pre-trained LLMs/SLMs are not optimized for conversations or instruction following. In a literal sense, they do not answer a prompt: they only append text to it. Instruction tuning helps make that appended text more useful from a conversational perspective.
• MMLU - Massive Multitask Language Understanding involves training a computer model to carry out multiple tasks simultaneously. These tasks could include translating languages, summarizing text, or analyzing the sentiment of a statement. The term “massive” indicates that the model is designed to handle a large number of tasks and process a vast amount of data. It’s calculated based on the proportion of correct answers the model provides for a set of multiple-choice questions it has been presented.
• MT-bench - multi-turn benchmark measures the ability of a model to engage in meaningful and engaging conversations. It assesses how well a model can understand and respond. Additional information here.

Training Methodology

The Phi-3 models, including the Phi-3-mini, Phi-3-small, and Phi-3-medium, are part of a new generation of SLMs that have been trained using a mix of heavily filtered web data and synthetic data. This approach deviates from traditional methods that rely on vast amounts of unfiltered data, leading to models that are not only smaller but also more efficient. Owing to the smaller footprint and higher precision in their training dataset they are more environmentally friendly. The training dataset for Phi-3 is a scaled-up version of the one used for its predecessor, Phi-2, ensuring a high-quality data foundation.

One of the key innovations in the Phi-3 models is their instruction-tuning, which means they are trained to follow various types of instructions reflecting natural human communication. This results in models that are ready to use out-of-the-box, with little need for additional fine-tuning. Additionally, the Phi-3 models have undergone extensive safety post-training, including reinforcement learning from human feedback (RLHF), automated testing, and evaluations across multiple harm categories to ensure that they behave in-line with the Responsible AI guidelines.

Performance Benchmarks

The Phi-3 models have set new standards in performance benchmarks. For instance, the Phi-3-mini, with 3.8 billion parameters, achieves a score of 69% (higher is better) on the MMLU benchmark and 8.38 on the MT-bench. The larger variants, Phi-3-small and Phi-3-medium, with 7 billion and 14 billion parameters respectively, score even higher, with 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench.

Brief comparison below:

These results are particularly impressive considering the models' smaller sizes compared to their counterparts. The fact that an SLM can compete with ChatGPT 3.5 demonstrates the efficacy of the training methodology.

Use Case - Potential Applications in Healthcare

There are numerous practical applications of the Phi-3 models across a variety of scenarios as they are particularly well-suited for resource-constrained environments, including on-device and offline inference scenarios.

In this use case let's explore the possibilities on Mobile Health Diagnostics and Assistance: In remote areas where healthcare resources are scarce and connectivity may be limited, Phi-3 can be deployed on mobile devices to assist healthcare workers and patients. With its small size and efficient processing capabilities, Phi-3 can be integrated into mobile health applications to provide:

• Symptom Analysis and Preliminary Diagnosis: Healthcare workers can input symptoms into the app, and Phi-3 can analyze the information to suggest possible conditions. This can help in prioritizing patient care and driving the most appropriate actions per the patient's need.
• Medical Information Retrieval: Phi-3 can quickly search through medical databases to provide healthcare workers with information on diseases, treatments, and medications, even with limited internet connectivity.
• Language Translation: In areas with diverse languages, Phi-3 can assist in translating medical instructions and patient histories, facilitating better communication between patients and healthcare providers.
• Training and Education: Phi-3 can be used to train healthcare workers in remote areas by providing access to the latest medical research and educational content in an interactive format.

Conclusion

The Phi-3 models represent a significant advancement in the field of generative AI, particularly in the context of SLMs. SLMs innovative training techniques, impressive performance benchmarks, and their ability to perform well in constrained environments should make them the preferred options to solve for some of humanity's most teething challenges!