Enabling on-device learning at scale
1 like383 views
The need for intelligent, personalized experiences powered by AI is ever-growing. Our devices are producing more and more data that could help improve our AI experiences. How do we learn and efficiently process all this data from edge devices while maintaining privacy? On-device learning rather than cloud training can address these challenges. In this presentation, we’ll discuss: - Why on-device learning is crucial for providing intelligent, personalized experiences without sacrificing privacy - Our latest research in on-device learning, including few-shot learning, continuous learning, and federated learning - How we are solving system and feasibility challenges to move from research to commercialization
More Related Content
What's hot (20)
Similar to Enabling on-device learning at scale (20)
More from Qualcomm Research (20)
Recently uploaded (20)
Enabling on-device learning at scale
- 1. October 28, 2021 @QCOMResearch Qualcomm Technologies, Inc. Enabling on-device learning at scale
- 2. 2 Today’s agenda • What is on-device learning and why is it crucial for scaling intelligence? • Our latest on-device learning research and results • Conclusions and future directions • Questions?
- 3. 3 3 Video monitoring Extended reality Smart cities Smart factories Autonomous vehicles Video conferencing Smart homes Smartphone The need for intelligent, personalized experiences powered by AI is ever-growing How do we maintain privacy and deal with all the data from edge devices?
- 4. 4 On-device intelligence is paramount Process data closest to the source, complement the cloud Privacy Reliability Low latency Efficient use of network bandwidth
- 5. 5 Public network Private networks Transformation of the Connected Intelligent Edge has begun at scale Edge cloud Cloud On-device Past Cloud-centric AI AI training and inference in the central cloud Future Fully-distributed AI With lifelong on- device learning Today Partially-distributed AI Power-efficient on-device AI inference Processing data closer to devices at the edge derives new system values (e.g., lower latency, enhanced privacy)
- 6. 6 Object classification Voice activation Text recognition Gesture/ hand tracking Computational photography Contextual awareness Face detection On-device security Voice recognition Fingerprint Local network analytics Low-latency interactive content Boundless XR On-demand computing Industrial automation and control Enterprise data + Edge cloud AI On-device AI Connected Intelligent Edge brings new and enhanced services
- 7. 7 Data and labels Training Offline training A model is trained in the cloud with data reflecting the target application On-device learning Modifying model after deployment based on the test environment Deploy What is on-device learning? Test data Adapt model Inference
- 8. 8 Data and labels Training With offline training, the test data can differ from training data (domain shift, distribution shift, anomalies) and may even change continuously Test data On-device learning can help to improve and maintain accuracy when original pre-trained model cannot generalize well Adapt model Inference Deploy On-device learning offers several benefits • Continuous learning • Personalization • Data privacy • Scale
- 9. 9 Overcoming challenges to achieve on-device ML benefits Important considerations for on-device learning to achieve benefits for different use cases Challenges Benefits • Better examples than training dataset • Ability to run with smaller models that adapt to the target data • Preservation of privacy during model development • Local data can be limited, e.g., noisy labels and class imbalance • Overfitting or catastrophic forgetting • Limited compute, storage, and/or power • Adversarial attacks to training • Federated learning communication overhead Test data Adapt model 9 On-device learning
- 10. 10 Our AI research areas address the key deployment challenges of on-device learning 10 How to adapt the model to a few labeled samples Few-shot learning How to use unlabeled data to do unsupervised learning Continuous learning with unlabeled data How to implement federate learning at scale and address deployment challenges Federated learning for global adaptation Low-complexity on-device learning How to implement on-device learning to improve efficiency
- 11. 11 11 Improve the target user’s model using the initial collected data, such as enrollment Learning from limited labeled data is crucial Few-shot learning Offline learning 11 Deploy global model On-device learning Target user labeled data (e.g., from enrollment) Limited samples with user-assisted labeling May be from cleaner environment Cloud data and training Enroll
- 12. 12 12 Qualcomm AI Research internal results . Few-shot learning for increased personalization Improving keyword spotting (KWS) performance of outlier users through on-device learning Detection rate for outlier users is over 30% worse, on average Identify when a keyword is spoken using always-on ML Keyword spotting • In practice, it is hard to collect all types of accented utterance • The KWS model may not be sensitive to users’ accents and have poor performance for outliers Keyword spotting challenge • Locally adapt the model to user enrollments • Personalize the model at enrollment time Keyword spotting solution
- 13. 13 Local adaptation • Collect enrollment data from target user • Adapt the global model on local data Personalized Model Local training Global model Keyword Non-keyword Dataset from local enrollment Device How to locally adapt keyword spotting for personalization Server Train a global KWS model • Global train/validation dataset Global training dataset Global model Validation Train Global validation dataset
- 14. 14 Qualcomm AI Research internal results. For the few-shot results, performance is the average of all the locally adapted models. Personalization improvements across the board but particularly for outliers Few-shot learning for KWS improves performance Average detection rate improvement Few-shot vs baseline model 0% 5% 10% 15% 20% 25% 30% Inlier Outlier Percentage improvement (%) 30% Improvement in detection rate* Up to
- 15. 15 15 Improve the target user’s model based on data from continuous operation, often unlabeled data Leveraging user data throughout deployment Continuous learning Offline learning Deploy global model On-device learning Target user data through continuous operation Plentiful samples of unlabeled user data Can be from various environment: clean, noisy Cloud data and training Enroll Day1, 2, …
- 16. 16 Solving the challenges for continuous learning Employ pseudo labeling and regularization to reduce impact from forgetting Challenge Training data are collected on the device without labels Unlabeled collected data Solution Assign pseudo labels to training data through the verification process Overfitting to small data Challenge Number of collected data is small Solution Exploit regularization loss that maintains some metrics from pre-trained model
- 17. 17 17 Global adaptation Local adaptation Federated learning brings on-device learning to new level Adaptation on the device, once or continuously, locally and/or globally for continuous model enhancement Federated learning for global adaptation 17 Offline learning Data On-device learning Locally adapt once to a few samples (e.g., few shot learning) or continuously (e.g., unsupervised learning) Adapt model based on local data Offline training prior to deployment Federated learning Aggregate model updates across multiple users to globally improve model from more diverse data
- 18. 18 18 Each device collects personal data and performs on-device training Federated learning over 5G is the way to scale intelligence Cloud A state-of-art global AI model is distributed to devices D1 D2 DN 2 Scale Personalization Privacy Processing is spread over many devices Model customized based on your personal data Raw data stays on the device 1
- 19. 19 19 Federated learning over 5G is the way to scale intelligence 1. Adds perturbation to the parameters of the AI model Cloud The cloud updates the AI model and then sends it to the devices again 4 D1 D2 DN 3 Each device: 2. Compresses the parameters 3. Encrypts them 4. Sends the update to the cloud Scale Privacy Network bandwidth is conserved Only noisy and encrypted weights sent to the cloud
- 20. 20 20 “Federated Learning of User verification Models Without Sharing Embeddings” (ICML 2021) Federated learning can be a powerful tool for user verification User verification The authentication problem needs big data to get a powerful verification model E.g., typical speaker verification system needs data from more than 600k different speakers Challenge How can we learn this model while keeping all data private? We do not want to compromise the sensitive biometric data of training participants Personal data available for authentication Deep learning approach for authentication Face Fingerprint Voice Medical Data Client 𝐾 resource 𝑥 𝑔(𝑥) 𝑧 Data User score 𝑔 𝑊 User A User B User C Embedding vector W Output 𝑔 𝑥
- 21. 21 “Federated Learning of User verification Models Without Sharing Embeddings” (ICML 2021) User embeddings need to be shared for training Traditional design of neural networks for user verification do not preserve privacy 𝑥 𝑔(𝑥) 𝑧 Data User score 𝑔 𝑊 User A User B User C Embedding vector W Output 𝑔 𝑥 For user verification, neural network 𝒈 𝒙 should be trained to: Minimize the loss to the target user A smaller loss means a higher user score Maximize the loss to the other users In traditional (one-hot) approaches, users share embeddings to calculate this loss (not private)
- 22. 22 “Federated Learning of User verification Models Without Sharing Embeddings” (ICML 2021) Generate user embeddings using error-correcting codes (ECC) We enable federated learning for user verification without users sharing their embeddings 𝑥 𝑔(𝑥) 𝑧 Data User score 𝑔 𝑊 User A User B User C Our method (FedUV) accomplishes this by using embeddings that are codewords of error correcting codes (ECC) and optimizes network 𝒈 𝒙 using only positive loss function Each user minimizes their own loss ECC ensures user embeddings are maximally spaced to reduce score to other users Embedding vector W Output 𝑔 𝑥
- 23. 23 “Federated Learning of User verification Models Without Sharing Embeddings” (ICML 2021) FedUV achieves state-of-the-art verification performance without users sharing their embeddings FedUV (127) FedUV (255) FedUV (511) FedAws Softmax 1.0 0.8 0.6 0.4 0.2 0.00 0.02 0.04 0.06 0.08 0.10 False positive rate True positive rate FedUV is better than existing method, which does not share user embeddings (FedAWS) FedUV is comparable to the best method, which shares user embeddings (softmax)
- 24. 24 24 RPC: remote procedure call. Snapdragon is a product of Qualcomm Technologies, Inc. and/or its subsidiaries. FL framework for research and application development on mobile FL demo of speaker verification • Enrollment from 1000 clients • Leverage PyTorch model & training pipeline from research framework Samsung Galaxy S21 device powered by Snapdragon® 888 Platform Python control ML experts Coordinator server Mobile devices Android app Pipe Worker host gRPC Torch host LibTorch C++ Android app Pipe Worker host gRPC Torch host LibTorch C++ Android app Pipe Worker host gRPC Torch host LibTorch C++ Android app Pipe Worker host gRPC Torch host LibTorch C++ Controller gRPC Worker manager PyTorch / TensorFlow code FL trainer gRPC TCP/IP network TCP/IP network Worker
- 26. 26 Learning with backprop is computationally demanding Updating the model weights using backprop can be expensive, especially on power-constrained devices Data and labels Training Backward pass with gradient computation = “backpropagation” Layer 1 In Out Grad Layer 2 In Out Grad Layer L In Target Out Grad 𝐿𝑖 𝑔𝑖 (𝐿) 𝑔𝑖 (2) 𝑔𝑖 (3) Forward pass = inference Layer 1 In Out Layer 2 In Out Layer L In Out 𝑥𝑖 𝑦𝑖 𝑎𝑖 (1) 𝑎𝑖 (2) 𝑎𝑖 (𝐿) AI model
- 27. 27 Overcoming challenges to efficiently adapt a neural net on a device 27 Backprop training requirements Large memory DRAM Training runtime High precision Support for quantized inference Adapt AI model on the device 1. Reduce complexity of backprop with quantized training 2. Efficient models for backpropagation 3. Adapt model using inference
- 28. 28 “In-Hindsight Quantization Range Estimation for Quantized Training” (CVPR ECV Workshop 2021) Challenge Maintain accuracy and reduce compute and memory using quantized gradients and activations Solution Quantization with In-Hindsight Range Estimation NN quantization is very effective for NN inference: low energy with high accuracy Can we use quantization in backpropagation to make NN training more efficient? Reduce backprop complexity with quantized training Backward pass with gradient computation = “backpropagation” Layer 1 In Out Grad Layer 2 In Out Grad Layer L In Target Out Grad 𝐿𝑖 𝑔𝑖 (𝐿) 𝑔𝑖 (2) 𝑔𝑖 (3)
- 29. 29 “In-Hindsight Quantization Range Estimation for Quantized Training” (CVPR ECV Workshop 2021) Estimating range with dynamic quantization • Uses statistics from the current feature map to quantize it • Requires writing the 32-bit feature map to memory before quantization • Is expensive to implement due to high memory transfers Existing quantized training techniques are too complex MAC Array Accumulator Quantization ෩ GY ෩ 𝑊 Memory Statistics ෩ GX GX 0 GX M GX 1 Dynamic quantization Static Memory Memory 𝑞min , 𝑞max Gradient method Activation method ResNet18 Memory transfer FP32 FP32 69.75 High Current min-max Current min-max 69.21 +/- 0.06 High Running min-max Running min-max 69.35 +/- 0.16 High
- 30. 30 “In-Hindsight Quantization Range Estimation for Quantized Training” (CVPR ECV Workshop 2021) Use pre-computed quantization parameters to quantize current tensor Extract statistics from current tensor for quantization parameters on next iteration Much lower complexity and data movement In-Hindsight Range Estimation reduces quantize training complexity while maintaining accuracy Gradient method Activation method ResNet18 Memory transfer FP32 FP32 69.75 High Current min-max Current min-max 69.21 +/- 0.06 High Running min-max Running min-max 69.35 +/- 0.16 High In-hindsight min-max In-hindsight min-max 69.37 +/- 0.11 Low Memory movement cost comparison between static and dynamic quantization MAC Array Accumulator Quantization ෩ GY ෩ 𝑊 Memory 𝑞min 𝑡+1 , 𝑞max 𝑡+1 ෩ GX GX 0 GX M GX 1 In-hindsight Memory Memory 𝑞min 𝑡 , 𝑞max 𝑡 79% Reduction in memory transfer*
- 31. 31 Typically, the backpropagation calculation requires large memory to store activations 𝑊 𝜎 𝑥 𝑧 𝑦 𝜕ℓ 𝜕𝑧 = 𝜕ℓ 𝜕𝑦 ⋅ 𝜎′ 𝑧 For backpropagation, activations need to be stored at every layer to calculate the gradient of the loss function Typical layer of a neural network Linear weights Non-linear activation function ℓ is the loss function to be minimized Input to layer Output to next layer
- 32. 32 Qualcomm AI Research internal results. Using invertible layers reduces memory requirements for backpropagation Typical layer of a neural network 𝑊 𝜎 𝑥 𝑧 𝑦 𝜕ℓ 𝜕𝑧 = 𝜕ℓ 𝜕𝑦 ⋅ 𝜎′ 𝑧 For backpropagation, activations need to be stored at every layer to calculate the gradient of the loss function Activations of each layer can be reconstructed exactly from next layer 𝑧 can be computed for each layer in backpropagation, so no need to store activations Invertible layer 𝑧 = 𝑦2𝑊 𝑊 𝑥1 𝑥2 𝑦1 𝑦2 𝜎 𝑧 𝑥1 = 𝑦2 𝑥2 = 𝑦1 − 𝐹 𝑦2 #params (M) / #MACs (B) Top 1 / Top 5 Activation mem. per image MobileNet-V2 3.4 / 0.3 72.0 / 91.0 43 MB Invertible network 3.24 / 0.3 72.5 / 90.7 3.7 MB 11x Reduction in activation memory
- 33. 33 Broad range of research directions for on-device learning Test data Adapt model Privacy, security, and robustness Privacy guarantees, adversarial attack, anomaly detection Tackling statistical heterogeneity in data Smartly combining model updates from a broad distribution Personalization and labeling Meta learning, active learning, learning with noisy labels Optimizing communication in FL Compressing information sent on the uplink and downlink On-device training efficiency Light-weight models, low complexity training, quantization Advanced topologies for FL Peer-to-peer, multi-cloud, and hierarchical privacy
- 34. 34 34 On-device learning is crucial for providing intelligent, personalized experiences without sacrificing privacy We are conducting leading research and development in on-device learning We are solving system and feasibility challenges to move from research to commercialization 34
- 36. Follow us on: For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog Thank you Nothing in these materials is an offer to sell any of the components or devices referenced herein. ©2018-2021 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm and Snapdragon are trademarks or registered trademark of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes our licensing business, QTL, and the vast majority of our patent portfolio. Qualcomm Technologies, Inc., a subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of our engineering, research and development functions, and substantially all of our products and services businesses, including our QCT semiconductor business.