Research Paper Deep Dive - The Sparsely-Gated Mixture-of-Experts (MoE)
Prodramp
In this video we are taking a deep dive to learn the more about the Mixture of Experts (or MoE), how it works and internal architecture, text and images data processing..
More Info:
We all know that large model sizes is necessary for strong generalization and robustness, so training large models while limiting resource requirements is becoming increasingly important.
There is a hidden problem underneath these superstar, eye-popping results and the problem is significant use of computation resources or the requirements, which includes supermassive hardware and that includes logistics, cost, power requirement, and top of the above feasibility to even move it outside labs..
GitHub Resources: https://github.com/prodramp/DeepWorks/tree/main/MoE
Research Paper and Code: https://arxiv.org/abs/1701.06538 https://github.com/davidmrau/mixture-of-experts
ā¬ā¬ā¬ā¬ā¬ā¬ ā° TUTORIAL TIME STAMPS ā° ā¬ā¬ā¬ā¬ā¬ā¬
- (00:00) Paper Introduction
- (00:22) Understanding the Problem
- (01:22) Significant computation requirement
- (02:33) Solution - Conditional Computation
- (03:35) Dense and Sparse Models
- (04:17) Pathway Models
- (06:20) MoE Introduction
- (08:38) MoE Internals
- (11:05) MoE Components
- (14:44) Data Processing in MoE
- (15:09) Text Data Processing in MoE
- (17:07) Image Data Processing in MoE
- (19:47) Text and Image Data Processing in MoE
- (20:46) Research Paper and Code
- (21:30) Resources and GitHub Reference
Connect
- Prodramp LLC (@prodramp)
- Website - https://prodramp.com
- LinkedIn - https://www.linkedin.com/company/prodramp
- GitHub- https://github.com/prodramp/
- AngelList - https://angel.co/company/prodramp
- Facebook - https://www.facebook.com/Prodramp
Content Creator: Avkash Chauhan (@avkashchauhan)
Tags: #moe #ai #cnn #ml #lime #aicloud #h2oai #driverlessai #machinelearning #cloud #mlops #model #collaboration #deeplearning #modelserving #modeldeployment #pytorch #datarobot #datahub #streamlit #modeltesting #codeartifact #dataartifact #modelartifact #onnx #aws #kaggle #mapbox #lightgbm #xgboost #classification #dataengineering #pandas #keras #tensorflow #tensorboard #cnn #prodramp #avkashchauhan #LIME #mli #xai ... https://www.youtube.com/watch?v=ADooiJ9V7jo
181089914 Bytes