V
主页
[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
发布人
Abstract: Fully Sharded Data Parallel (FSDP) is the newest tool we’re introducing. It shards an AI model’s parameters across data parallel workers and can optionally offload part of the training computation to the CPUs. As its name suggests, FSDP is a type of data-parallel training algorithm. Although the parameters are sharded to different GPUs, the computation for each microbatch of data is still local to each GPU worker. This conceptual simplicity makes FSDP easier to understand and more applicable to a wide range of usage scenarios (compared with intra-layer parallelism and pipeline parallelism). Compared with optimizer state+gradient sharding data parallel methods, FSDP shards parameters more uniformly and is capable of better performance via communication and computation overlapping during training. With FSDP, it is now possible to more efficiently train models that are orders of magnitude larger using fewer GPUs. FSDP has been implemented in the FairScale library and allows engineers and developers to scale and optimize the training of their models with simple APIs. At Facebook, FSDP has already been integrated and tested for training some of our NLP and Vision models. My notes: https://blog.olewave.com/olewaves-tech-review-fully-sharded-data-parallel-faster-ai-training-with-fewer-gpus/
打开封面
下载高清视频
观看高清视频
视频下载器
[Long Review] Axial Attention in Multidimensional Transformers
十分钟看懂微软大力金刚掌WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack
[Long Review] Cascaded Diffusion Models for High Fidelity Image Generation
[Long Review] GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
[Long Review] Conformer: Convolution-augmented Transformer for Speech Recogniti
十分钟看懂谷歌铁布衫BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised ...
[Long Review] CLAS: Deep context: end-to-end contextual speech recognition
[Long Review] Deduplicating Training Data Makes Language Models Better
[Long Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using
十分钟看懂谷歌W2v-BERT: Combining Contrastive Learning and Masked Language Modeling
十分钟看懂谷歌金钟罩Transformer以及语音的LAS模型
语音文本技术论文阅读 Branchformer: Parallel MLP-Attention Architectures and E-Branchformer
[Olewave's Long Review] Efficient Training of Neural Transducer for Speech Recog
十分钟看懂谷歌易筋经BERT
解锁天顶星科技ChatGPT
语音文本技术论文阅读 RefineGAN - Universally Generating Waveform Better than Ground ...
[Long Review] Towards Zero-Label Language Learning
语音文本技术论文阅读 RNN-T: Sequence Transduction with Recurrent Neural Networks
【2024最新完整版】不愧是李宏毅教授!一口气学完机器学习、深度学习、强化学习、NLP、生成式AI等课程!一套全解决!
语音文本技术论文阅读 UniSpeech-SAT - Universal Speech Representation Learning with Speaker
语音文本技术论文阅读 SNRi Target Training for Joint Speech Enhancement and Recognition
语音文本技术论文阅读 XLS-R: Self-supervised Cross-lingual Speech Representation Learning a
语音NLP论文阅读 Token-level Sequence Labeling for SLU using Compositional E2E Models
语音文本技术论文阅读 Joint Unsupervised and Supervised Training for Multilingual ASR
详解微软零样本语音合成VALL-E
语音文本技术论文阅读 Scaling Laws for Neural Language Models
[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs
[Short Review] Transfer Learning from Speaker Verification to Multispeaker TTS
详解AudioLM: a Language Modeling Approach to Audio Generation
详解OpenAI GPT-3: Language Models are Few-Shot Learners(2/3)
福奇博士小声嘟囔议员蠢货,结果忘记关麦克风 -- analysis from a research perspective
CV论文阅读OPENAI CLIP(1/3):Learning Transferable Visual Models From Natural Language
[Short Review] Axial Attention in Multidimensional Transformers
[Short Review] Xception: Deep Learning with Depthwise Separable Convolution
[Short Review] Cascaded Diffusion Models for High Fidelity Image Generation
[Short Review]Conformer Convolution-augmented Transformer for Speech Recognition
[Long Review] 用GShard训练超级大AI模型:GShard: Scaling Giant Models
[Short Review] Deduplicating Training Data Makes Language Models Better
语音文本技术论文阅读 OpenAI最新的Whisper ASR也会像GPT-3一样火起来吗?
CV论文阅读OpenAI CLIP(2/3):Learning Transferable Visual Models From Natural Language