V
主页
Asynchronous Local-SGD Training for Language Modeling
发布人
【加群】 一起来刷arxiv,请加vx: pwbot02(请备注:b站arxiv) 【论文标题】 Asynchronous Local-SGD Training for Language Modeling 【论文简述】 本论文通过对异步本地随机梯度下降算法在训练语言模型中的应用进行实证研究,探讨了工作设备异构性、模型规模、工作人员数量和优化器对学习性能的影响。研究发现,尽管通过更频繁地更新(全局)模型参数,异步本地随机梯度下降算法比同步算法需要更多的迭代次数才能收敛。这是因为工作人员梯度陈旧时全局参数的动量加速成为了一个关键挑战。为解决此问题,论文提出了一种新的方法,利用延迟的Nesterov动量更新,并根据计算速度调整工作人员的本地训练步骤。经过在C4数据集上对含有150M参数的模型的评估,该方法在每次更新步骤的困惑度方面与同步本地随机梯度下降算法相匹配,并在墙时钟时间上显著优于同步算法。 【论文链接】 https://arxiv.org/abs/2401.09135
打开封面
下载高清视频
观看高清视频
视频下载器
Amortizing intractable inference in large language models
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
BitNet: Scaling 1-bit Transformers for Large Language Models
Memory Augmented Language Models through Mixture of Word Experts
Interactive Task Planning with Language Models
Llemma: An Open Language Model For Mathematics
Toward Joint Language Modeling for Speech Units and Text
FlashDecoding++: Faster Large Language Model Inference on GPUs
Language Models can be Logical Solvers
Can Large Language Models be Good Path Planners? A Benchmark and Investigation o
Are Large Language Models Post Hoc Explainers?
Large Language Model Cascades with Mixture of Thoughts Representations for Cost-
Video Language Planning
Exponentially Faster Language Modelling
Compressing Context to Enhance Inference Efficiency of Large Language Models
RLVF: Learning from Verbal Feedback without Overgeneralization
AutoMix: Automatically Mixing Language Models
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Langu
Retrieval meets Long Context Large Language Models
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
Large Language Models Cannot Self-Correct Reasoning Yet
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Contrastive Chain-of-Thought Prompting
The Consensus Game: Language Model Generation via Equilibrium Search
Moral Foundations of Large Language Models
Making Large Language Models Perform Better in Knowledge Graph Completion
LayoutPrompter: Awaken the Design Ability of Large Language Models
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
TiC-CLIP: Continual Training of CLIP Models
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations
Tuna: Instruction Tuning using Feedback from Large Language Models
Controlled Decoding from Language Models
FLAP: Fast Language-Audio Pre-training
AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort
System 2 Attention (is something you might need too)
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models