V
主页
In-Context Learning Creates Task Vectors
发布人
论文简述:在《In-Context学习创建任务向量》这篇论文中,作者提出了一种名为in-context学习(ICL)的新颖学习方法。这种方法在大型语言模型(LLM)中的应用已经取得了显著的成功。然而,其背后的机制仍然不太清楚。本文通过展示ICL所学的函数往往具有非常简单的结构,为解决这一问题迈出了重要一步:这些函数对应于仅使用查询$x$和一个从训练集计算出的任务向量作为输入的变压器LLM。因此,ICL可以被看作是将$S$压缩成一个单一的任务向量$\boldsymbol\theta(S)$,然后使用这个任务向量来调制变压器以产生输出。通过在一系列模型和任务上的全面实验,作者支持了上述观点。这篇论文的主要发现是,in-context学习可以通过将训练集压缩成一个任务向量来实现。这使得ICL能够利用这个任务向量来调整变压器,从而在各种不同的情况下实现高效的学习。这一发现为理解ICL的机制提供了新的视角,并为未来的研究奠定了基础。 论文链接: https://arxiv.org/pdf/2310.15916
打开封面
下载高清视频
观看高清视频
视频下载器
Visual In-Context Prompting
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
Retrieval meets Long Context Large Language Models
GridFormer 表结构识别方法
PALP: Prompt Aligned Personalization of Text-to-Image Models
Localizing and Editing Knowledge in Text-to-Image Generative Models
Context-Aware Meta-Learning
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Interactive Task Planning with Language Models
图像到视频迁移学习中空间和时间学习的解耦
Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with
Aligning Text-to-Image Diffusion Models with Reward Backpropagation
ConvNets Match Vision Transformers at Scale
DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagat
Learning to Learn Faster from Human Feedback with Language Model Predictive Cont
RLVF: Learning from Verbal Feedback without Overgeneralization
CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
Memory Consolidation Enables Long-Context Video Understanding
Farzi Data: Autoregressive Data Distillation
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirect
Compressing Context to Enhance Inference Efficiency of Large Language Models
Toward Joint Language Modeling for Speech Units and Text
An Image is Worth Multiple Words: Learning Object Level Concepts using Multi-Con
Contrastive Prefence Learning: Learning from Human Feedback without RL
Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression
基于语言模型的知识探测和推理方法
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
生成高质量的长视频:SEINE视频扩散模型
图像生成美学优化方法研究
In-Context Principle Learning from Mistakes
Amortizing intractable inference in large language models
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State
Offline Actor-Critic Reinforcement Learning Scales to Large Models
SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents
基于GPT-4V的视频理解系统MM-VID
弱到强泛化:用弱监督激发强大模型的潜力
Making Large Language Models Perform Better in Knowledge Graph Completion