Generalization Capabilities, Limitations, and Emerging Approaches to Zero-Shot and Few-Shot Learning in Foundation Models

Abstract

By enabling broad task transfer through large-scale pre-training, prompting, in-context learning, and parameter-efficient adaptation, foundation models have revolutionized machine learning. In few-shot learning, a few demonstrations or examples are provided to help the adaptation, while in zero-shot learning the model needs to solve a task without any labelled example. This paper summarizes the state of the art in zero-shot and few-shot learning in language, vision, and multimodal foundation models, and discusses their generalization potential, their limitations, and novel methods. It reveals that foundation models can attain a high zero-shot and few-shot performance when the target tasks are semantically close to their pre-training distribution and when prompts, demonstrations, or instructions specify the target task well enough. It is important to note, however, that performance is still sensitive to prompt design, type and ordering of the demonstrations, label bias, domain shift, context length, and task recognition vs. real task learning. In-context learning is simple to compute as no parameters need to be updated, but it can be unstable and could be taking advantage of a surface-level correlation, not the intended task rule. There are complementary methods that can enhance data efficiency and robustness, such as parameter-efficient fine-tuning, retrieval-augmented prompting, instruction tuning, and adaptive prompting. It suggests standardized protocols for future evaluations of zero-shot and few-shot learning, which clearly distinguish between task recognition and task learning, include information on variability across prompt permutations, measure calibration and robustness, and explicitly test low-resource and cross-domain scenarios.