Few-Shot Prompting

Although large language models demonstrate remarkable abilities in the zero-shot setting, they still show weaknesses on more complex tasks in zero-shot mode. Few-shot prompting is a technique that enables in-context learning by providing demonstrations in the prompt to lead the model to improved performance. The demonstrations serve as conditioning for subsequent examples where the model should generate a response.

According to Touvron et al. (2023)¹, few-shot properties first emerged when models were scaled to a sufficient size (Kaplan et al., 2020²). Few-shot prompting can be illustrated with the following example presented in Brown et al. (2020)³. The example shows how a new word is used correctly in a sentence.

Ein "Whatpu" ist ein kleines, pelziges Tier, das in Tansania heimisch ist. Ein Beispiel für einen Satz, der das Wort Whatpu verwendet, ist:
Wir reisten in Afrika, und wir sahen diese sehr süßen Whatpus.
"farduddeln" bedeutet, sehr schnell auf und ab zu springen. Ein Beispiel für einen Satz, der das Wort farduddeln verwendet, ist:

Als wir das Spiel gewannen, begannen wir alle zu farduddeln, um zu feiern.

It can be observed that the model learned to perform the task even though only one example (i.e., 1-shot) was provided. For more complex tasks, you can experimentally investigate how increasing the number of demonstrations (e.g., 3-shot, 5-shot, 10-shot, etc.) affects performance.

According to the findings of Min et al. (2022)⁴, the following aspects are relevant for demonstrations/examples in few-shot prompting:

• “The label space and the distribution of the input text specified by the demonstrations are both important (regardless of whether the labels for individual inputs are correct).”
• The format used also plays a key role for performance; even with random labels, this is significantly better than the complete absence of labels.
• Further results show that selecting random labels from a real label distribution (instead of a uniform distribution) is also beneficial.

This can be illustrated with a few examples. First, an example with random labels is presented, where the labels “Negativ” and “Positiv” are randomly assigned to inputs:

Das ist toll! // Negativ
Das ist schlecht! // Positiv
Wow, dieser Film war klasse! // Positiv
Was für eine fürchterliche Vorstellung! //

Negativ

The correct answer is still generated even though the labels were assigned randomly. The format is preserved, which also contributes to performance. Newer GPT models also show increasing robustness to random formats. An example:

Positiv Das ist toll!
Das ist schlecht! Negativ
Wow, dieser Film war klasse!
Positiv
Was für eine fürchterliche Vorstellung! --

Negativ

Although there is no consistency in the format in the example above, the model still predicts the correct label. A more comprehensive analysis is required to confirm whether this holds for different and more complex tasks and different prompt variations.

Limitations of Few-Shot Prompting

Standard few-shot prompting works satisfactorily for many tasks, but it is not a perfect technique, especially for more complex reasoning tasks. The following demonstration illustrates this limitation. The next example describes the task:

Die ungeraden Zahlen in dieser Gruppe addieren sich zu einer geraden Zahl: 15, 32, 5, 13, 82, 7, 1.

A:

When this is tested again, the model outputs the following answer:

Ja, die ungeraden Zahlen in dieser Gruppe addieren sich zu 107, was eine gerade Zahl ist.

This answer is not correct, which not only shows the limits of the systems but also highlights the need for more advanced prompt engineering.

By adding a few examples, we test whether few-shot prompting improves the results:

Die ungeraden Zahlen in dieser Gruppe addieren sich zu einer geraden Zahl: 4, 8, 9, 15, 12, 2, 1.
A: Die Antwort ist falsch.

Die ungeraden Zahlen in dieser Gruppe addieren sich zu einer geraden Zahl: 17, 10, 19, 4, 8, 12, 24.
A: Die Antwort ist wahr.

Die ungeraden Zahlen in dieser Gruppe addieren sich zu einer geraden Zahl: 16, 11, 14, 4, 8, 13, 24.
A: Die Antwort ist wahr.

Die ungeraden Zahlen in dieser Gruppe addieren sich zu einer geraden Zahl: 17, 9, 10, 12, 13, 4, 2.
A: Die Antwort ist falsch.

Die ungeraden Zahlen in dieser Gruppe addieren sich zu einer geraden Zahl: 15, 32, 5, 13, 82, 7, 1.
A:

Die Antwort ist wahr.

The result shows that few-shot prompting is not sufficient in this case to generate reliable answers for this type of reasoning task. The example provides basic information about the task. A closer look shows that the task requires additional reasoning steps. One possible improvement is to break the problem into sub-steps and demonstrate them to the model. In this context, so-called Chain-of-Thought (CoT) Prompting⁵ has recently become popular, addressing more complex arithmetic, everyday, and symbolic reasoning tasks.

In summary, providing examples can help solve certain tasks. If zero-shot prompting and few-shot prompting are not sufficient, this suggests that the knowledge stored in the model is not adequate for the task. In such cases, it is recommended to consider fine-tuning the models or using advanced prompting techniques. The following section explains one of the popular methods, chain-of-thought prompting, in more detail.