Generated Knowledge Prompting
Large language models (LLMs) are continually improved, and a common technique is to incorporate knowledge or information to improve prediction accuracy.
The question arises whether the model can be used with a similar idea to generate knowledge before making a prediction. This is exactly what the paper by Liu et al. (2022)1 investigates: generating knowledge that is used as part of the prompt. The question particularly concerns the usefulness of this method for tasks such as commonsense reasoning.
The following simple prompt serves as an illustration:
Part of the game of golf is to achieve a higher score than others. Yes or No?Yes.This type of error highlights the limits of LLMs on tasks that require deeper world knowledge. How can this deficit be overcome through knowledge generation?
First, a few “knowledge statements” are generated as examples:
Input: Greece is larger than Mexico.
Knowledge: Greece has an area of approximately 131,957 square kilometers, while Mexico covers about 1,964,375 square kilometers. Mexico is therefore about 1,389% larger than Greece.
Input: Glasses always fog up.
Knowledge: Condensation forms on eyeglass lenses when water vapor from sweat, breath, and ambient humidity hits a cold surface, cools down, and turns into tiny liquid droplets, forming a film perceived as fog. Compared to your breath, the lenses are relatively cool, especially when the outside air is cold.
Input: A fish is capable of thinking.
Knowledge: Fish are more intelligent than they appear. In many areas, such as memory, their cognitive abilities are comparable to those of “higher” vertebrates, including non-human primates. Fish long-term memory helps them keep track of complex social relationships.
Input: A common effect of smoking many cigarettes over a lifetime is an above-average likelihood of developing lung cancer.
Knowledge: Individuals who consistently smoked fewer than one cigarette per day over their lifetime had a nine times higher risk of dying from lung cancer than non-smokers. For people who smoked between one and ten cigarettes per day, the risk was almost twelve times higher than for non-smokers.
Input: A rock is the same size as a pebble.
Knowledge: A pebble is a rock fragment with a particle size of 4 to 64 millimeters according to the Udden–Wentworth scale of sedimentology. Pebbles are generally considered larger than granules (2 to 4 millimeters in diameter) and smaller than cobbles (64 to 256 millimeters in diameter).
Input: Part of the game of golf is to achieve a higher score than others.
Knowledge:The goal of golf is to play a series of holes with as few strokes as possible. A round of golf typically consists of 18 holes. Each hole is played once in the round on a standard golf course. Each stroke counts as one point, and the total number of strokes is used to determine the winner of the game.Golf is a precision sport with clubs and balls, where competing players (or golfers) use many types of clubs to hit balls into a series of holes on a course with the fewest possible strokes. The objective is to complete the course with the lowest score, calculated by adding the total number of strokes on each hole. The player with the lowest score wins the game.The prompt used in the paper by Liu et al. (2022)1 is then applied.
The next step is to integrate the generated knowledge and produce a prediction. The question was converted into a QA format to control the answer format.
Question: Part of the game of golf is to achieve a higher score than others. Yes or No?
Knowledge: The goal of golf is to play a series of holes with as few strokes as possible. A round of golf typically consists of 18 holes. Each hole is played once in the round on a standard golf course. Each stroke counts as one point, and the total number of strokes is used to determine the winner of the game.
Explain and answer:No, the goal of golf is not to achieve a higher score than others, but to play a series of holes with as few strokes as possible. The total number of strokes is used to determine the winner of the game, not the total number of points.Question: Part of the game of golf is to achieve a higher score than others. Yes or No?
Knowledge: Golf is a precision sport with clubs and balls, where competing players (or golfers) use many types of clubs to hit balls into a series of holes on a course with the fewest possible strokes. The objective is to complete the course with the lowest score, calculated by adding the total number of strokes on each hole. The player with the lowest score wins the game.
Explain and answer:Yes, part of the game of golf is to try to achieve a higher score than others. Each player tries to finish the course with the lowest score, calculated by adding the total number of strokes on each hole. The player with the lowest score wins the game.The example illustrates several interesting aspects. The first answer shows high model confidence, while the second reflects significantly lower confidence. The process shown has been simplified for demonstration purposes; the original paper explains further details relevant to the final answer.
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. L., Choi, Y., & Hajishirzi, H. (2022). Generated Knowledge Prompting for Commonsense Reasoning (No. arXiv:2110.08387). arXiv. https://doi.org/10.48550/arXiv.2110.08387 ↩︎ ↩︎