A Simple Example for Limits on LLM Prompting Complexity | by Oscar Health | Oscar Tech | Oct, 2023 | Medium

LLMs are capable of spectacular feats, and they are also capable of spectacularly random flame-outs. A big systems engineering issue remains figuring out how to tell one from the other. Here is an example for the latter. It is well-known that LLM performance is very sensitive to prompt formulation. In some cases, that makes “algorithmic sense,” like when prompting an LLM to operate in chain-of-thought (“think through this step by step”): that effectively makes the LLM use its own output as short-term memory, which means you’re effectively chaining many LLMs together. In other cases, it doesn’t make any algorithmic sense, but human intuition helps realize why a particular prompt works: for example, prompting the LLM to “take a deep breath” apparently increases performance by a little. In this example, there is perhaps enough pattern matching in the pre-training data that this is associated with better performance, so it biases the LLM output towards that better performance. But in other

LLMs are capable of spectacular feats, and they are also capable of spectacularly random flame-outs. A big systems engineering issue remains figuring out how to tell one from the other. Here is an example for the latter. It is well-known that LLM performance is very sensitive to prompt formulation. In some cases, that makes “algorithmic sense,” like when prompting an LLM to operate in chain-of-thought (“think through this step by step”): that effectively makes the LLM use its own output as short-term memory, which means you’re effectively chaining many LLMs together. In other cases, it doesn’t