StreamingLLM gives language models unlimited context

About StreamingLLM gives language models unlimited context

This is a web article from TechTalks that explains StreamingLLM, a research technique for extending the context of large language models. Language models have a fixed context length set by their architecture and training, so long chat sessions can reach that limit. The article says StreamingLLM was developed by researchers from Meta AI, MIT, and Carnegie Mellon University, and it describes the technique as able to extend a model's context to millions of tokens without vast compute and memory resources while preserving high-quality performance.

The article contrasts StreamingLLM with other approaches. It notes that longer context usually requires retraining, which can be expensive, and that a sliding context window forces recomputation of the cached attention values, which lowers throughput and can reduce quality. It also describes the researchers' observation that autoregressive models such as Llama-2 and GPT-3.5 assign a large share of attention to the initial tokens. The page excerpt ends before the method is fully explained.

Key features

• Describes extending a language model's context to millions of tokens
• Aims to avoid the large compute and memory costs of longer context
• Targets long chat sessions that reach a model's context limit
• Uses the observation that initial tokens receive large attention scores
• Discusses Llama-2 and GPT-3.5 as examples of autoregressive models

StreamingLLM gives language models unlimited context at a glance