Longformer

Longformer, developed by the Allen Institute for AI (AI2), addresses the scalability limitations of standard Transformer models like BERT, which utilize a self-attention mechanism with quadratic $O(n^2)$ complexity relative to sequence length. By implementing a localized sliding window attention mechanism combined with task-specific global attention, Longformer reduces this complexity to $O(n)$, enabling the processing of sequences up to 4,096 tokens and beyond on standard hardware. This architectural shift is critical for document-level tasks such as long-form question answering, coreference resolution, and document classification, where traditional 512-token limits result in significant information loss. As of 2026, Longformer remains a foundational architecture in the research community, frequently used as a backbone for specialized models in legal, medical, and scientific domains. It is fully integrated with the Hugging Face Transformers ecosystem, allowing for seamless deployment across various compute environments. The model's ability to maintain local context while selectively attending to global anchors makes it uniquely suited for structured document analysis where both local syntax and global semantics are required for accurate inference.