Neural Networks Jan 2026 · #93 most cited · 63 citations

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Xin Cheng et al.

arXiv:2601.07372

In short

Engram adds a lookup memory to transformers, a modern take on N-gram embeddings retrieved in constant time, as a second kind of sparsity alongside mixture-of-experts. At 27B parameters it beats an equal-compute MoE model, helping reasoning, code and long-context retrieval more than expected.

Why it matters

It proposes memory lookup as a new scaling axis for LLMs, separate from compute.

Read first

The 4 Field Guide ideas this paper leans on.

Language & LLMs · Landmark Transformer A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model. Neural Networks · Standard Mixture of Experts An architecture where multiple specialized sub-networks (experts) process inputs, with a gating network routing to relevant experts. Language & LLMs · Reference N-gram A contiguous sequence of n items (words, characters) from text, used in language modeling and feature extraction. Language & LLMs · Landmark Embedding A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

Starting from scratch? The full route 13 ideas · basics first

Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Feature ✓ understood
A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.
Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Deep Learning ✓ understood
A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.
Representation Learning ✓ understood
Learning useful features or representations of data automatically, rather than hand-crafting them.
Embedding · read first ✓ understood
A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.
Attention Mechanism ✓ understood
A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.
Transformer · read first ✓ understood
A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.
Feedforward Network ✓ understood
A neural network where information flows in one direction from input to output without cycles.
Mixture of Experts · read first ✓ understood
An architecture where multiple specialized sub-networks (experts) process inputs, with a gating network routing to relevant experts.
Token ✓ understood
The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.
N-gram · read first ✓ understood
A contiguous sequence of n items (words, characters) from text, used in language modeling and feature extraction.

In the frontier

Rank: #93 of 100
Citations: 63; as of Aug 9, 2026
Published: Jan 2026

Topics: Model architecture , Efficiency and serving , Long context and attention

Selection: 1kpapers.com by Together AI, most-cited as of Aug 9, 2026

Nearby papers

Canon · 2002 BLEU: a Method for Automatic Evaluation of Machine Translation It showed that an automatic metric can drive a field’s progress, and it is still reported in translation papers. Canon · 2013 Efficient Estimation of Word Representations in Vector Space It made embeddings, meaning as geometry, the starting point of modern NLP. Canon · 2017 Attention Is All You Need Every major LLM, and most modern vision and speech models, is a transformer. Canon · 2018 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding It established “pre-train once, fine-tune everywhere”, and still powers much of search and classification.