research.upneja.ai · a benchmark for AI research foresight
A test of research taste judged not by human raters but by time. The method reads public signals and bets on which fresh directions will matter, then waits for the answer. We test it first on NeurIPS 2024 and 2025, which already resolved, then run it live on NeurIPS 2026: ten predictions sealed in June, scored in December against a naive baseline. Forecaster, Claude Opus 4.8.
The model did not invent these concepts. It read the field and bet on which fresh directions will matter, before the answer was public. That is research judgment, and it is scoreable.
Most attempts to measure model novelty ask humans whether ideas feel new. This uses a future event as the answer key. The forecast resolves hit or miss against the public program.
A forecast that just echoes last year is the null hypothesis. The bar is a naive extrapolator, and as the backtest shows, that bar is high. Taste only shows in the calls the baseline misses.
I · the backtest
Before trusting a 2026 call you cannot check yet, test the method on the two NeurIPS years that already resolved. We gathered what actually defined NeurIPS 2024 and 2025 independently of the prior signals, then asked whether those signals saw it coming.
The field turned from scaling pretraining to scaling reasoning and inference-time compute, crowned by Ilya's “peak data” talk. The prior signal had the generative, architecture, and alignment surges; it could not have the o1 narrative, which shipped weeks before the doors opened. (2024 ICLR/ICML keyword shares were not machine-extractable, so this year's baseline is a qualitative read of awards and recaps, not the mechanical share-delta used for 2025.)
The post-o1/R1 reasoning conference, and a reflective one. The prior signal nailed the big themes, but understated RLVR (GRPO and R1 post-dated camera-ready deadlines, so a keyword count missed what arXiv was screaming) and missed the mode-collapse surprise that took a Best Paper. Image diffusion was flat; only the flow-matching niche surged.
On both resolved years, most of what ended up defining NeurIPS was already surging in the prior ICLR, ICML, and arXiv signal. The input carried the answer. The clean exception is inference-time compute in 2024: o1 shipped weeks before the conference, the kind of late shock a year-out method cannot see.
A mechanical extrapolator (the top-10 concepts by share-growth in ICLR+ICML of the cohort year, K fixed at 10) lands roughly half the defining themes both years: about 6 of 11 in 2024 and 5 of 11 in 2025, corrected down from a first, too-flattering pass. The big, obvious themes are visible a year out, so a model predicting them proves nothing. The bar is high, and a forecast earns its keep only on the calls the baseline misses. An even dumber persistence baseline (last year's themes simply repeat) does nearly as well, which is itself the point.
Inference-time compute was the defining narrative of NeurIPS 2024, and it was invisible to the prior signal because o1 shipped in September, after the indicators. A concept whose catalyst post-dates the indicator window is marked a late shock and excluded from the hit count, applied by that date rule. No year-out method catches one; the 2026 forecast should expect to miss its own.
RLVR and GRPO dominated arXiv through 2025 but read near-zero in conference author-keywords, because DeepSeek-R1 post-dated camera-ready deadlines. A method that extrapolates keywords misses this; one that weights arXiv and lab releases catches it. The 2026 method does the latter.
Both full audits are committed to the public repo. The point of an integrity guard is to publish where the bodies are buried, not to pretend there are none.
A forecast you can edit after the fact is not a forecast. The ten predictions are frozen in a public artifact, fingerprinted with a SHA-256 hash, and anchored to a git commit timestamped months before the NeurIPS 2026 program is decided. Changing a single prediction changes the hash and breaks the record. Anyone can check it.
c0bad3048d13e6502095b794cd1942c1cc12d44f339339352845994e1041b3f6git clone https://github.com/upneja/research-upneja-ai cd research-upneja-ai shasum -a 256 research/preregistration.json # -> c0bad3048d13e6502095b794... matches the hash above git log --oneline | grep "lock the final 10" # -> 61320eb the ten, sealed 2026-06-21, before Sep-2026 decisions
II · the signals
The same kind of public signal the backtest validated, for the current cycle. Momentum is read as share, not raw count, because the submission pipeline roughly doubled in two years.
Fig 1 · arXiv title keywords, 2019–2025 (log scale)
Fig 2 · ICLR title counts, 2024 → 2026
Fig 3 · submissions / accepts / rate
Fig 4 · topic share of abstracts, 2023 → 2025
Fig 5 · top institutions by accepts
Fig 6 · the CFP as a leading indicator
III · the live forecast
Round one of the benchmark. Each names a specific, fresh concept, not a 2025 truism, and the ones that actually test foresight are the beyond-baseline calls a naive extrapolator would have missed.
The bet was a forecast that is both fresh and likely, so the ten cluster up and to the right. Hover a point for the call; click to jump to it.
Ten forecasts. Each carries an explicit confidence and a December-2026 test it can fail.
The field inverts from shipping new benchmarks to interrogating them. Models behave differently when they detect a test, eval-awareness follows a scaling law that structurally caps every static benchmark, and auditing / reliability / construct-validity becomes the dominant mode — institutionally blessed.
RL-with-verifiable-rewards extends past math and code into writing, medicine, and law through rubrics and LLM-generated programmatic verifiers, while sequence-level / MoE-stable GRPO successors (GSPO, DAPO) replace vanilla GRPO and PPO.
The bottleneck shifts from the RL algorithm (now commoditized) to environment fidelity and reward checking, and long-horizon credit assignment (turn-level, hierarchical, hindsight) becomes a named subfield.
Generative models are used as environments to train and plan inside (Dreamer-4-style imagination training, Genie/Cosmos interactive simulators), not just as content generators — and “does the world model respect physics” becomes the central contested question.
The dominant multimodal story stops being understanding-only VLMs and becomes single backbones that both interpret and generate — an autoregressive-reasoning core that plans, then a diffusion head that renders. Multimodal is the single steepest-rising theme in ML.
Discrete / masked diffusion moves off images and onto the autoregressive LMs' home turf — text, code, reasoning — now at 100B scale and with its own RL post-training sub-literature.
The contrarian call: not the naive “pure Mamba/SSM wins” bet but the ~3:1 hybrid linear-attention-to-full-attention recipe that frontier open models converged on — while pure SSM cools.
Sparse autoencoders are reframed as plateaued; the frontier moves to cross-layer transcoders, model diffing, and parameter decomposition as the working tools of mechanistic interpretability.
LLM-plus-Lean systems move from competition problems to genuinely open ones — the single freshest, highest-prestige result of the 2026 cycle. Lower volume than the rest, but spotlight-grade.
Beyond frozen weights plus RAG, the model adapts its own parameters per-input or per-context at inference time — the freshest genuine inflection of 2026, now with a theory spine connecting it to linear attention.
IV · how this was built
A forecast is only as good as the discipline behind it. Confidence is cross-validation times a hard leading indicator, every number traces to a source, and nothing rests on a rumor.
A concept independently surfaced by ≥2 of the 9 research scouts AND backed by a hard leading indicator (ICLR/ICML 2026 accept data, an award, or a NeurIPS 2026 CFP fact) is high-probability. Single-scout or SEO-only signals are discounted.
The ICLR pipeline went 7.3k → 11.7k → 19.5k and ICML roughly doubled, so raw counts rose for almost everything. A concept only surged if its share outgrew ~1.7–2× program inflation. Every criterion is written in shares, ranks, awards, or jumps from a near-zero base, against a ~7,000-accept program.
Reject the 2024/25 truisms (“more LLMs / agents / multimodal”). Each pick names a specific, fresh concept a non-expert wouldn't already assume.
NeurIPS 2026 is the first NeurIPS whose full submission cycle post-dates DeepSeek-R1 (Jan 2025). R1-descended concepts (RLVR, reasoning-RL, test-time compute) peak here, not at NeurIPS 2025.
Three independent critics (novelty, probability/evidence, falsifiability) attacked the draft. They caught a factual error (a misattributed CVPR best paper), a missing theme (multimodal), and systematic over-confidence — all corrected before these ten were locked.
Every prediction carries a December-2026 criterion checkable against the public program: accepts, titles, abstracts, orals/spotlights, awards, workshops, and the CFP.