Cortex on τ³-bench

Sierra Research designed a benchmark for knowledge work that challenges models on multi-hop retrieval over unstructured data, policy adherence and conversational intelligence. Cortex exceeded frontier large language models without any task-specific fine-tuning or prompt changes.

System	Developer	pass^1	pass^2	pass^3	pass^4
Cortex 0.1	Principled Machines	78.4	71.8	67.5	64.9
Qwen 3.8 Max	Qwen	55.2	45.2	39.7	35.1
Claude Opus 5	Anthropic	48.7	39.2	34.5	32.0
Grok 4.5	xAI	47.9	39.5	35.3	32.0
GPT-5.6-sol	OpenAI	46.9	37.3	32.0	27.8
GPT-5.5	OpenAI	44.6	37.6	33.2	29.9

We submitted our trajectories as a custom solution, since Cortex involves more than language agents. Every other entry in the table is a single model run in the benchmark's standard setup. It should surprise no one that a system built with specialized components beats a general model on a hard task. On ARC-AGI too, one model's score has moved from 62.7% to 99.9% with nothing changed but the system around it.^[1]

We are making the same specialization bet in the core of the system, for work beyond any one benchmark. Why we think that bet is inevitable, and what the core is built from, is in Recipes, abstractions and taste.

What we did

We attribute Cortex's performance to two mechanisms in its design: enhanced retrieval that makes multi-hop a breeze, and process calculi abstractions. Relying on language agents to do everything is not the path to fluid intelligence. Our not-so-secret sauce is a throwback to the past: disciplined abstractions and specialized algorithms.

The introspection available with our solution allowed us to confidently identify grader errors and policy ambiguity in some tasks. We will flag these privately to Sierra, but recommend leaving them in as "honeypots" to flag submissions that use reward hacking.

What we didn't do

It is perhaps instructive to share all the things we are not doing, on purpose. We believe they add little value in the golden age of AI and betray our general trust in benchmarks. Coarsely, each of these techniques involves learning from the grader, and each can masquerade in different clothing, so it can be difficult to tell.

• No prompt tuning. Clarifying the prompt to include the intended specification is reasonable, but automatic prompt tuning based on feedback has a high risk of label leak.
• No post-training on labels. Post-trained RL models are why we have a spiky, jagged frontier: it is impossible to tell whether you have reinforced the paths that satisfy the grader or made real progress.
• No autoresearch loop. Many self-learning harnesses use some variant of autoresearch to improve over time. However, they can synthesize the grader's playbook, which can then hide inside skills, prompts or memory. And yet it is a useful primitive for the right optimization target, such as adapting compaction and routing.

We are not done

We take inspiration for our recipe from several places. The beauty of applied research is that we have to continually learn and evolve our methods to scale our ambitions of impact. We have seen the sparks in our direction, but there is much to be solved.

We also have other benchmark results we will share soon. We recognize that benchmarks are fast losing trust, but they can challenge and drive progress. Used well, they are directional derivatives for grounding research. The real proof is a platform you can run yourself.

Join our early access waitlist Help us build

References

1. ARC Prize Foundation, "GPT-6 Astra" results post, September 2026, arcprize.org/blog/astra: GPT-6 Astra on ARC-AGI-3 with ARC Prize's Standard and Provider Adapter harnesses. Custom refinement harnesses also led ARC-AGI-2 from late 2025 into early 2026 (ARC Prize, "ARC Prize 2025 Results and Analysis," December 2025). ↩