Operating a long‑horizon agentic optimization swarm

I used a swarm of AI coding agents to write GPU code for the Open Kernel Challenge and reached #7 of about 1,270 teams.

Almost all of the progress came from the setup around them and a handful of structural decisions.

How the contest worked

The Open Kernel Challenge asked teams to make one AI model's layers run faster on an NVIDIA H200 GPU. Each entry was a kernel, timed against SGLang.

In 13 days my score went from 1.41 to 2.47. Most of that gain came in the first week, but other teams improved faster, and my rank fell from #9 to #144. The setup below brought me back to #7 in the last four days.

How my setup worked at a high level

The whole harness was a Python script and a few shell scripts on my workstation, plus one or two rented spot H200s. A single Claude Code session acted as the supervisor: it wrote the harness, directed each lane's work, ran research, watched for stalls and submitted the best file.

Worker lanes ran in parallel: each one a loop that started a fresh Codex agent (codex exec on my subscription) for every attempt, with the lane's files and the current best code as its only context.

When an agent finished, the harness sent its code to the judge on a GPU box, which timed it against the current best using the contest's own timing code. The keep rule then decided: a kept change became the new best code, and anything else became one line in the attempt log. A separate research agent wrote what it found into the same files every agent read, and everything was committed to git every hour.

Agents never stepped on each other's toes, because none of them wrote to the shared files. Each attempt worked in its own folder on a copy of the current best code, and only the harness wrote a result back, after the keep rule had decided. Lanes worked on different parts of the program, one integrator lane merged their kept changes, and judge runs on each GPU waited in line behind a lock, so two timings never overlapped.

Tokens came almost entirely from my subscriptions: the Codex workers ran on my ChatGPT plan, with a couple of usage resets when it ran low, and the supervisor ran on my Claude plan. GPU rentals came to about $600 over the 13 days. About $240 of that went in the first two days on on-demand H200s, and the four-day final phase on spot H200s cost under $175.

The optimization lineage

My decoder layer went through four generations. For the first five days the agents wrote their own kernels from scratch: a fused attention kernel on FP8 tensor-core instructions, a lossless 12-bit packing of the BF16 projection weights to cut memory traffic, a fused routing kernel, and expert kernels that unpack the 4-bit weights on the fly. That took the score from 1.41 to about 1.95.

On Oct 27 vjeux's public entry beat my layer on 11 of 12 shapes, so I replaced mine with it and kept tuning from there. On Oct 30 the harness started from alygg77's public file, which was ahead of everything else I had.

From then on, progress came from about 110 kept changes, most worth 0.3 to 0.6% each, and nearly all of them about moving data and synchronizing work: removing a grid-wide barrier in the expert step, fusing the residual add into the normalization, overlapping each kernel's startup with the previous kernel's tail, merging small kernels so the CUDA graph has fewer steps, and re-tuning how the work splits across the GPU for each batch size. On Nov 2 the agents rewrote the accumulated file into readable modules, and that clean version is the line that reached #7.

Stay tuned for a detailed write-up of what the agents actually did inside the kernels and how the whole system fits together. That understanding is the most valuable output of this exercise for me, because steering the swarm gave me an in-depth understanding of this kind of low-level optimization work.

Key takeaways

Here are the key takeaways from learning how to design, operate and evolve a system like this.

1. Make your evaluation match the real scorer before optimizing anything

An agent swarm finds the flaws in your benchmark faster than it finds real speedups. In my first week my test setup differed from the contest's in a few small ways, and the agents kept finding gains that were really quirks of my setup and never showed up in the official score.

So I built a judge that runs the contest's own timing code on my own GPU, and it matched the official score almost exactly.

2. Never let an agent judge its own change

Agents grade their own work generously, so in a speed contest they report a steady stream of fake gains. In my setup the agents only proposed changes, and a separate script decided what to keep. It timed each change against the current best version several times and kept it only if it won by more than the normal run-to-run noise.

One public experiment shows what happens without such a rule: its dashboard reported twice the real gain, and 29% of the changes it kept made results worse.

3. Start every attempt with a fresh agent and keep the memory in files

Agents degrade over a long session, and I think this is the main reason I had to keep steering by hand. An agent would start from first principles, make real progress, and then get stuck on small variations of its earlier attempts until I stepped in.

After some quick research, I'd attribute this to a few things: models get less reliable as their input grows (Context Rot, Chroma), they make more mistakes once their own earlier mistakes are in the context (Sinha et al.), their edits get smaller and more cautious as a session goes on (one experiment), and separate runs tend to stay stuck near the ideas they started with (Fork-and-Flush).

So I threw the agent away after every attempt. Each attempt started a fresh agent, and files carried everything it needed: the best code so far, a log of past attempts, and a list of ideas that had already failed.

4. Give the swarm a research agent so it never works in isolation

A swarm that never looks outside spends its attempts reinventing the wheel, so unless the rules forbid external research and prior art, it's important to use the work that's already public.

I spun up a research agent whose job is targeted search: papers, public code, existing kernels, discussions, upstream pull requests and vendor docs. It was able to autonomously discover alygg77's public entry, vjeux's entries, and keell0renz's repo (ZhimaoL's attention kernel), which gave my agents useful guidance and a foundation to build on.

I also guided it to discover harness designs: it found EAT's supervisor design and the published work I built and evolved my own setup from (Decagon, NVIDIA AVO, HAN Lab's ablation, Fork-and-Flush).

The research agent needed steering too, because on its own it searched naively. I kept finding things it had missed over and over again, so I changed its routine by hand. It started searching Google for the past day, checking GitHub for new repositories every 10 minutes, reading X threads and the contest forum, and following SGLang, FlashInfer and TensorRT-LLM pull requests. On top of that, I gave it frequent context-dependent steers.

5. Make parallel agents build on one shared program

Six agents working on six copies of a program are a best-of-six contest: you submit one copy, and the other five agents' work is thrown away. Until Nov 1 my lanes worked that way, and several of them ended up repeating each other's ideas.

So I turned the lanes into parts of one program. Each lane worked on its own part of the shared file, and an integrator lane merged every kept change into the version I submitted, so changes from different lanes added up instead of competing.

My harness automated all of it. Each lane ran as a loop of fresh agents, every attempt started from the integrator's latest file, and twice a day the harness restarted every lane from the best file, following Fork-and-Flush.

6. Treat code readability as a performance lever

Every fresh agent starts by reading the code, so a messy file slows down every attempt. Because my keep rule rewarded only speed, the agents never cleaned anything up and just stacked new layers on top, until the file was close to a megabyte with the same step implemented many times over.

So I had the file rewritten into readable modules. The clean version was easier for every agent to work in, let separate lanes improve separate modules in parallel, and the next change on top of it took me to #7.

7. Make the structural decisions yourself and let the agents act on everything else

Agents keep executing a plan long after it stops working, so the decisions that change course have to come from you. My most useful decisions were all of that kind: stopping a team that wasn't making progress, building a proper judge, reorganizing the workers when they stalled, and rewriting the code when it became a mess. Each of my big jumps came right after one of them.

Everything else, the agents should decide and do on their own. Twice my GPUs sat idle overnight because the supervisor waited for my answer instead of fixing the problem itself.

8. Run on cheap spot GPUs, keep them saturated, and make every job resumable

Spot GPUs cost about a sixth of the on-demand price, but the provider can take them back at any moment, so they only pay off if every machine stays busy and an interruption is cheap. I sized the number of machines to the work in the queue and got an alert whenever a GPU sat idle, and my whole four-day final phase used under $175 of GPU time.

To make interruptions cheap, have your agent build a resume and rehydration mechanism before the first preemption. Every job saves its progress and copies its results off the machine, everything a fresh machine needs lives somewhere fast to download from, like Cloudflare R2, and one script turns a new machine into a working one. When a machine gets preempted, the other machines pick up its queue during the rehydration.

9. Know how noisy the score is before you trust a result

My best official score was luck. The same file can score about 1% higher or lower from one official run to the next, and my 2.4708 was a good draw of a file that typically scores about 2.43. So a change under 1% can look like a win or a loss by pure chance, and since the finals use the median of several reruns, the typical score is the one that counts.