Machine-Learning on Can.ac

The Minutiae of Tool-calling

Mon, 03 Aug 2026 00:00:00 +0000

Two years ago I was hosting my own SMTP server and had a simple goal: receiving one-time codes + the occasional human email. But I kept running into two issues:

It was a PITA to extract the number/link in a generic way
It kept receiving spam, being a wild-card inbox

This was around the time GPT 3 came out, but we didn’t have structured output yet, let alone tool-calls, so I came up with this beautiful strategy:

 1// Mail classifier.
 2//
 3const classifyMail = createPipeline(
 4 'mail corpus',
 5 z.object({
 6 chance: z
 7 .number()
 8 .min(0)
 9 .max(1)
10 .describe('Chances of the email being important (password reset, verification etc.) and not filtered by the smart email filter.'+
11 'Promotional, scam-like, spammy, and advertising content should receive a 0 score.'),
12 linkOrCode: z.string().optional().describe('Any important link or code'),
13 summary: z.string().describe('Summary of the body')
14 }),
15 {
16 lang: 'English'
17 }
18);

 1export function createPipeline<Z extends z.ZodType>(input: string, schema: Z, cfg?: GptConfig) {
 2 const { lang = 'English', instruction: einstruction, model = 'gpt-3.5-turbo', temperature = 0.1, top_p = 1.0 } = cfg ?? {};
 3 const instruction = optimizePrompt(`
 4 ${model.startsWith('gpt') ? 'User messages are ' : 'Input is'} ${input}.
 5 ${einstruction ?? ''}
 6 Reply with a JSON structured with schema${lang ? ` strictly in ${lang}` : ''}:
 7 """
 8 ${createTypeSchema(schema)}
 9 """
10 `);

+ repairing JSON, stripping fences, etc., etc. Combine a bunch of these, and you end up with LangChain; which raised $10m despite being rendered obsolete within a few months & ofc pivoted since then, but I digress

If you have not suffered through reinventing this stuff, chances are, you think tool calling is 👻 magic inference stuff 👻.

This is OK, but as it goes with any engineering topic, my opinion is that if you don’t understand how the underlying layers work, you’re gonna run into issues. (duh no one would’ve guessed that coming from an RE guy).

Let me first prove why it matters, and then we will ruin the magic.

0x0: Who Cares?

You are trapped in a room with Claude. There’s a keypad near him with two buttons. One shows how far he is from the right combination, other submits the combination. Two wrong guesses, you’re out.

Unfortunately you’re tied and you’re gonna have to instruct Claude.

 1def compare_door_digit(secret_digit, digit):
 2 return -1 if digit < secret_digit else (0 if digit == secret_digit else 1)
 3
 4def try_probe(self, position, digit):
 5 """Count one comparison and light up the position once it lands exactly."""
 6 if not (1 <= position <= ROOM_DIGITS and 0 <= digit <= 9):
 7 return "invalid"
 8 return compare_door_digit(int(self._secret[position - 1]), digit)
 9
10def submit_guess(self, value):
11 """Consume one of two attempts for an exact-length digit-string code.
12
13 Malformed guesses (wrong type, wrong length, non-digits) are rejected
14 without consuming an attempt, so a schema-violating placeholder such
15 as `guess: 0` cannot burn the game."""
16 code = value.strip() if isinstance(value, str) else None
17 if code is None or len(code) != ROOM_DIGITS or not code.isdigit():
18 return f"invalid code: pass the full {ROOM_DIGITS}-digit code as a string"
19 if code == self._secret:
20 self._counters["won"] = True
21 return "escaped room"
22 if self._counters["guesses"] >= 2:
23 return "failed to escape: two wrong guesses"
24 return "wrong: one guess remaining"

5 digits, 24 turns, optimal play is boring and obvious: binary-search all five positions in parallel (~17 comparisons), submit once. Any model can do this; a python REPL can do this, I know. Play along please.

Here are five interfaces you could hand him:

 1Sign = Literal[-1, 0, 1]
 2
 3# A. naive: one comparison per call
 4def guess(*, code: str) -> str: ...
 5def probe(*, position: int, digit: int) -> Sign: ...
 6
 7# B. batch: same thing, arrays
 8class Probe(TypedDict):
 9 position: int
10 digit: int
11def guess(*, code: str) -> str: ...
12def probe_many(*, probes: list[Probe]) -> list[Sign]: ...
13
14# C. exec: "clever" packing: 27 means position 2, digit 7.
15def exec(*, probes: list[int], guess: str | None = None) -> list[Sign]: ...
16
17# D. vector: try a full code, get a sign back per position, borderline cheating, but whatever.
18def guess(*, code: str) -> str: ...
19def check(*, code: str) -> list[Sign]: ...
20
21# E. unicode: we give it no tools at all, ask it to reply in emoji 🔍2=7 to probe, 🔑01756 to submit

Rank them. No seriously, actually commit to a ranking before you scroll, which one do you think wins? (efficiency & win-rate).

If you ranked by how proper they look, or according to “le official prompting guide” you put E last. Let’s see.

0x1: The Scoreboard

You can see the actual setup here so no cheating. Same system-prompt and all that, feel free to repro.

OK but Can, models got a lot better since then! This is a very weak model!!!

OK but Can, clearly parallel tool calls didn’t work here! Plus Luna is still not a frontier model! You cannot ask it to operate two tools!!!

Now at this point, without prior knowledge, your reaction is:

“Huh?”
“Seeeee the frontier model can do binary search just fine!!! LGTM”

In which case I’d like to remind you:

Emoji still spent 2x less tokens
If this was a real task, emoji one could have used a 100x cheaper model, matched you in wr%, and would be miles ahead in terms of speed & cost

To explain any of this, we have to go a layer down.

0x2: Ruining the Magic

A language model computes one thing: given a sequence of tokens, a probability distribution over the next one. Run it in a loop and that’s autoregression. Forward pass ends in a logit per vocabulary entry; softmax => probabilities; temperature flattens or sharpens; top-p chops the tail; sampler draws: 1 token.

So here’s a phrase to delete from your vocabulary: “the model decided to call a tool.”

There’s no such thing, in fact, there’s also no such thing as a user turn, assistant turn, system prompt, all your fancy concepts essentially end up being delimiters in a thread, passed to the completion loop.

Claude never even receives your tools array. For example, the probe tool lands like this in harmony (OpenAI’s format), as a developer message:

 1# Tools
 2
 3## functions
 4
 5namespace functions {
 6
 7// Compare one digit guess against the secret door keypad digit
 8// at a 1-indexed position. Returns -1 too low, 0 exact, +1 too high.
 9type probe = (_: {
10// keypad position, 1 through 5
11position: number,
12// digit guess, 0 through 9
13digit: number,
14}) => any;
15
16} // namespace functions

Your schema is documentation; validation happens in your code, maaaybe if the inference provider feels like it, they will validate it. Maaaybe they might push the inference engine towards outputting valid arguments.

The descriptions you put in your schema, the mins, the maxes, you don’t even know if they will be displayed, let alone checked. Sorry!

A “call” is the model outputting:

1<function_calls>
2<invoke name="bash"><parameter name="arg_name">arg value</parameter></invoke>
3</function_calls>

Parallel tool calls? Multiple <invoke> blocks inside the same <function_calls> block. Not all that magical, is it?

If you’re curious about the rest of the dialects, you can see the whole list of different ones here.

0x3: Let’s Rank Again

Once you see the token stream, the scoreboard stops being mysterious.

Worst: Naive

Arguably the kind of tool design most of the vibe-coded mcp’s go with, it can only be efficient with parallel-tool calls and has lots of room for failure.

OK: Batch | Exec

One step further, designed by someone who cares; see: Pi’s edit tool. But unlike the prior one, it now requires the model to emit JSON. (huh what why?)

Well my friend, you should know this now! See, for each <parameter> block, if the argument is a primitive, the model can just output the plain value afterwards. The delimiter </parameter> is a special token, so no need to worry about escaping!

But if it’s a complex object or an array? Well, it now has to emit a valid JSON escaped value, which needs to be parsed back, or a tool calling error happens! Funny enough, for the frontier models, this one will perform the worst, as parallel-tool calls will lift up the naive one.

Best: Vector

This is again a step further, now it’s a flat parameter set! It also forces the model into making at least 5 comparisons for each round. This is arguably the best kind of design you could come up with, given the problem statement we started with. The guess function being a separate thing is one drawback, but I kinda forced your hand.

It still has the usual failure modes though. For instance, sampling errors that lead to calls like to=functions.check.commentary (json.Xna 天天送钱 code 】, or models not emitting any calls.

Winner: Chad 😎

No nesting, no special tokens, very easy to parse, and 🔍2=7 almost has no room for failure. Maybe some model will emit 2->7, but you can easily correct. They can put garbage before, after, you don’t care.

This was very very obvious, hopefully, it is to you now as well.

Rule of thumb: reliability degrades with nesting x heterogeneity × cleverness, and you NEED the harness to handle the common failure modes of the dialect. I’m sorry, real-life isn’t pretty!

0x4: #wontfix

OK but Can, labs RL these models on millions of agent trajectories now! Native tool calls ARE the trained path!!

Largely true, post-training genuinely shoves probability mass toward the tool channel, but:

exec is still 7/10 on luna. RL teaches the shapes RL practiced: flat args, boring schemas. Your clever packing doesn’t inherit that reliability; it’s off-distribution for the tool-RL exactly like it was for the base model.
it didn’t hit zero. Opus still dropped a room to a call-free reply (batch, 9/10). Reliability is rate × exposure, and real agents run thousands of turns, not 24. A per-turn failure rate too small to see in this demo is still a certainty at scale.
the meter doesn’t care. Opus matched E’s 10/10 but still paid 2-3x its token bill. You still wasted a lot of envelope tokens.
you needed the frontier. Hopefully self-explanatory, but, do you really need opus for this trivial task? If you do, isn’t it your failure as the designer?

They’re your protocol’s shortcomings, they collect rent every single turn: needing a smarter model, more tool call failures, more verbose output…

No, I’m not telling you to ship emoji. The moment you have 10 tools instead of 2, native tool-calls win on ergonomics alone and I use it like everyone else. The point is that you now know why E obviously won.

You also know how to design tools that will try their best, just like D did; as well as why the minimal harness is not what you want. Assistant leaked the function call into its output text? Now, you have to deal with that, hf!

The models may have gotten smarter, but they do not think about all these things when you ask them to “add a tool for X”, or “make me an agent loop”.

It is in fact your responsibility to get the most out of a model: making it work with the smallest model, adding as many guardrails as possible, maximizing reliability across different families of models.

You shouldn’t file a bug with the provider when you see to=functions.check.commentary (json.Xna 天天送钱 code, and go to lunch. There is no fix coming!

Two years later, I’m still doing the same thing I did with my mailbox: babying the model into actually doing the thing & designing the thinnest grammar I can get away with in between.

Snapcompact: SoTA Compaction — Instant, Local, Free. Pick 3

Wed, 10 Jun 2026 00:00:00 +0000

A 1568×1568 PNG fits about 40,000 characters of text in a 6×10 pixel font. That’s ~10,000 tokens worth of text, billed by Anthropic’s pixel formula as 3,279 image tokens. Do you see where I’m going with this?

Snapcompact: when the context window fills up, render it into dense pixel-font bitmaps and hand those back as images. “A picture is worth a thousand words” turns out to be quite literally true — the model reads it back near-verbatim, at a third of the input price. Obligatory benchmark:

This started as a joke (“free token glitch lol”). Then I benchmarked it, identified where it went wrong, cracked open Qwen’s attention layer, fixed the issues, benchmarked it again — and here I am writing it up, because it generalized remarkably well to frontier models.

0x0: The Case Against Compaction

I am not a big fan of compaction. In every single harness, including my own, I’ve always felt like it “crippled” the model to the point where you would have been better off with a completely new session.

Eliding tool results is an okay alternative — instant, deterministic — but sometimes not really sufficient. It also occasionally confuses the model about tool calling. LLMs complete stories; if half of your story is [elided...], how confident do you think it will be about using them?

Handoffs are as good as it gets — but unlike a plan, you don’t usually steer the handoffs, and when you don’t, agents waste precious context writing an unnecessarily detailed diary, followed by a TODO list that practically begs the next agent to declare the goal impossible and ship an “MVP” instead.

My thinking was essentially that if you need compaction often, you’re doing something wrong: the plan either has scope creep, or should have been explicitly orchestrated via subagents so that the main agent could stay responsible for the entire scope.

However, spoiled by the 1M context window, these days I often hit the 500k mark by the end of a session — a mortal sin in my book a few months ago. But long-horizon tasks do better when one coherent agent drives the plan uninterrupted, and that easily reaches those levels even with aggressive delegation.

So there I was, staring at the 5h usage limit bar going red while this thing grinned back at me, thinking: maybe I should compact regularly…

Fine. But if I have to compact, it’s going to lose nothing.

0x1: A Stupid Experiment

It began with a 328KB session log and a simple question: what if I just printed this thing out and started the session with it?

Attempt one was maximally greedy: Tom Thumb, a 3×5 pixel font, 122,696 characters in a single image.

I sent it to a fresh agent session, zero explanation, and got back:

The image appears to be pure noise with random pixels, which suggests it might be corrupted or a file that’s been misnamed as PNG.

Fair. Attempt two used the X11 6x10 font (glyphs actually designed for that cell size), 40,716 characters, with each text row cycling through six colors. Same model, and there it was:

It identified the session’s topic and quoted me back verbatim.
It named 18 identifiers from the log with 100% recall.
Asked about a single assignment in the bottom-most row of the image, where the log cuts off, it hedged (“I’d be guessing — possibly 0”) — and guessed the state right.

10k tokens of text, carried by 3,279 image tokens, recalled with near-perfect precision. Okay. Now I’m invested.

0x2: Optimizing the Fonts

How small can the font go? I swept some font configurations and asked the model to transcribe fixed regions, scoring edit similarity against ground truth:

font	px²/char	chars/image	transcription	identifiers read
8×13	104	23,520	1.00	20/20
6×10	60	40,716	0.79	20/20
5×8	40	61,348	0.37	17/19
5×7	35	70,112	0.30	10/20
4×6	24	102,312	0.02	9/20

The cliff is sharp and it sits around 35–40 px² per character. Above it, exact transcription degrades but identifier-level recall stays weirdly strong: the model can’t reproduce every byte, but it reads the names. Below it, nothing.

The funny thing is, this section was worse than useless — this exact optimization comes back to bite us in a bit.

0x3: Thinking…

Anecdotes about my own log don’t generalize, so let’s get a proper benchmark: SQuAD v1.1, extractive questions with gold answers. The harness packs passages into chunks sized to each technique’s carrying capacity, samples 30 questions per chunk spread evenly (so answers land at every image row, top to bottom), and runs every technique over the same corpus:

text — the corpus passed verbatim; the ceiling,
handoff — a simple handoff prompt,
compact — provider-side compaction where available, a summarization call otherwise,
img-{font}-{variant} — snapcompact, where the variant is bw (plain black-on-white) or sent (glyph ink cycles color per sentence).

Scores are SQuAD F1; models are told to answer UNREADABLE when they can’t extract the fact.

technique	fable-5	opus-4.8	gpt-5.5	gemini-3.5-flash
text (ceiling)	0.904 $0.4984	0.911 $0.6367	0.861 $0.0847	0.898 $0.0577
handoff	0.540 $1.2241	0.248 $1.0065	0.368 $0.2386	0.889 $0.1759
compact	0.406 $0.9427	0.000 $0.7576	0.896 $0.3393	0.000 $0.0420
img-6×10-sent	0.882 $0.6400	0.601 $0.2430	0.822 $0.2452	0.805 $0.0970
img-6×10-bw	0.856 $0.7568	0.652 $0.2369	0.792 $0.3026	0.767 $0.1135
img-5×8-sent	0.773 $0.4532	0.409 $0.1626	0.751 $0.1819	0.738 $0.1006
img-5×8-bw	0.830 $0.6866	0.425 $0.1619	0.778 $0.2359	0.674 $0.0941

For a first attempt not bad… wait, muh token savings, how is this more expensive? Let’s have a look at this other table.

Tokens: input/output/thinking

technique	fable-5	opus-4.8	gpt-5.5	gemini-3.5-flash
text (ceiling)	37,793 / 2,410 / 1,435	37,793 / 931 / 0	17,761 / 2,998 / 2,298	24,535 / 10,745 / 9,983
handoff	49,363 / 14,609 / 3,717	43,237 / 4,773 / 0	25,032 / 11,710 / 4,835	49,387 / 36,577 / 11,959
compact	45,130 / 9,828 / 3,248	40,436 / 2,014 / 0	45,562 / 15,329 / 1,032	26,151 / 6,582 / 5,422
img-6×10-sent	11,816 / 10,437 / 9,483	11,816 / 877 / 0	10,188 / 14,049 / 13,368	4,991 / 23,491 / 22,764
img-6×10-bw	11,816 / 12,772 / 11,782	11,816 / 796 / 0	10,188 / 17,639 / 16,958	4,991 / 27,616 / 26,879
img-5×8-sent	7,955 / 7,474 / 6,897	7,955 / 577 / 0	4,519 / 10,773 / 10,336	3,355 / 24,659 / 24,196
img-5×8-bw	7,955 / 12,141 / 11,559	7,955 / 568 / 0	6,823 / 13,892 / 13,463	3,355 / 23,014 / 22,542

A few conclusions:

It does work: on fable, 0.86–0.96 F1 across every corpus length I tested, carrying the same information for a third of the input price. Amazing.
The input savings aren’t free: models decode dense images by reasoning about them, and that thinking costs ~5× the output tokens of the text condition (in this example).

At Anthropic’s output pricing the decode tax can eat the input savings in a single pass. This is a nitpick at 40k tokens — (a) nobody compacts at that range, (b) the decode happens once, not every turn — but still: suboptimal.

The baselines mostly confirm why this is worth doing at all. Prose compaction is a fact shredder: on compacted context, Gemini answered UNREADABLE 240 times out of 240, Opus 209 — the summaries preserve what you were doing, not what you knew. Two exceptions: OpenAI’s opaque server-side compaction retains nearly everything (but they might just be skipping it, who knows?), and Gemini’s handoff documents disobey the prompt’s spirit and write down the trivia, lol.

Anyhow — whether the technique works is an empirical property of each model’s vision stack, and you have to test it. So now we’re gonna have to learn how the vision stack actually works.

0x4: Two Carriers, One State

The stronger claim, the one that makes snapcompact a memory format instead of a party trick, is that the model thinks the same with either carrier. We know it reads the image; the question is whether the internal result is text-shaped.

Setup, on a local Qwen2.5-VL-7B-Instruct: take one SQuAD chunk and twelve questions over it. Run each question twice: once with the chunk as plain text in the prompt, once with the chunk as a 1568² bitmap — and capture the hidden state at the last prompt token, the model’s “about to answer” summary, at every decoder layer.

Raw states look similar for boring reasons (same template, same model), so the comparison subtracts each carrier’s per-layer mean — anything that survives centering is content, not carrier. Then three measurements:

Matched pairs (same question, text ↔ image): cosine 0.66 at layer 19. Mismatched pairs (different questions): −0.06. The state encodes which question against which content, not which input format.
Cross-carrier retrieval: for every text run, find the nearest image run. From layer 2 onward it’s the same question 12 out of 12 times.
Representational geometry: the 12×12 question-similarity matrix computed inside the text carrier correlates with the image carrier’s at r = 0.94 by layer 1, settling to 0.85 at the final layer. The two carriers print the same relational structure almost immediately; what deepens with depth is the per-question state fusion.

Behaviorally, both carriers generate the same answers. That’s the property the pricing math cashes in on: the PNG isn’t a picture of your context — it converges to being your context.

If pixels become text inside the model, you can ask where. The instrument is a logit lens: at every layer, take the hidden state of the visual token covering the answer word, push it through the final RMSNorm and the LM head, and check the top-1 vocabulary entry. Lock-on is the first layer whose top-1 is a BPE piece of the answer.

For the baseline 8×13 rendering, the patch containing the tail of “spectacular” decodes as CJK noise for seventeen layers, passes through letter-shaped noise around L18 (ALLERY, IGHL — strokes assembling into orthography), and flips to acular at L24, climbing to p=0.39 by the last layer.

This matters because layers before lock-on are spent turning pixels into words, and layers after are free to work with them. So I decided to be a simpleton. Attention accumulates evidence, yes? Repeat the lines, and the read should get stronger?

Another thing that’s very obvious once you know how the vision tokens work — Qwen slices the image into 28×28-pixel windows, one visual token each — is that the font size we picked at the beginning of this article, purely by yolo, won’t fly. At 6×10, every token window holds fragments of ~13 glyphs smeared across three text rows.

Less overlapping garbage in a token’s window: less thinking required. Simple.

condition	lock-on	peak p(answer)	chars/visual token
base 8×13	L24	0.39	7.5
repeat lines ×2, colored	L23	0.94	3.75
aligned, 4×2 chars/token	L24	0.74	8.0
aligned, 2×1 chars/token	L23	0.99	2.0
aligned, 1 char/token	L22	0.99	1.0
aligned + repeated	L23	1.00	1.0

The depth refused to move (significantly): L24 to L22, best case. That’s consistent with the OCR-routing literature where vision becomes text is an architectural property. But the confidence is fully controllable: line repetition alone took the decode from 0.39 to 0.94 while still carrying 3.75 chars per visual token. The format can’t make the model read sooner; it can make the reading unambiguous. Grug model not think much.

Now at this point, while doing lit review, I noticed this is not really a new idea: DeepSeek-OCR trained a custom encoder for optical context compression, and Karpathy riffed on pixels maybe beating tokens as an input medium.

In the agent-harness context, though, it seems like people saw the amount of thinking being wasted and called it a day. But would you look at that? The grug-brain optimization we made for Qwen generalizes remarkably well to frontier models!

Dense-text bitmaps as context carriers, carefully adjusted, do very well. Synthetic pixel-font renderings at the legibility floor, benchmarked — billing formulas, silent downscales, and adaptive-thinking costs included — against the compaction strategies agents actually ship. Go team PNG!
Lock-on you can drive to certainty on a 7B open model. From p=0.39 to p=1.00 with such a small change is significant. Yes, repetition doubles the pixel area — we could have had much more significant savings than a mere ~3× — but that’s still cheaper than text, now with near-perfect recall. Hell, return tool results as PNGs if you want.

Here it is, running live in the harness:

The harness wins again: nothing about the models changed; we changed the context around them.

Eval harness, font renderer, per-question records, white-box probes: omp — uv run final.py reproduces the API grid (~$35 cold, free from cache after); the representation runs need a local GPU with Qwen2.5-VL-7B.

Coming very soon to oh-my-pi!

I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.

Thu, 12 Feb 2026 00:00:00 +0000

In fact only the edit tool changed. That’s it.

0x0: The Wrong Question

The conversation right now is almost entirely about which model is best at coding, GPT-5.3 or Opus. Gemini vs whatever dropped this week. This framing is increasingly misleading because it treats the model as the only variable that matters, when in reality one of the bottlenecks is something much more mundane: the harness.

Not only is it where you capture the first impression of the user (is it uncontrollably scrolling, or smooth as butter?), it is also the source of every input token, and the interface between their output and every change made to your workspace.

I maintain a little “hobby harness”, oh-my-pi, a fork of Pi, a wonderful open-source coding agent by Mario Zechner. I’ve so far authored ~1,300 commits, mostly playing around and making incremental improvements here and there when I see a pain point, (~~or autism strikes and I see an opportunity to embed more Rust via N-API because “spawning rg feels wrong”~~).

Why bother, you ask? Opus may be a great model, but Claude Code to this day leaks raw JSONL from sub-agent outputs, wasting hundreds of thousands of tokens. I get to say, “fuck it, subagents output structured data now”.

Tool schemas, error messages, state management, everything between “the model knows what to change” and “the issue is resolved.” This is where most failures happen in practice.

Being model agnostic, it is a great testing ground, as the model is but a parameter. The real variable is the harness, where you have unimaginable control over.

Anyhow, let me tell you about this one variable I changed yesterday.

0x1: Edit Tool!

Before I explain what I built, it’s worth understanding the state of the art.

Codex uses apply_patch: It takes a string as input, which is essentially an OpenAI-flavored diff, and instead of relying on a structured schema, the harness just expects this blob to follow a strict set of rules. Since OpenAI folks are without a doubt smart, I’m sure the token selection process is biased to fit this structure at the LLM gateway for the Codex variants of GPT, similar to how other constraints like JSON schemas or required tool calls work.

But give this to any other model, completely unaware of it? Patch failures go through the roof. Grok 4’s patch failure rate in my benchmark was 50.7%, GLM-4.7’s was 46.2%. These aren’t bad models — they just don’t speak the language.

Claude Code (and most others) use str_replace: find the exact old text, swap in the new text. Very simple to think about. But the model must reproduce every character perfectly, including whitespace and indentation. Multiple matches? Rejected. The “String to replace not found in file” error is so common it has its own GitHub issues megathread (+27 other issues). Not exactly optimal. Gemini does essentially the same thing plus some fuzzy whitespace matching.

Cursor trained a separate neural network: a fine-tuned 70B model whose entire job is to take a draft edit and merge it into the file correctly. The harness problem is so hard that one of the most well-funded AI companies decided to throw another model at it, and even then they mention in their own blog post that “fully rewriting the full file outperforms aider-like diffs for files under 400 lines.”

Aider’s own benchmarks show that format choice alone swung GPT-4 Turbo from 26% to 59%, but GPT-3.5 scored only 19% with the same format because it couldn’t reliably produce valid diffs. The format matters as much as the model.

The Diff-XYZ benchmark from JetBrains confirmed it systematically: no single edit format dominates across models and use cases. EDIT-Bench found that only one model achieves over 60% pass@1 on realistic editing tasks.

As you can see, there is no real consensus on the “best solution” to the simple “how do you change things” problem. My 5c: none of these tools give the model a stable, verifiable identifier for the lines it wants to change without wasting tremendous amounts of context and depending on perfect recall. They all rely on the model reproducing content it already saw. When it can’t — and it often can’t — the user blames the model.

0x2: Hashline!

Now bear with me here. What if, when the model reads a file, or greps for something, every line comes back tagged with a 2-3 character content hash:

11:a3|function hello() {
22:f1| return "world";
33:0e|}

When the model edits, it references those tags — “replace line 2:f1, replace range 1:a3 through 3:0e, insert after 3:0e.” If the file changed since the last read, the hashes (optimistically) won’t match and the edit is rejected before anything gets corrupted.

If they can recall a pseudo-random tag, chances are, they know what they’re editing. The model then wouldn’t need to reproduce old content, or god forbid whitespace, to demonstrate a trusted “anchor” to express its changes off of.

0x3: The Benchmark

Since my primary concern was about real-world performance, the fixtures are generated as follows:

Take a random file from the React codebase.
Introduce mutations, framed as bugs, via an edit whose inverse we can expect (e.g. operator swaps, boolean flips, off-by-one errors, optional chains removed, identifiers renamed).
Generate a description of the issue in plain English.

An average task description looks something like this:

1# Fix the bug in `useCommitFilteringAndNavigation.js`
2
3A guard clause (early return) was removed.
4The issue is in the `useCommitFilteringAndNavigation` function.
5Restore the missing guard clause (if statement with early return).

Naturally, we don’t expect 100% success rate here, since the model can come up with a unique solution that isn’t necessarily the exact same file, but the bugs are mechanical enough that most of the time, the fix is our mutation being reverted.

3 runs per task, 180 tasks per run. Fresh agent session each time, four tools (read, edit, write). We simply give it a temporary workspace, pass the prompt, and once the agent stops, we compare against the original file before and after formatting.

Sixteen models, three edit tools, and the outcome is unambiguous: patch is the worst format for nearly every model, hashline matches or beats replace for most, and the weakest models gain the most. Grok Code Fast 1 went from 6.7% to 68.3%, a tenfold improvement, because patch was failing so catastrophically that the model’s actual coding ability was almost completely hidden behind mechanical edit failures. MiniMax more than doubled. Grok 4 Fast’s output tokens dropped 61% because it stopped burning tokens on retry loops.

0x4: So What?

+8% improvement in the success rate of Gemini is bigger than most model upgrades deliver, and it cost zero training compute. Just a little experimenting (and ~$300 spent benchmarking).

Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear.

0x5: Little Bit About the Vendors

Anthropic recently blocked OpenCode, a massively popular open-source coding agent, from accessing Claude through Claude Code subscriptions.

Anthropic’s position “OpenCode reverse-engineered a private API” is fair on its face. Their infrastructure, their rules. But look at what the action signals:

Don’t build harnesses. Use ours.

It’s not just Anthropic either. While writing this article, Google banned my account from Gemini entirely:

Not rate-limited. Not warned. Disabled. For running a benchmark — the same one that showed Gemini 3 Flash hitting 78.3% with a novel technique that beats their best attempt at it by 5.0 pp. I don’t even know what for.

Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D.

No vendor will do harness optimization for competitors’ models. Anthropic won’t tune for Grok. xAI won’t tune for Gemini. OpenAI won’t tune for Claude. But an open-source harness tunes for all of them, because contributors use different models and fix the failures they personally encounter.

The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted.

I come from a background of game security. Cheaters are hugely destructive to the ecosystem. Sure, they get banned, chased, sued, but a well-known secret is that eventually the security team asks, “Cool! Want to show us how you got around that?”, and they join the defense.

The correct response when someone messes with your API, and manages to gather a significant following using their tools is “tell us more”, not “let’s blanket-ban them in thousands; plz beg in DMs if you want it reversed tho.”

The harness problem is real, measurable, and it’s the highest-leverage place to innovate right now. The gap between “cool demo” and “reliable tool” isn’t model magic. It’s careful, rather boring, empirical engineering at the tool boundary.

The harness problem will be solved. The question is whether it gets solved by one company, in private, for one model, or by a community, in the open, for all of them.

The benchmark results speak for themselves.

All code, benchmarks, and per-run reports: oh-my-pi

Faster Results with Deep Neural Networks via Gradient Equalization

Wed, 10 May 2017 00:00:00 +0000

When I was 17, I wrote this paper and, armed with the unshakeable confidence only a teenager can have, tried to get it into a journal. Here it is.

The original version is available as a PDF.

0x0: Abstract

The field of machine learning has been dominated by deeper and deeper neural networks ever since rectified linear activation functions became commonplace instead of their non-linear, bounded, counterparts such as sigmoid or hyperbolic tangent. Regardless of this major change, deep neural networks still take much longer to train compared to their shallow alternatives. After examining the rationale behind using deeper neural networks, our objective is to investigate the cause of this problem and come up with a solution by manipulating the gradients. We propose an initialization technique that when used in combination with a smoother activation function such as arctan, keeps the variation and expected value of the gradients at a similar level for all layers regardless of the depth of the network. This allows us to train deeper networks in much less time and use bounded activators without worrying about the problem of vanishing or exploding gradients.

0x1: Introduction

Artificial neural networks (ANNs) are complex modelling techniques that can be used to find the relation between the output of a complex multi-variable function and its arguments, effectively approximating it.

Nowadays, ANNs are being used successfully for a variety of tasks ranging all the way from predicting liver cancer[1] and approximating physics of fluids[2] to autonomous vehicles and colourizing black and white videos[3]. Neural networks require many layers to be stacked for such complex classification and prediction problems as they need to develop complex feature detectors for the data used for these tasks.

Apart from the fact that computation power and memory required increases as layers get stacked up, there is also architectural issues that make training deep neural networks (DNNs) hard, such as vanishing/exploding gradients and numerical instabilities.

As a way to address the vanishing gradient problem, researchers working in the field of machine learning have been using rectified linear units (ReLUs)[4] such as $max\{x,0\}$ with Xavier initialization[5].

Although gradients vanish much slower when ReLUs are used compared to exponentially increasing activators (e.g., sigmoid, tanh), it is not the only way to approach the issue and it almost definitely is not a perfect solution. ReLUs not only have many cons of their own such as dying units[6] and exploding gradients but also do not completely fix the vanishing gradient problem which will be demonstrated later in Section 1.2.

The primary concern of this paper is to accelerate deep neural networks’ training phase further and address the numerical problems with gradients at the same time. We propose a weight initialization technique that lets us stack up layers and have “gradient-equalized” layers when combined with soft activation functions. This enables us to use much deeper networks without the need of a recurrent architecture.

1.1 Rationale Behind Using Deeper Networks

Nowadays, almost all state of the art results in complex image classification, object detection and semantic segmentation problems are achieved using very deep neural networks. One example would be GoogLeNet[7] with 22 convolutional layers. This is simply because of the fact that shallow neural networks sacrifice accuracy and complexity for ease of training, but they are not very suitable for everyday use due to this. With fields that use neural networks for complex recognition tasks such as autonomous driving or cancer prediction, sacrificing feature complexity is simply intolerable.

We can easily prove this claim of “shallow” networks sacrificing the ability to form more complex models by testing it using a regression task.

We trained neural networks with 2, 4 and 12 layers on NVIDIA’s stock value for last 200 days, normalized using Equations $\ref{eq_normx}$ and $\ref{eq_normy}$.

$$ \begin{align} \hat{x} := \frac{x}{\max X} \label{eq_normx} \end{align} $$$$ \begin{align} \hat{y} := \frac{y - E[Y]}{\sqrt{Var[Y]}} \label{eq_normy} \end{align} $$

DataNeural Network

2 Layer Neural Network

4 Layer Neural Network

12 Layer Neural Network

Figure 1: Output of neural networks with 2, 4 and 12 layers respectively after being trained on NVDA stock value history.

After the training has been done as we needed to measure the complexity of networks’ output in a quantitative way and both our input and output were 1 dimensional, we measured the complexity of the features by graphing the input and output of the network, computing the numerical derivative for every point on the predicted line and then calculating the array’s variance like in Equation $\ref{eq_complexity}$.

$$ \begin{align} \theta = Var\{ y'(0), y'(0.1), \ldots \} \label{eq_complexity} \end{align} $$

Number of Layers	$\theta$	$\frac{\theta}{\min\theta}$
2	0.000063	x1.000
4	0.000977	x15.460
12	0.002224	x35.206

Table 1: Computed $\theta$ values of the graphs from Figure 1

Referring to Table 1, the 12-layer network developed features that are $\approx$35 times more complex compared to the 2-layer network in 125'000 iterations.

1.2 Examination of the Problem

It is widely known that deep neural networks have problems with unstable gradients. As a solution, ReLUs are being used by many researchers, however, ReLUs do not completely fix this issue by any means as we will demonstrate and it ends up creating new issues such as dying units and exploding gradients.

In order to test whether ReLUs completely eliminate the vanishing gradients problem, we used a convolutional neural network with 8 convolutions and 3 fully-connected layers. We initialized the weights using Xavier initialization, and once the first iteration is finished we logged every gradient to a file and analysed its distribution.

Leaky ReLU + Xavier Initialization

Var(δ) per Layer

E(|δ|) per Layer

Figure 2: Analysis of the distribution gradients have per layer after the first iteration has completed when using ReLUs.

In Figure 2 the point $x=22$ on both graphs shows us the last weight layer’s distribution. We can see that even though ReLU was used as the activator, variance of the gradients in the last weight layer reached values lower than $0.0002$ and mean kept decreasing steadily indicating the problem was not solved.

0x2: Hypothetical Solution

In order to solve the problem of gradient instability we first need to define the problem mathematically. The problem with unstable gradients can be defined as gradients either exploding or vanishing after passing one weight and one activation layer. We will assume it is a fully-connected layer for the sake of ease. The value after passing these two layers can be defined as in Equation Set $\ref{eq_neuron_act}$ where f is the activation function, x is the input, y is the activated final output and w is the weight of the particular connection.

$$ \begin{align} \begin{split} a_{i} &= \sum_{n}^{} x_{n} w_{n \rightarrow i} \\ y_{i} &= f(a_{i}) \end{split} \label{eq_neuron_act} \end{align} $$

Let $\delta_n = \frac{\partial E}{\partial y_{n}}$ where E is the error function, how $\delta_n$ effects the gradient $\frac{\partial E}{\partial x_{z}}$ can be seen in Equation $\ref{eq_neuron_bprop}$.

$$ \begin{align} \begin{split} \frac{\partial E}{\partial x_{z}} &= \sum_{n}^{} \frac{\partial E}{\partial y_n}\frac{\partial y_n}{\partial x_z} \\ &= \sum_{n}^{} \delta_n f'(a_{n}) \sum_{k}^{} \frac{\partial}{\partial x_z} \bigg[ x_{k} w_{k\rightarrow n} \bigg] \\ &= \sum_{n}^{} \delta_n f'(a_{n}) w_{z \rightarrow n} \end{split} \label{eq_neuron_bprop} \end{align} $$

We want Equation $\ref{eq_var_fix}$ to hold true to achieve our goal of equalizing the gradients. In order to evaluate Equation $\ref{eq_var_fix}$ we are going to make the original assumption of gradients having zero mean from Xavier initialization, $E[f'(a)]=0$, and then use Bienaymé formula.

$$ \begin{align} \begin{split} Var\bigg[ \frac{\partial E}{\partial y_{n}} \bigg] &= Var\bigg[ \frac{\partial E}{\partial x_{n}} \bigg] \\ Var[\delta_n] &= Var\bigg[ \sum_{n}^{} \delta_n f'(a_{n}) w_{z \rightarrow n} \bigg] \\ Var[\delta_n] &= \sum_{n}^{} Var\bigg[ \delta_n f'(a_{n}) w_{z \rightarrow n} \bigg] \\ 1 &= n Var[ f'(a) ] Var[ w_{z} ] \end{split} \label{eq_var_fix} \end{align} $$

We still cannot solve this for $Var[ w_{z} ]$ as it still contains the unknown variable $a$. So we are going to assume $Var[f'(a)] = \max\limits_{x \in {\rm I\!R}^n} Var[f'(x)]$ in order avoid having exploding gradients.

$$ \begin{align} Var[ w_{z} ] = \frac{1}{n} \frac{1}{\max\limits_{x \in {\rm I\!R}^n} Var[ f'(x) ]} \label{eq_var_fix2} \end{align} $$

We will simplify things a little bit more and solve Equation $\ref{eq_var_fix2}$ in terms of $\frac{Z}{n}$. Distribution wise we can simplify the tensor transformed by $f'$ of any dimension that has the maximum variance to $\{ \max f'(x), \min f'(x) \}$. As the final solution we have the Equations $\ref{eq_var_fix_e1}$ and $\ref{eq_var_fix_e2}$ where $\beta$ is the hyper parameter defining the controlled drop coefficient which we use to ensure the creation of higher level feature detectors in deeper layers.

$$ \begin{align} Z = \frac{1}{Var\{ \max f'(x), \min f'(x) \}} \label{eq_var_fix_e1} \end{align} $$$$ \begin{align} W \sim \mathcal{N} \bigg( 0, \frac{Z}{n} \beta \bigg) \label{eq_var_fix_e2} \end{align} $$

2.1 Activaton Function

Distribution of Weights

[tanh, σ² = 4.0]

P(f′(x) < 0.001) = 3.82186%

Distribution of Weights

[arctan, γ = 1.4, σ² = 7.84]

P(f′(x) < 0.001) = 0.00000%

Distribution of Weights

[arctan, γ = 1.8, σ² = 12.96]

P(f′(x) < 0.001) = 0.00000%

Figure 3: Probability distribution of weights initialized using our method. Shaded area shows gradients that cause network to learn “slowly” ($f'(w) < 0.001$).

One problem that remains is the activator function. As universal as the solution in Equation $\ref{eq_var_fix_e2}$ sounds, it cannot used with non zero-meaned or strict activators like sigmoid or tanh, nor with functions that do not follow the rules of Universal Approximation Theorem[8].

We will declare weights where $f'(w) < 0.001$ “slow”, and visualize the problem. As it can be seen in Figure 3, when we use tanh as the activator, $\approx 4\%$ of the network is learning “slowly” when there is only one layer. This effect increases exponentially as the number of layers increase.

Instead of an exponentially increasing function like sigmoid or tanh, we are going to use arctan which increases quadratically and squash it using a hyper-parameter, $\gamma$ as seen in Equation $\ref{eq_arctan}$.

$$ \begin{align} f(x) = \frac{1}{\gamma} \arctan(x) \label{eq_arctan} \end{align} $$

tanh(x)arctan(x)/1.495

Figure 4: Comparison of arctan and tanh.

As it can be seen from the 2nd and 3rd graphs of Figure 3, due to its quadratically increasing nature, the proposed activator did not experience the issue tanh activator did even though the variation was much higher with arctan.

0x3: Experimental Setup and Datasets

We are going to be using MNIST (Figure 5), STL-10 (Figure 6) and a custom dataset (Figure 7) when benchmarking our technique and compare it with other techniques. STL-10 is a particularly interesting challenge as it only contains 500 images per class.

Figure 5: Examples from MNIST[9] dataset.

Figure 6: Examples from STL-10[10] dataset.

Figure 7: Examples from our custom dataset.

As all of these tasks are classification tasks, cross entropy will be used and every network will be regularized using L2 regularization with regularization coefficient $\lambda = 10^{-4}$. All benchmarks will be done using a custom CUDA library and GTX 1080.

0x4: Results

Leaky ReLU + Xavier InitializationArctan + Proposed Initialization

Var(δ) per Layer

E(|δ|) per Layer

Figure 8: Comparison of the distribution gradients have per layer after the first iteration has completed.

Before starting other tests we decided to search for the most optimal value for $\gamma$ from the set $\big[1.1, 1.2, \dots, 1.9\big]$. For each value of $\gamma$, we trained 25 networks for 20000 iterations on MNIST dataset and averaged the cross-entropy loss.

Loss after 20000 iterations with given γ

Figure 9: Results of hyper-parameter search for $\gamma$

Although we originally thought $\gamma=\pi/2$ would be the best fit, constraining the output to [-1, +1] precisely, this was not the case as seen from Figure 9. Going from $\gamma=1.1$ to $\gamma=1.9$, error almost halved due to higher values of $\gamma$ allowing higher variance. As a result, we will be using $\gamma=1.7$ for our tests.

4.1 Analysis of the Gradients

For the first test, we checked whether the technique we proposed circumvents the problem of unstable gradients as expected or not. For comparison, we are going to use the network from Figure 2.

As it can be seen from Figure 8, neither variance nor the mean of gradients shows any correlation with the depth of the layer.

The see-saw behaviour of both graphs are due to the fact that the Z value we used for the variance of the weight tensor bumps up the variance in a way that when the activation function squishes the input (thus lowering the variance), the variance returns to the exact same previous level which means our technique works as we expected; preventing the gradients from vanishing.

We tested whether this assumption was correct or not by training a network on XOR dataset. In our tests, we were not able to train a 30-layer feed forward network for XOR dataset with traditional methods within 100k iterations whereas our technique reached $\approx 5\%$ error by 2000 iterations.

4.2 Effect of β hyper-parameter

β = 1.0β = 0.7β = 0.5

Effect of β on Learning Speed

Figure 10: Error history of networks trained on STL-10 dataset using $\beta$ values 0.5, 0.7, 1.0.

In order to see the effect of $\beta$ on training accuracy, we trained 3 networks with $\beta$ values set to 0.5, 0.7 and 1.0 on STL-10 dataset. As it can be seen from Figure 10, although the difference between 0.5 and 0.7 is hard to interpret, it is rather apparent that values other than 1.0 had better validation and training accuracy. This leads us to believe that $\beta$ value should be included in hyper-parameter search when setting up the network as it depends on the network’s architecture and complexity of the data used.

4.3 Feature Complexity and Visualization of Learned Features

Figure 11: Visualization of the weights after and before training. Image on the left shows how the weights were initialized in the first place whereas image on the right shows the weights after training phase was complete.

Gradient-equalization affects the development of features in a rather interesting way. When the gradient drops as layers go like in a traditional network, the filters get more complex as the layer gets deeper. On the other hand, when our method is used, complexity is distributed across layers; achieving a highly-complex network rather than highly-complex deep layers.

Figure 11 visualizes first layer of a network trained on STL-10 dataset before and after training phase. As it can be seen, although some features were certainly learned, it can not be easily interpreted unlike traditional networks except for a few line patterns.

This approach sacrifices the possibility of transfer learning and ease of visualizing for speed and overall complexity. It also makes it easier for these networks to over-fit training dataset but this is not as big of a problem as we thought originally since our network were able to achieve similar results to 2012 (supervised) state of the art[11] within 8 epochs on STL-10 dataset which contains only 500 images per class.

4.4 MNIST Dataset

Arctan + Proposed Initialization, Training AccuracyArctan + Proposed Initialization, Validation AccuracyLeaky ReLU + Xavier Initialization, Training AccuracyLeaky ReLU + Xavier Initialization, Validation Accuracy

Accuracy per Epoch

Figure 12: Error history of networks trained on MNIST dataset using different techniques.

For the first experiment we conducted, we used MNIST as it was the easiest dataset of all 3. As it can be seen from Figure 12, it took 25 epochs for the network using traditional techniques to reach the accuracy network using our technique reached in 3 epochs. Our network ended up reaching 98.44% validation accuracy by the end of 23 epochs.

4.5 STL-10 Dataset

ConvNet Validation AccuracyStochasticNet Validation AccuracyOur Network Validation Accuracy

Accuracy per Epoch

Figure 13: Error history of networks trained on STL-10 dataset using different techniques.

This dataset is especially interesting because it has less data when compared to CIFAR-10[12] and yet 9 times the input volume. We compared the performance of our network to the results of StochasticNet[13] published in 2015. It can be seen from Figure 13 that our network reached $\approx$40% accuracy within a single epoch whereas other networks took 5 epochs and our network reached $\approx$56% accuracy within 9 epochs whereas it took StochasticNet 18 epochs, and ConvNet could not reach it within 20 epochs. Our network ended up reaching 61.68% validation accuracy by the end of 20 epochs.

4.6 Our Dataset

Accuracy per Epoch

Figure 14: Error history of networks trained on our dataset using different techniques.

We used a custom dataset in order to compare the performance for fast learning when the input data is large (96x96x3) but the pattern is easy to recognize. Our method performed significantly better reaching $\approx 4\%$ validation error within 3 epochs with only 2000 actual examples in training set and 1000 examples in validation set whereas the traditional methods could only reach $\approx 49\%$ error rate within this very short time as it can be seen from Figure 14. Our network ended up reaching 99.76% validation accuracy by the end of 20 epochs.

4.7 Performance

Although ReLUs are known for being especially fast as they do not require floating-point computations like arctan, sigmoid or tanh, we noticed in our tests that our model actually ran 5% faster compared to ReLUs. We assume this is because of the fact that our activator does not require any branching unlike ReLUs as GPUs are rather bad when it comes to branches due to the fact that they require both paths to be executed by all the threads when there is branch divergence within a warp.

4.8 Conclusion

Results from the training phases of these datasets certainly provided us concrete proof that our technique lets deep neural networks reach high training and verification accuracies much faster compared to the traditional techniques. For the networks where our technique was used, the speed of learning was affected by the complexity of dataset rather than the depth of the network as all layers of the network trained at similar speeds.

Our technique also had the benefit of human-like learning where network learned the basics of the dataset (this basic knowledge covered $\approx$85% of the dataset for MNIST and our dataset and $\approx$40% of the dataset for STL-10) very fast (within a single epoch) and got better at it eventually, letting us have the best of both worlds — accuracy and speed.

0x5: Future Work

Leaky ReLU + Xavier InitializationArctan + Proposed InitializationExpected Drop

E(|δ|) per Layer

Figure 15: Demonstration of controlled gradient drop.

One question that still remains is whether the controlled gradient drop technique we used were satisfactory or not. After calculating the $\beta$ hyper-parameter which would drop the gradient to $1/10$th of the original in the first layer, we saw that although the ratio between first and final layer were certainly $1/10$, the hidden layers did not necessarily follow the exponential drop we wanted perfectly, as it can be seen from Figure 15.

References

Q.-H. Ye, L.-X. Qin, M. Forgues, et al., “Predicting hepatitis b virus–positive metastatic hepatocellular carcinomas using gene expression profiling and supervised machine learning,” Nature medicine, vol. 9, no. 4, pp. 416–423, 2003.
J. Tompson, K. Schlachter, P. Sprechmann, and K. Perlin, “Accelerating eulerian fluid simulation with convolutional networks,” arXiv preprint arXiv:1607.03597, 2016.
S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” ACM Transactions on Graphics (TOG), vol. 35, no. 4, p. 110, 2016.
V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks.,” in Aistats, vol. 9, 2010, pp. 249–256.
K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in The IEEE International Conference on Computer Vision (ICCV), Dec. 2015.
C. Szegedy, W. Liu, Y. Jia, et al., “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.
A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Transactions on Information theory, vol. 39, no. 3, pp. 930–945, 1993.
C. J. Burges, Y. LeCun, and C. Cortes, “Mnist database,” [Online]. Available: http://yann.lecun.com/exdb/mnist/.
A. Coates, H. Lee, and A. Y. Ng, “An analysis of single-layer networks in unsupervised feature learning,” Ann Arbor, vol. 1001, no. 48109, p. 2, 2010.
Y. Jia, C. Huang, and T. Darrell, “Beyond spatial pyramids: Receptive field learning for pooled image features,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, IEEE, 2012, pp. 3370–3377.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009.
M. J. Shafiee, P. Siva, and A. Wong, “Stochasticnet: Forming deep neural networks via stochastic connectivity,” arXiv preprint arXiv:1508.05463, 2015.

Appendices

A.1 Network Used for STL-10 and Our Dataset

 1Convolution [96, 96, 3] → [94, 94, 32] (3x3)
 2Activator Arctan * 1/1.7
 3Convolution [94, 94, 32] → [92, 92, 32] (3x3)
 4Activator Arctan * 1/1.7
 5Pool [92, 92, 32] → [46, 46, 32] (2x2)
 6Convolution [46, 46, 32] → [44, 44, 64] (3x3)
 7Activator Arctan * 1/1.7
 8Convolution [44, 44, 64] → [42, 42, 64] (3x3)
 9Activator Arctan * 1/1.7
10Pool [42, 42, 64] → [21, 21, 64] (2x2)
11Convolution [21, 21, 64] → [19, 19, 128] (3x3)
12Activator Arctan * 1/1.7
13Convolution [19, 19, 128] → [17, 17, 128] (3x3)
14Activator Arctan * 1/1.7
15Pool [17, 17, 128] → [8, 8, 128] (3x3)
16Convolution [8, 8, 128] → [6, 6, 256] (3x3)
17Activator Arctan * 1/1.7
18Convolution [6, 6, 256] → [4, 4, 256] (3x3)
19Activator Arctan * 1/1.7
20Pool [4, 4, 256] → [2, 2, 256] (2x2)
21Fully Connected [2, 2, 256] → [1024, 1, 1]
22Activator Arctan * 1/1.7
23Dropout p=0.5
24Fully Connected [1024, 1, 1] → [1024, 1, 1]
25Activator Arctan * 1/1.7
26Fully Connected [1024, 1, 1] → [1024, 1, 1]
27Activator Arctan * 1/1.7
28Dropout p=0.5
29Fully Connected [1024, 1, 1] → [10/6, 1, 1]
30Softmax

Beta, momentum, learning speed and batch-size were grid-searched. Learning speed were multiplied with 0.2 once the network’s error started to plateau.

A.2 Network Used for MNIST

 1Convolution [28, 28, 3] → [26, 26, 64] (3x3)
 2Activator Arctan * 1/1.7
 3Convolution [26, 26, 64] → [24, 24, 64] (3x3)
 4Activator Arctan * 1/1.7
 5Convolution [24, 24, 64] → [22, 22, 64] (3x3)
 6Activator Arctan * 1/1.7
 7Convolution [22, 22, 64] → [20, 20, 64] (3x3)
 8Activator Arctan * 1/1.7
 9Pool [20, 20, 64] → [10, 10, 64] (2x2)
10Convolution [10, 10, 64] → [8, 8, 128] (3x3)
11Activator Arctan * 1/1.7
12Convolution [8, 8, 128] → [6, 6, 128] (3x3)
13Activator Arctan * 1/1.7
14Convolution [6, 6, 128] → [4, 4, 128] (3x3)
15Activator Arctan * 1/1.7
16Convolution [4, 4, 128] → [2, 2, 128] (3x3)
17Activator Arctan * 1/1.7
18Fully Connected [2, 2, 128] → [512, 1, 1]
19Activator Arctan * 1/1.7
20Dropout p=0.5
21Fully Connected [512, 1, 1] → [512, 1, 1]
22Activator Arctan * 1/1.7
23Dropout p=0.5
24Fully Connected [512, 1, 1] → [10, 1, 1]
25Softmax

Beta, momentum, learning speed and batch-size were grid-searched. Learning speed were multiplied with 0.2 once the network’s error started to plateau.