The Bicycle That Taught Me My Agent Loop Had No Ground Truth

In June 2026, I wrote a design document for what I called an Autonomous Semantic Vector Graphics Engine: a locally hosted multimodal agent where an LLM acts as an architect, decomposes a prompt into named components, draws them in a see-think-act loop, and hands you back clean, semantic SVG.

It was, in the reviewer's words, "a well-organised brainstorm, not a buildable spec."

Four months later, there is a working engine. It is not the thing in the document. It is a constraint-solving vector compiler with a text front door I now use as a smoke test, and a sketch front door I trust. This is the story of how it turned around - including the parts where I was wrong and the parts I still haven't finished.

The Pitch

The document opened with four problems, and they were real:

Problem 1 - Context Window Bloat. Sending a base64 image to the LLM after every single line drawn rapidly consumes thousands of visual tokens, causing out-of-memory errors or 'forgetting' the original prompt.

Problem 2 - Mathematical & Spatial Hallucinations. LLMs struggle with absolute coordinate math (e.g., guessing the exact X/Y intersection of two angled lines) and complex SVG path commands (Bezier curves).

Problem 3 - The 'Spaghetti' Vectorization Problem. When tracing raster images, traditional algorithms output thousands of overlapping, unnamed <path> nodes.

And the answer, in my own words:

The system uses a Closed-Loop Spatial Reasoning Engine. An LLM acts as an architect, using semantic abstractions (named components, scene graphs) to draw logical, mathematically precise, and reusable vector assets.

Three design decisions came with it: decoupled dual-buffer rendering through Processing, deep cloning over <defs>/<use>, and a text-heavy feedback loop - "the agent relies on SVG XML snippets and Scene Graph summaries for 80% of its micro-corrections."

Then there was the tracing mode, which I was quietly proud of. I'd call it agentic onion skinning: render the target logo onto the canvas at 50% opacity, have the VLM trace over it, and critique with a two-image prompt. Compared to potrace, it sounded obviously better.

Six reviewers, one sentence that ended it.

I sent the design to six models for review. Two of the verdicts:

Claude: "a well-organised brainstorm, not a buildable spec. A dev team picking this up today would hit blockers immediately."

Kimi: "DraftsLLMan is a promising conceptual prototype but reads like a research fantasy rather than an engineering blueprint. Its core error is conflating semantic understanding (where LLMs excel) with geometric execution (where they fail)."

And one from Minimax that I did not understand for about a week:

You describe a 'Closed-Loop Spatial Reasoning Engine' but never define what the agent is converging toward. For text→SVG, the agent's 'see' step is just self-confirming its own mental model - there is no reference signal to correct against. The loop is closed, but it's closed on a moving target.

For text-to-SVG this is a fundamental flaw. The only meaningful use of a visual critique loop is image-to-SVG (where you have a reference image).

Now, I can't unsee it. My loop had a renderer but not a reference. The agent was grading its own homework with a rubric it invented on the fly, and every "correction" was applied to geometry nobody had measured.

The onion-skinning proposal died the same week, and it was the sharpest review of the set:

Agentic Onion Skinning restates the hard problem without solving it. … Step (c) directly re-encounters Problem 2: the LLM must generate precise X/Y coordinates for polygon vertices, which is exactly the capability that LLMs lack. The document offers no mechanism to bridge this gap.

I had used the LLM to trace because tracing was hard, not because the LLM was good at it.

What survived the reviews was a division of labor

v2.0 of the spec contains one sentence that reorganised the entire project:

Key Architectural Pivot Required: Reframe from "LLM draws vectors" → "LLM plans semantically, deterministic engine executes geometrically."

That is the Zero-Coordinate kernel, and it is the part that's still alive:

• The model declares components, sizes and relational intents - align, distance_ratio, connect, contain, relative_size, distribute.
• A Cassowary constraint solver computes every position. If the constraints contradict each other, it returns Unsatisfiable instead of quietly drawing nonsense.
• Skia replaces Processing as the single source of truth, which also deleted an entire class of bug: two renderers that disagree about stroke width and rounding means the agent corrects against a ghost image.
• Gemini pointed out a contradiction I'd missed - reserving images for macro-milestones means the agent literally cannot see enough pixels to correct. That became foveated rendering: a 256px global map plus a 512–768px region-of-interest crop.

The honest slogan, which I got wrong in my own README and fixed in October, is this: the model never places anything. It declares sizes, and it proposes structure.

Then the hypothesis itself broke

I built the kernel expecting it to fix things. On the 1 of August, 2026, I ran a live bicycle and wrote down what happened, in the spec, under the heading Real finding:

The merged fixes ended the empty-SVG, but the output is still unrecognizable as a bicycle, and the cause is structural, not coordinate-related:

1. Components cannot connect. … the Coordinator's merge merely offsets each fragment by its region. No agent owns two components, so nothing links the frame to the wheels, the seat to the frame. A bicycle is recognizable chiefly by its topology (two wheels joined by a frame); the offset-merge architecture structurally cannot produce that topology.

The Zero-Coordinate Policy (the engine owns all geometry) remains the means; inter-component connection is the actual end.

That is the sentence that reorganised me. I had spent three weeks hardening the wrong thing. The model wasn't failing at arithmetic; my architecture had no way to express "the frame's rear axle is the wheel's centre," so no amount of constraint hygiene could save it. Slices 3–5 were reclassified as load-bearing rather than optional.

The policy fights that followed.

The kernel then spent a month arguing with me, and losing.

• Runtime snapping had to go. Every tool call was being run through an interceptor that rewrote all JSON floats to one decimal. A live run declared size_ratio: 0.05 for spokes; the snap turned it into 0.1, so every spoke rendered twice its intended size. The root cause was historical: snapping existed to fight jitter in the old pixel-valued arguments, and it outlived the vocabulary it was built for. Now there is exactly one rounding in the system - two decimals, at SVG serialization.
• The Draftsman prompt was still leaking coordinates. It told the Phase-3 agent "already solved at global position top-left=(x, y) size w×h", and live traces showed the model echoing the numbers back in its reasoning. That was the last standing violation of the policy, fixed on the 19 of August, 2026.
• The safest thing I wrote is a wrapper, not a feature. Every LLM→engine call goes through the same five steps, and rollback is a snapshot restore - not an inverse function. grep -rni undo across the Kotlin source returns nothing, because restoring an immutable value you already hold isn't an operation that can be wrong.

The Reshape

On the 24 of August, 2026, I wrote a plan whose first line is a status marker about the previous six weeks:

Status: adopted 2026-08-24. Supersedes the "autonomous vector art" framing for product direction. The zero-coordinate kernel stays - the product around it changes.

DraftsLLMan is a semantic constraint-solving vector engine for icon and technical-asset production, not an autonomous picture generator.

Under a heading titled Drop completely, the first item was the original thesis:

The "autonomous pretty-picture generator" ambition - one-shot elaborate art (bicycles, scenes) as the goal. The bicycle remains only as a kernel smoke test.

And the replacement product, in one line: drop a raster logo in → get an editable, constraint-correct SVG component → instantiate variants → export a set.

Local Ollama got demoted in the same document, for a reason that reads like a verdict on a whole class of tooling:

Local qwen3-vl:8b burned 5.5K-token reasoning chains and only emitted a skeleton on the capped 4th nudge - that class of failure is a tax, not a feature.

That was the first time I stopped treating "it runs on my laptop" as a feature and started treating it as a constraint with a price tag.

Why sketch-in is the hospitable shape.

The turn wasn't ideological. It's that a sketch fixes exactly the two things the text path was missing.

It supplies ground truth. This is Minimax's point, implemented. With a text prompt, the loop has nothing to compare against; with a raster in front of it, "wrong" becomes an observable fact, and the same foveated critique machinery that was useless in text mode becomes the whole point. The eval harness is built on it: WireframeDSL generates a wireframe and co-derives its ground truth, so the scoring is seeded and repeatable instead of a human squinting at a PNG.

Its strokes are already near primitives. A hand-drawn sketch is a much better input for a primitive-fitting layer than a finished logo is. The spec's three-stage sketch pipeline is exactly this:

Stage 1 - Stroke Extraction. Raw contours, wobbly, 1px wide. Stage 2 - Geometric Primitive Fitting (the "math" layer). Line fitting, circle/arc fitting, corner detection. "The Scene Graph is now populated with mathematically perfect <circle>, <rect>, and <line> nodes, completely stripping away the hand-drawn jitter." Stage 3 - Semantic Grouping (the "LLM" layer). "The LLM is not doing any math or tracing. It is acting purely as an art director."

That is the onion-skinning idea with the LLM removed from the part it was bad at, and a human sketch removed from the part it was bad at too. The deterministic tracer owns the pixels; the model owns the naming; the solver owns the geometry.

Both front doors exist in the code today (sketch/SketchRunner.kt for the vision path, the Architect/Draftsmen pipeline for text). The honest difference is not that one works better - it's that only one of them can be graded.

How do you know it's any good?

"How did it look?" is not a metric, and I refuse to report vibes. The harness generates a seeded wireframe, rasterizes it into something hand-drawn-looking (Sketchify procedurally, or a local ComfyUI pipeline), hands it to a vision agent, and scores the recovered scene graph.

The scoring is semantic on purpose. A component counts as recovered when its role and kind agree - bounding boxes are a tie-breaker, never a gate, because a layout recovered at shifted positions is a success, not a miss. A separate relation-consistency check verifies that left-of and above hold in both truth and prediction, which is what makes "sidebar on the wrong side" a real failure instead of a coordinate wobble.

The gate: F1sem ≥ 0.80 AND relCons ≥ 0.70. Above it, commit to vision-first. Below it, the text-first hybrid wins.

And here is the part I have to be honest about: there are no recorded results. The constants exist, the test prints its scores, and it asserts nothing. The manifests in output/ record a different gate entirely - sketch integrity - and contain no F1 or relation-consistency numbers anywhere. The M1 decision has not been made. Everything I've written about the vision path is architecture, not evidence.

Where it actually stands

Works, and is verified by 454 offline tests (no network, headless, ~6,400 lines of main Kotlin against ~8,300 lines of test):

• The solve pipeline: emit skeleton → solve → verify → commit, or roll back. Unsatisfiable constraints fail loudly; degenerate solves get a deterministic Safe-Harbor spread rather than a blob.
• The zero-coordinate kernel, including the coordinate-free handoff between agents.
• Foveated visual critique through headless Skia.
• Three interchangeable providers.
• A component library with prototype/override semantics - built, tested, wired to the SVG exporter, and not reachable from any tool call. I can render an instance; I can't ask for one.

Doesn't work well:

• Reliability. Of the 26 SVGs in my output/ directory, 5 are empty <svg> shells. One run in five produces nothing. This is the single biggest gap between this and something I'd call done.
• The decision gate is a hypothesis.
• The library is unreachable from the pipeline.

And the bugs worth remembering were never geometric. Cassowary is the boring, reliable part. The four that cost me real time:

• The 4096 wall. Local models died mid-reasoning, every time, at exactly totalTokensCount=4096 - Ollama's default context silently truncating once my ~2.7K-token prompt loaded. The fix was pinning num_ctx explicitly and refusing Ollama's FitPrompt auto-sizing, because its chunk rounding would faithfully reproduce the same wall at a larger prompt.
• Reasoning-only loops. Thinking models sometimes answer with pure chain-of-thought and no routable action. The retry-with-a-nudge has to live inside the LLM-call node rather than as a new node, because Koog's maxAgentIterations counts node executions, not LLM calls. Add a node to inject behaviour, and you silently eat your budget.
• The finish-edge race. A model emitted "done!" and a pending tool call in one turn; Koog finished on the text and silently dropped the drawing.
• …and the fix for that third one, which is the only lesson here I'd call generalizable. My first fix applied the guard at each call site, and two nodeSendToolResult sites were missed. Writing this article is what made me grep for it. The real fix was to move the guard into requestLlmToleratingReasoning - the single choke point every agent-loop LLM call passes through - so a call site cannot forget it. A guard applied N times is N opportunities to forget; a guard applied once, in the right place, is a property of the system.

What I'd tell my past self

Write the closing paragraph of the pitch last. I described a system that draws pictures from words, and by the end I had built a system that solves constraints and accepts a drawing as input. Both are useful. Only one of them was ever going to be gradeable - and I could have known that in June, if I'd asked what the loop converges toward, instead of what it draws.

Try it

git clone https://github.com/ciszkin/DraftsLLMan && cd DraftsLLMan
cp local.properties.example local.properties   # one provider config
./gradlew test                                  # 454 tests, offline, headless
./gradlew run --args='"a circle centered above a rectangle"'

JDK 21 is auto-provisioned by Gradle's Foojay resolver. Kotlin 2.3.21, Koog 1.1.1, kasuari 0.1.6 (Cassowary), Skia via Skiko for headless raster. MIT licensed.

The goal is still "skeleton lands on the first attempt, no nudge forensics." Then run the gate and publish the numbers either way.