Structuring an AI agent that generates pipeline

Most “AI agent” demos are a single prompt in a while loop. That falls over the moment the work is real: dozens of tool calls, partial failures, data that has to be fetched, deduplicated, and enriched before a model ever sees it. The agents we run to generate pipeline look less like a chatbot and more like a small, well-typed distributed system with a language model in the decision seat.

Here’s the shape we keep coming back to.

Keep the loop boring

The control loop should be the least clever part of the system. It picks the next step, runs it, appends the result, and stops when the model says it’s done or a budget is exhausted. Everything interesting lives in the tools and the guardrails, not here.

import type { Message, ToolResult } from "./types";
import { model } from "./model";
import { tools } from "./tools";

export async function runAgent(goal: string, budget = 20): Promise<Message[]> {
  const messages: Message[] = [{ role: "user", content: goal }];

  for (let step = 0; step < budget; step++) {
    const reply = await model.next(messages, tools.schema);
    messages.push(reply);

    if (!reply.toolCalls?.length) break; // model is finished

    const results: ToolResult[] = await Promise.all(
      reply.toolCalls.map((call) => tools.run(call)),
    );
    messages.push(...results.map(toToolMessage));
  }

  return messages;
}

The budget matters more than it looks. An unbounded agent that “keeps trying” is an agent that quietly burns money and rate limits. Every loop we run has a hard ceiling on steps, tokens, and wall-clock time.

Make tools typed, and validate at the boundary

Tool calls are the one place a model reaches into the real world, so it’s the one place we’re strict. Each tool declares a schema, and we validate arguments before executing — a hallucinated field should fail loudly, not silently corrupt a downstream record.

import { z } from "zod";

export const searchAccounts = {
  name: "search_accounts",
  description: "Find companies matching an ICP description.",
  schema: z.object({
    industry: z.string(),
    employeeRange: z.tuple([z.number(), z.number()]),
    limit: z.number().int().max(50).default(25),
  }),
  async run(args: unknown) {
    const { industry, employeeRange, limit } = searchAccounts.schema.parse(args);
    return db.accounts.search({ industry, employeeRange, limit });
  },
};

Two things earn their keep here: .parse() turns a bad tool call into a caught error instead of a 3am incident, and .max(50) on limit stops the model from politely requesting the entire database.

Grade the outcome, not the transcript

The failure mode of agent evals is grading how nice the reasoning looked. We don’t care whether the transcript reads well — we care whether the pipeline it produced is real. So our evals run the agent end to end and then check the artifact against ground truth.

# Run the eval suite against last week's frozen fixtures
bun run eval --suite pipeline --fixtures 2026-w26

#  ✓ contactable rate      0.94   (target ≥ 0.90)
#  ✓ dedupe precision      0.99   (target ≥ 0.98)
#  ✗ ICP match (manual)    0.86   (target ≥ 0.90)  ← regressed

When a number regresses, the fix is almost never “prompt it harder.” It’s a missing validation, a stale enrichment source, or a tool returning something the model misread. The loop stays boring; the system gets better.

The takeaway

Treat the model as one component in a typed system, not the system itself. Put the cleverness in the tools and the guardrails, bound every loop, and measure the outcome you actually sell. That’s most of what separates a demo from something you can put in front of a Fortune 500.