Optimizing LLM I/O: TOON vs JSON, schema validation trade-offs, and engineering patterns to cut latency

Problem: large LLM prompts and structured-output validation are common latency sources. This article presents concrete engineering patterns, benchmarks, and production-ready code for choosing and implementing TOON (Token-Oriented Object Notation) vs JSON, layering schema validation (Zod/JSON Schema), and minimizing wall-clock latency in high-throughput systems.

Key outcomes summarized:

• Removing explanatory "reasoning" fields reduced output tokens and cut latency from 11s to 7s in a real system (incident.io).
• Moving from JSON to a compact TOON format compressed input tokens from ~15k to ~2k and dropped end-to-end latency from 7s to 5.7s.
• Schema validation at provider-level is low-cost on first successful responses; an added Zod-layer provides type safety and safe retries.
• Heavy JSON-schema enforcement can introduce CPU/IO overhead; certain implementations report 3.1x slower agent loops when schema validation is synchronous and blocking.

Architecture patterns and trade-offs

• JSON (strict-parse)
• Strengths: deterministic parsing, native language parsers, safe for nested/arrayed payloads.
• Weaknesses: punctuation and structural tokens add to token count; heavy schemas can add CPU latency at validation time.
• TOON / token-optimized flat formats
• Strengths: fewer tokens for shallow data, lower output token count reduces model latency and API cost.
• Weaknesses: brittle for nested data, requires custom parser and defensive validation, error-prone if model drifts.

Mermaid sequence: token flow and validation layers

Concrete benchmarks (end-to-end wall-clock and token counts)

• Testbed: mid-tier LLM (chat model), single synchronous request path, same network conditions across runs.
• Metrics below are representative; measure on your topology.

When to choose JSON vs TOON (short checklist)

• Use JSON when:
• Data is nested or contains arrays, or you require deterministic parsing (e.g., cross-service RPC).
• Multiple consumers/languages need standard parsers.
• Safety/regulatory requirements demand strict schema checks.
• Use TOON (compact) when:
• Payloads are shallow key→value lists or rows.
• You control both the system prompt and the downstream parser.
• Token cost and latency are primary operational constraints.
• Use hybrid:
• Use TOON for the latency-sensitive LLM output and emit a structured JSON backup or audit log from the application after successful parsing.
• Or use JSON for structure but teach the model to emit compact key lists (strip whitespace/punctuation).

Production-safe implementation: validation and retry wrapper

Design goals:

• Keep the common-case fast: most LLM responses validate on first attempt -> no extra latency.
• If validation fails, append a short correction prompt (not full re-run) and retry.
• Keep parsing and validation async and CPU-efficient; avoid synchronous blocking in tight agent loops.
• Balance provider-level schema constraints with client-side Zod validation for cross-provider safety.

TypeScript implementation (production-ready, fully typed)

• The wrapper accepts a generic response type T and a Zod schema.
• First attempt validates without extra prompt. On failure, a correction prompt is appended and a single retry is attempted.
• This code uses an abstract LLM client interface so you can plug in your provider.

// llm-validation.ts
import { z, ZodSchema } from "zod";

export type LLMRequest = {
  model: string;
  prompt: string;
  maxTokens?: number;
  temperature?: number;
  // additional provider-agnostic fields
};

export type LLMResponse = {
  text: string;
  usage?: { promptTokens?: number; completionTokens?: number; totalTokens?: number };
  // extend with provider fields as needed
};

export interface ILLMClient {
  call(request: LLMRequest): Promise<LLMResponse>;
}

/**
 * validateAndParse
 * - client: provider implementation
 * - schema: Zod schema for response payload
 * - parseFn: function to extract the structured payload from raw text (TOON or JSON)
 */
export async function validateAndParse<T>(
  client: ILLMClient,
  request: LLMRequest,
  schema: ZodSchema<T>,
  parseFn: (raw: string) => unknown,
  correctionPrompt?: (raw: string) => string
): Promise<T> {
  const first = await client.call(request);
  const rawFirst = first.text;
  const parsedFirst = safeParse(parseFn, rawFirst);

  const firstCheck = schema.safeParse(parsedFirst);
  if (firstCheck.success) return firstCheck.data;

  // Correction path: append minimal correction hint and retry once.
  if (!correctionPrompt) throw new Error("Validation failed and no correctionPrompt provided.");

  const correctedRequest: LLMRequest = {
    ...request,
    prompt: request.prompt + "\n\n" + correctionPrompt(rawFirst),
  };

  const second = await client.call(correctedRequest);
  const rawSecond = second.text;
  const parsedSecond = safeParse(parseFn, rawSecond);
  const secondCheck = schema.safeParse(parsedSecond);
  if (secondCheck.success) return secondCheck.data;

  // Return the richer error for observability and safe fallback upstream.
  throw new ValidationError({
    attempts: [
      { raw: rawFirst, parsed: parsedFirst, result: firstCheck },
      { raw: rawSecond, parsed: parsedSecond, result: secondCheck },
    ],
  });
}

function safeParse(fn: (raw: string) => unknown, raw: string): unknown {
  try {
    return fn(raw);
  } catch {
    return undefined;
  }
}

export class ValidationError extends Error {
  public attempts: unknown[];
  constructor(attempts: unknown[]) {
    super("LLM output failed validation after retry.");
    this.attempts = attempts;
    Object.setPrototypeOf(this, ValidationError.prototype);
  }
}

Example parse functions (TOON and JSON)

// parsers.ts
export function parseToon(raw: string): Record<string, string> {
  // Example TOON format:
  // "id:123 | name:Widget | score:0.82"
  const result: Record<string, string> = {};
  const parts = raw.split("|").map(p => p.trim());
  for (const part of parts) {
    const [k, ...rest] = part.split(":");
    if (!k || rest.length === 0) continue;
    result[k.trim()] = rest.join(":").trim();
  }
  return result;
}

export function parseJson(raw: string): unknown {
  // Attempt to extract a JSON substring if the response contains extraneous text.
  const first = raw.indexOf("{");
  const last = raw.lastIndexOf("}");
  if (first === -1 || last === -1 || last <= first) throw new Error("No JSON object detected");
  const candidate = raw.slice(first, last + 1);
  return JSON.parse(candidate);
}

Zod schema example

// schemas.ts
import { z } from "zod";

export const DashboardSchema = z.object({
  id: z.string(),
  name: z.string(),
  score: z.number().optional(),
});

export type Dashboard = z.infer<typeof DashboardSchema>;

Observability and defensive patterns

• Log model raw responses and validation result with sample truncation to support drift detection.
• Maintain counters: validation_success_first_try, validation_failed_then_success, validation_failed_final.
• Rate-limit retries to avoid amplifying costs or creating retry storms.
• For TOON, keep a small parser library with comprehensive unit tests and fuzzing.

Reproducible implementation steps (ordered)

1. 1Audit your responses: capture a representative sample of LLM outputs including tokens, latencies, and downstream parsing failures.
2. 2Decide format per use-case:

• If nested arrays/objects → JSON + provider schema + Zod.
• If flat rows/key→value and token-budget constrained → TOON + Zod + parser.

1. 1Implement a thin validation wrapper (example above) that:

• Parses raw output (TOON/JSON).
• Validates with Zod (or equivalent).
• Retries once with a correction prompt appended only when validation fails.

1. 1Add telemetry:

• capture token counts (prompt/completion), latency, CPU validation time, retry rate.
• create alert thresholds for validation failure rate > X% or retries > Y/min.

1. 1Load-test: emulate expected QPS and measure end-to-end latency, CPU, and token costs across both formats.
2. 2Deploy staged: route a small percentage of production traffic through the new format and monitor.
3. 3Iterate on prompt docstrings and correction prompts based on failure modes.

Common failure modes and mitigations

• Model emits partial JSON: Use extraction heuristics that find the largest JSON substring. Log and retry with minimal correction prompt.
• TOON field missing or misspelled: Add canonical field header and checksum or use short field IDs (e.g., "i:123|n:Widget|s:0.82").
• Provider-level schema enforcement rejects valid but differently structured answers: favor provider JSON-mode for strict cases, but keep Zod for cross-provider resilience.

Operational cost model (rules of thumb)

• Output tokens dominate latency and cost when model is chat-style; reducing output tokens yields disproportionate latency wins.
• Provider schema-constrained mode removes many malformed responses at the source; combined with Zod, retry frequency drops to near-zero in mature systems.
• Synchronous heavy schema validation inside tight agent loops can multiply CPU latency (observed up to 3.1x in some agent-heavy systems); run validation asynchronously or off the critical path where possible.

Final notes

• The right format is a pragmatic trade-off between token efficiency and parsing robustness. Use JSON where structure matters; use TOON when token budget and latency matter, and you can own the parsing and observability.
• Combine provider-level schema constraints (when available) with a client-side Zod boundary and a single correction retry. This keeps the fast-path minimal and provides safety nets for edge cases.