Streaming Tool Input

OverviewCopy Anchor Link

A model that writes a long tool call, such as a file of a few hundred lines, can take many seconds before the call is complete. In a streamed run the call shows up as it is written: its id and name as soon as the model picked the tool, then each piece of its JSON arguments.

let written = '';

await cog.run(agent, {
  input: 'Write a README for this project',
  stream: true,
  onToolInputStart: ({ toolCallId, toolName }) => ui.open(toolCallId, toolName),
  onToolInputDelta: ({ toolCallId, delta }) => {
    written += delta;
    ui.preview(toolCallId, written);
  },
  onToolInputAvailable: ({ toolCallId, input }) => ui.complete(toolCallId, input),
  onToolInputAbandoned: ({ toolCallId, reason }) => ui.remove(toolCallId, reason),
});

A delta is the text that follows what came before, not valid JSON on its own: join the pieces of a call, or feed them to a partial JSON parser, to show the arguments as they grow.

These callbacks need stream: true, and a run with them streams also without onToken. A run without stream: true reports nothing to them. Streaming switched on only by them keeps structured output repair working: repair is skipped only when text streamed to onToken or onReasoning.

onToolInputAvailable fires as soon as a call is complete: when the provider says the model finished writing it, or when the call arrives whole. A call that never had argument text (Anthropic sends none for an empty input) gets its arguments as one delta, such as {}, before it is available. Callbacks that throw are logged, the turn goes on.

Calls That Never RunCopy Anchor Link

Every call onToolInputStart reported ends in onToolCall or in onToolInputAbandoned, so a UI never keeps a call that went nowhere. reason says why:

• incomplete: its arguments never came complete, because the turn was cut at the token limit, failed or was aborted
• discarded: it was complete, but the turn kept no such call. An afterModel middleware dropped it or gave it another tool name, the turn may not call tools (the last turn of a run at its iteration limit), or the turn was cut at the token limit after the call was complete

A complete call may still not run for other reasons: a person may decline it at an approval, and onToolCall reports the calls that run, with the arguments they run with.

Hooks on the ToolCopy Anchor Link

A tool hears the calls of its own with the same events, under the names of the AI SDK:

const writeFile = tool({
  name: 'write_file',
  description: 'Writes a file',
  parameters: z.object({ path: z.string(), content: z.string() }),
  onInputStart: ({ toolCallId }) => editor.openDraft(toolCallId),
  onInputDelta: ({ toolCallId, delta }) => editor.append(toolCallId, delta),
  onInputAvailable: ({ toolCallId, input }) => editor.settle(toolCallId, input),
  onInputAbandoned: ({ toolCallId }) => editor.discardDraft(toolCallId),
  execute: async ({ path, content }) => files.write(path, content),
});

The hooks run for every streamed run that has the tool, next to the run's own callbacks, and a run whose agent has a tool with such hooks streams too. They are read for each model turn, so after a handoff the tools of the agent that took over get them.

ProvidersCopy Anchor Link

The OpenAI-compatible backends keep the first id and name a call got, so a server that repeats them empty or changed in later pieces neither restarts nor renames the call.

A run with middleware that holds the stream back until the turn is checked, such as output guardrails, gets no input events, as it gets no tokens.

Over the WireCopy Anchor Link

In a streamed run of a server adapter, tool-call-start, tool-call-delta and tool-call-end follow the model: tool-call-start once the call's id and name are known, a tool-call-delta with each piece of its arguments in argsTextDelta, and tool-call-end as soon as the call is complete, so the events of two calls do not interleave unless the provider itself sends their pieces mixed (then a call stays open until its arguments are complete, and no piece is lost). Without input events (a run whose stream is held back), the three go out at once right before the call runs, the arguments in one delta, as before.

A call that will never run gets a tool-result with dropped: true and its reason (incomplete or discarded) instead of a result, after a tool-call-end with dropped: true when the model never finished it. The stream also drops, when it completes or fails, every call it announced that never came to run, while calls that run or wait for an approval are kept. useCogitatorChat of @cogitator-ai/next removes a dropped call from the message and calls onToolCallDropped. A client that does not know dropped sees such a call as a call with an empty result.

Custom BackendsCopy Anchor Link

A backend of your own streams tool input in delta.toolInput of its chatStream chunks: a { type: 'start', toolCallId, toolName } part once the call's id and name are known, then { type: 'delta', toolCallId, delta } parts of its JSON arguments, and a { type: 'end', toolCallId } part once the model finished writing the call, which makes it available right away. The complete call still comes in delta.toolCalls, and a call without an end is available when it arrives there. A backend that leaves toolInput out has each call reported in one piece when it arrives.

async *chatStream(request: ChatRequest): AsyncGenerator<ChatStreamChunk> {
  yield { id, delta: { toolInput: [{ type: 'start', toolCallId: 'call_1', toolName: 'search' }] } };
  yield { id, delta: { toolInput: [{ type: 'delta', toolCallId: 'call_1', delta: '{"query":' }] } };
  yield { id, delta: { toolInput: [{ type: 'delta', toolCallId: 'call_1', delta: '"cats"}' }] } };
  yield { id, delta: { toolInput: [{ type: 'end', toolCallId: 'call_1' }] } };
  yield {
    id,
    delta: { toolCalls: [{ id: 'call_1', name: 'search', arguments: { query: 'cats' } }] },
    finishReason: 'tool_calls',
  };
}

MockLLMBackend of @cogitator-ai/test-utils takes toolInput in the chunks of setStreamChunks(), to test code that reads tool input without a model.