eval - Chirp

testing.eval

module

LLM and agent eval helpers for TestClient-based regression tests.

These utilities mock provider HTTP at the httpx transport layer so AI routes andAgentRunloops can be tested without live API keys.

LLM and agent eval helpers for TestClient-based regression tests.

These utilities mock provider HTTP at the httpx transport layer so AI routes andAgentRunloops can be tested without live API keys.

testing.eval

Name	Type	Default	Description
`type`		—
`qualified_name`		—
`element_type`		—
`description`		—
`source_file`		—
`line_number`		—
`is_autodoc`		—
`autodoc_element`		—
`_autodoc_template`		—
`_autodoc_url_path`		—
`_autodoc_page_type`		—
`title`		—
`doc_content_hash`		—

Symbols on this page

class LLMScript

Scripted OpenAI-compatible chat completion responses for mocked LLM calls.

class LLMCallTracker

Counts how many complete vs stream requests the mock served.

function openai_completion

Build a mock/v1/chat/completionsresponse body.

function openai_tool_call

Build a single OpenAItool_callsentry.

function install_mock_transport

Patchhttpx.AsyncClientso chirp.ai provider calls use handler.

function install_llm_script

Install a scripted LLM mock for AgentRun / LLM HTTP calls.

function collect_sse_message_text

Join defaultmessageSSE event payloads into one string.

function assert_tool_messages_contain

Assert a later completion round includes tool result messages.

LLMScript

class

Scripted OpenAI-compatible chat completion responses for mocked LLM calls.

LLMCallTracker

class

Counts how many complete vs stream requests the mock served.

openai_completion

function

def openai_completion(content: str = '', *, tool_calls: list[dict[str, Any]] | None = None) -> dict[str, Any]

Build a mock/v1/chat/completionsresponse body.

Parameters

Name	Type	Default	Description
`content`	`str`	`''`
`tool_calls`	`list[dict[str, Any]] \| None`	`None`

openai_tool_call

function

def openai_tool_call(name: str, /, **arguments: Any) -> dict[str, Any]

Build a single OpenAItool_callsentry.

Parameters

Name	Type	Default	Description
`name`	`str`	—
`**arguments`	`Any`	—

install_mock_transport

function

def install_mock_transport(monkeypatch: pytest.MonkeyPatch, handler: Callable[..., Any]) -> None

Patchhttpx.AsyncClientso chirp.ai provider calls use handler.

Parameters

Name	Type	Default	Description
`monkeypatch`	`pytest.MonkeyPatch`	—
`handler`	`Callable[..., Any]`	—

install_llm_script

function

def install_llm_script(monkeypatch: pytest.MonkeyPatch, script: LLMScript, *, tags_response: dict[str, Any] | None = None) -> LLMCallTracker

Install a scripted LLM mock for AgentRun / LLM HTTP calls.

Parameters

Name	Type	Default
`monkeypatch`	`pytest.MonkeyPatch`	—
`script`	`LLMScript`	—
`tags_response`	`dict[str, Any] \| None`	`None`

collect_sse_message_text

function

def collect_sse_message_text(result: SSETestResult) -> str

Join defaultmessageSSE event payloads into one string.

Parameters

Name	Type	Default	Description
`result`	`SSETestResult`	—

assert_tool_messages_contain

function

def assert_tool_messages_contain(messages: list[dict[str, Any]], *, tool_name: str | None = None, text: str | None = None) -> None

Assert a later completion round includes tool result messages.

Parameters

Name	Type	Default
`messages`	`list[dict[str, Any]]`	—
`tool_name`	`str \| None`	`None`
`text`	`str \| None`	`None`

View source · /home/runner/work/chirp/chirp/site/../src/chirp/testing/eval.py:1