skill.smoke

Page actions AI-ready formats and sharing
Open LLM text
Share with AI
Ask Claude Ask ChatGPT Ask Gemini Ask Copilot
module

Skill smoke harness — golden NL corpus + faithful-answer scorer.

Agentic-COBOL methodology (InvestorClaw / Orrery publish oracle): a skill passes smoke only when each corpus prompt yields an answer that is faithful…

Skill smoke harness — golden NL corpus + faithful-answer scorer.

Agentic-COBOL methodology (InvestorClaw / Orrery publish oracle): a skill passes smoke only when each corpus prompt yields an answer that is faithful to the tool's engine JSON — not a refusal, capability catalog, or section-skip fallback.

This module is the L3.2 harness; publish-gate wiring lives in chirp.skill.publish / chirp skill publish.

skill.smoke

Name Type Default Description
type —
qualified_name —
element_type —
description —
source_file —
line_number —
is_autodoc —
autodoc_element —
_autodoc_template —
_autodoc_url_path —
_autodoc_page_type —
title —
doc_content_hash —

Symbols on this page

class CorpusPrompt

One golden natural-language prompt tied to a skill tool invocation.

Jump to symbol class SmokeVerdict

Pass/fail result for one answer scored against engine JSON.

Jump to symbol class SmokeResult

Per-prompt smoke outcome.

Jump to symbol class SmokeReport

Aggregate report for a corpus run.

Jump to symbol function score_answer

Score a user-facing answer against engine JSON (Agentic-COBOL verdict).

Failures (checked in order):

  • empty / stub body
  • refusal markers
  • section-skip markers
  • capability-catalog blurbs
  • missing…
Jump to symbol function render_faithful_answer

Default answer_fn: narrate engine JSON facts (used by the fixture harness).

Jump to symbol function run_smoke

Run the golden corpus against a frozen app's MCP tools and score answers.

answer_fn(prompt, engine_payload)supplies the user-facing narrative scored against the tool result.…

Jump to symbol function make_fixture_skill

Build a minimal signed skill withecho + weathertools.

Used by tests and as the canonical smoke fixture for the publish oracle. Generates…

Jump to symbol
CorpusPrompt
class

One golden natural-language prompt tied to a skill tool invocation.

SmokeVerdict
class

Pass/fail result for one answer scored against engine JSON.

SmokeResult
class

Per-prompt smoke outcome.

SmokeReport
class

Aggregate report for a corpus run.

score_answer
function 
def score_answer(answer: str, engine_json: Any, *, required_facts: Sequence[str] = ()) -> SmokeVerdict

Score a user-facing answer against engine JSON (Agentic-COBOL verdict).

Failures (checked in order):

  • empty / stub body
  • refusal markers
  • section-skip markers
  • capability-catalog blurbs
  • missing required facts (or no overlap with engine JSON facts)

Parameters

Name Type Default Description
answer str —
engine_json Any —
required_facts Sequence[str] ()
render_faithful_answer
function 
def render_faithful_answer(prompt: CorpusPrompt, engine_json: Any) -> str

Default answer_fn: narrate engine JSON facts (used by the fixture harness).

Parameters

Name Type Default Description
prompt CorpusPrompt —
engine_json Any —
run_smoke
function 
def run_smoke(app: Any, corpus: Sequence[CorpusPrompt], *, answer_fn: AnswerFn | None = None, call_tool: AsyncToolCaller | None = None) -> SmokeReport

Run the golden corpus against a frozen app's MCP tools and score answers.

answer_fn(prompt, engine_payload)supplies the user-facing narrative scored against the tool result. Defaults torender_faithful_answer() so a correctly mounted fixture skill passes end-to-end.

Parameters

Name Type Default Description
app Any —
corpus Sequence[CorpusPrompt] —
answer_fn AnswerFn | None None
call_tool AsyncToolCaller | None None
make_fixture_skill
function 
def make_fixture_skill(*, private_key: Any | None = None, key_id: str = 'fixture-1') -> Any

Build a minimal signed skill withecho + weathertools.

Used by tests and as the canonical smoke fixture for the publish oracle. Generates an Ed25519 keypair whenprivate_keyis omitted (requires chirp[skill]/ cryptography).

Parameters

Name Type Default Description
private_key Any | None None
key_id str 'fixture-1'

View source · /home/runner/work/chirp/chirp/site/../src/chirp/skill/smoke.py:1