- name
- langsmith-cli
- description
- >
- metadata
- {"openclaw": {"requires": {"env": ["LANGSMITH_API_KEY"]}, "primaryEnv": "LANGSMITH_API_KEY"}}
LangSmith CLI Skill
CLI: scripts/langsmith.py. Requires LANGSMITH_API_KEY in env (or ~/.zshrc).
No second API key needed — the ask command fetches and formats traces as structured context for your agent to analyze. No trace data is sent to any third-party LLM.
Commands
Tier 0 — Ask (agent Q&A over traces)
python3 scripts/langsmith.py ask "<question>" --project <name> [--since 24h] [--limit 50]Fetches recent runs and prints them as structured JSON context. Your agent reads the output and answers the question — no external LLM calls, no data leaving your machine beyond the LangSmith API.
Examples:
ask "why is my chain slow this week" --project my-projectask "what do failing runs have in common" --project my-project --since 7dask "did the system prompt change on Friday affect output quality" --project my-project
Tier 1 — Situational Awareness
python3 scripts/langsmith.py runs <project> [--since 2h] [--status error|success] [--limit 20]
python3 scripts/langsmith.py cost <project> [--since 7d] # token spend by chain/node
python3 scripts/langsmith.py latency <project> [--since 24h] # p50/p95/p99 per run nameTier 2 — Before/After Comparisons
python3 scripts/langsmith.py diff <project> --before <ISO_date> --after <ISO_date>
python3 scripts/langsmith.py prompt-diff <run_id_a> <run_id_b>diff compares avg latency, error rate, cost, output length across two time windows. prompt-diff shows side-by-side system prompts + outputs for two specific runs.
Tier 3 — Deep Analysis (stubs, expand as needed)
python3 scripts/langsmith.py cluster-failures <project> [--since 7d]
python3 scripts/langsmith.py replay <run_id>Auth Setup
export LANGSMITH_API_KEY=<your-key>
# or add to ~/.zshrcTest with: python3 scripts/langsmith.py runs <project> --limit 3
Security & Data Flow
This skill makes outbound network requests only to api.smith.langchain.com (the LangSmith API). That's it.
LANGSMITH_API_KEY— sent as an HTTP header toapi.smith.langchain.comonly. Never logged or stored.- Trace data — fetched from LangSmith and printed to stdout for your agent to read. No trace data is sent to any third-party LLM or external service.
- No second API key required — the
askcommand outputs structured trace context for your existing agent to analyze, rather than making its own LLM calls. - No telemetry — the script collects no usage data.
The script is ~300 lines of pure Python with no obfuscation. Audit it at scripts/langsmith.py.
API Reference
See references/langsmith-api.md for endpoint details and run object schema.