Claude上下文优化器
将Claude Code代币消费量减少高达97% --由真实的基准而不是估计来证明。
通过 阿扎兹·阿尔菲拉斯
](https://www.npmjs.com/package/claude-context-optimizer)  ](https://nodejs.org)   
______________________________________________________________________
真实基准结果
这些数字是 不是估计。它们是通过在真实机器上对真实文件运行实际工具而产生的。通过克隆repo并运行以下命令,可以复制本节中的每个数字 npm run benchmark.测试环境
| 日期 | 2026-05-01 |
| 平台 | macOS 15(Darwin 25.3)·苹果硅 |
| Node.js | v24.x |
| 测试文件 | tests/fixtures/sample.log (84行)· tests/fixtures/AuthService.ts (137行) |
| 项目 | 这个仓库(45+个源文件,约3500行) |
______________________________________________________________________
每个工具的结果
┌─────────────────────────────────────────────────────────────────────┐
│ Tool Before (tokens) After (tokens) Saved % │
├─────────────────────────────────────────────────────────────────────┤
│ compress_logs 1,508 597 911 60% │
│ smart_read (1st read) 1,245 64 1,181 95% │
│ smart_read (cache hit) 1,245 64 1,181 95% │
│ function_extractor 1,245 249 996 80% │
│ project_map 95,000 1,012 93,988 99% │
│ bulk_search 50,000 2,331 47,669 95% │
│ symbol_index (find) 18,000 44 17,956 100% │
├─────────────────────────────────────────────────────────────────────┤
│ TOTAL 168,243 4,361 163,882 97% │
└─────────────────────────────────────────────────────────────────────┘视觉
Token consumption — before vs after
Before ████████████████████████████████████████ 168,243 tokens (100%)
After █ 4,361 tokens ( 3%)
┌────────────────────────────────────────────────────────────────┐
│ │
│ 97% of tokens never reach Claude's context window. │
│ They were noise. We removed the noise. │
│ │
└────────────────────────────────────────────────────────────────┘大规模成本影响
Pricing: Claude Opus 4 at $15 / 1M input tokens
┌──────────────────┬────────────────┬────────────────┬────────────────┐
│ Session scale │ Without │ With │ Saved │
├──────────────────┼────────────────┼────────────────┼────────────────┤
│ 1 session │ $2.524 │ $0.065 │ $2.459 │
│ 10 sessions/day │ $25.24 │ $0.65 │ $24.59 │
│ 100 sessions │ $252.40 │ $6.50 │ $245.90 │
│ 1,000 sessions │ $2,524.00 │ $65.00 │ $2,459.00 │
└──────────────────┴────────────────┴────────────────┴────────────────┘
A team of 10 developers doing 5 sessions/day saves ~$1,229/day.执行速度
All tools run in well under 250ms.
Most run in under 5ms. The only "slow" path is the one-time symbol
index build, which then makes every subsequent lookup ~free.
compress_logs ██ 2ms
smart_read ███ 1ms
function_extractor ██ 2ms
project_map ██████████ 12ms ← walks disk
bulk_search ███ 3ms
symbol_index (build) ████████████████████████████████ 224ms ← one-time
symbol_index (find) █ Retrying in 5s...
> Attempt 47 of 50
Line 3891: JWT verification failed: token expired
> User: user_abc123
> Endpoint: POST /api/orders______________________________________________________________________
2. smart_read
它解决的问题是: 您需要了解身份验证是如何工作的。克劳德阅读了全部800行 AuthService.ts 当只有 login() 和 validateToken() 函数(80行)是相关的。
它是如何工作的:
- 检查会话内存--此文件在此会话之前是否已被读取?
- 检查文件哈希值——自上次读取以来是否发生了变化?
- 如果未更改且正在会话中: 零磁盘读取,返回摘要
- 如果新增/更改:读取文件,运行AST chunker(TS/JS/Python)或滑动窗口(其他文件)
- 根据您的查询对每个块进行评分 BM25+标识符感知标记化 —
auth点击AuthService,validate点击validateToken罕见的名字多于常见的名字 - 仅返回得分高于零的块,按相关性排序,上限为代币预算
语言支持:
- Types/JavaScript:通过AST提取函数、类和接口
- Python:提取符合缩进结构的defs和类
- Go、Rust、Java、C#、Ruby、PHP:基于正则表达式的签名提取
- YAML、JSON、Markdown、任何文本:带有相关性评分的滑动窗口
例子:
smart_read({ file_path: "/app/src/auth/AuthService.ts", query: "JWT token validation" })
// Returns only:
## /app/src/auth/AuthService.ts (from cache — unchanged)
600 lines | typescript
### Lines 145–187 — `validateToken`async validateToken(token: string): Promise { // ... only this function }
3. file_diff_only
The problem it solves: You changed 5 lines in a 400-line file. Claude reads all 400 lines to understand the change.
How it works: Runs git diff and returns only the changed lines with configurable context. Works against HEAD, any commit, any branch, or staged changes.
Example:
file_diff_only({ file_path: "/app/src/server.ts", base: "main" })
// Returns:
## Diff: server.ts vs main@@ -45,6 +45,8 @@ app.use(cors()) +app.use(helmet()) +app.use(rateLimit({ windowMs: 15 * 60 * 1000, max: 100 })) app.use(express.json())
令牌:完整文件为~150,而不是~4000。
4. project_map
The problem it solves: You open a new codebase. Claude reads 20 files to understand the structure. You could have understood the entire project in 300 tokens.
How it works: Walks the directory tree (ignoring node_modules, dist, .git, etc.), collects every source file, identifies languages, estimates token costs, groups by directory, and returns a single compressed map.
Example output:
## 项目地图:/app
47个文件|12450行|总共约31k个令牌
### 按语言
- 打字稿:32个文件
- 标记:8个文件
- yaml:4个文件
- json:3个文件
### 文件
**/src/auth/**
- AuthService.ts--服务(约1.2k令牌)
- JWTUtil.ts--实用程序(约400个令牌)
- middleware.ts——服务(约300个令牌)
**/src/api/**
- router.ts--路由(约500个令牌)
- handlers.ts——控制器(约800个令牌)
5. context_budget
The problem it solves: You don't know how close you are to the context limit until Claude stops working or starts forgetting things. By then it's too late.
How it works: Analyzes items in your context (or auto-pulls from session history), estimates tokens for each, categorizes them by whether they should be kept or removed, and gives specific recommendations with projected savings.
Budget categories:
keep— core files actively being worked onconsider-removing— large files read early in the session, now staleremove— log files, lock files, generated code
6. bulk_search
The problem it solves: You need to find where validateUser is called across the codebase. Claude reads 30 files to find 8 matches.
How it works: Recursively searches all files (respecting ignore patterns), runs regex against each line, returns only matching lines with 2 lines of context per match. Never returns full file content.
Example:
bulk_search({ pattern: "validateUser", file_extensions: [".ts"] })
// Returns:
## Search: `validateUser` in /app
8 matches in 5 files
### src/api/handlers.ts
L45: `const user = await validateUser(req.headers.authorization)`
> if (!user) return res.status(401).json({ error: 'Unauthorized' })
### src/auth/AuthService.ts
L112: `async validateUser(token: string): Promise`______________________________________________________________________
7. recall_file
它解决的问题是: 你让克劳德“再次查看AuthService.ts”。它读取整个文件。文件在30分钟内没有更改。
它是如何工作的: 检查会话内存中的文件路径。如果找到,则计算当前的统计哈希(快速——不读取文件),并与缓存的哈希进行比较。如果未更改,则返回缓存的摘要并确认不需要重新读取。
未更改文件的标记为零。 这是该组合中杠杆率最高的工具。
______________________________________________________________________
8. dependency_graph
它解决的问题是: 在修改共享实用程序之前,您需要知道它依赖于什么。理解这一点通常需要阅读许多文件。
它是如何工作的: 解析所有代码文件中的导入语句,构建一个有向图 imports → imported by 关系,返回文件级视图或项目级视图,显示导入最多的模块。
______________________________________________________________________
9. function_extractor
它解决的问题是: 您需要从600行文件中看到一个特定的函数。你只需要30行。
它是如何工作的: 使用AST分块器按确切名称定位函数或类。如果找不到确切的名字,则返回相关性评分。仅返回匹配的函数及其文件路径和行号。
例子:
function_extractor({ file_path: "/app/src/auth/AuthService.ts", name: "login" })
// Returns:
## `login` — /app/src/auth/AuthService.ts:67
async login(email: string, password: string): Promise { const user = await this.userRepo.findByEmail(email); if (!user) throw new AuthError('User not found'); const valid = await bcrypt.compare(password, user.passwordHash); if (!valid) throw new AuthError('Invalid credentials'); return { token: this.jwt.sign({ userId: user.id }), user }; }
令牌:完整文件为~200,而不是~6000。
10. session_snapshot
The problem it solves: Long tasks get interrupted. You come back to Claude, it's lost context of what was being worked on, and re-reading everything costs tokens.
How it works: Saves a snapshot of the current session — which files were read, their hashes, and a summary of the current state. On restore, returns this snapshot so Claude can resume without re-reading files that haven't changed.
11. task_manager
The problem it solves: A 30-step task fills the context window before it's done. Without persistence, you start over and lose all decisions and partial progress.
How it works: Breaks a task into subtasks, persists them to disk along with decisions, observations, and changed files. When the context fills up, checkpoint produces a ~300-token *resume prompt*; in a new Claude Code session, resume restores the full state in that one tool call.
Supports semantic observation types — bugfix | feature | decision | discovery | warning — so resume prompts come back grouped and scannable instead of a flat blob of notes.
task_manager({ action: "create", title: "...", tasks: [...] }) // start
task_manager({ action: "complete", task_id: "1", outcome: "..." })// progress
task_manager({ action: "checkpoint", observations: [...] }) // before context fills
task_manager({ action: "resume" }) // in new session看 上下文崩溃问题 全程步行。
______________________________________________________________________
12. context_watchdog
它解决的问题是: 直到克劳德开始忘记事情,你才会注意到上下文是完整的。那么你就输了。
它是如何工作的: 根据会话内存+您传入的任何额外令牌估计当前上下文使用情况,并返回分层状态:
70% → ⚡ warning — good time to checkpoint
85% → 🔴 critical — checkpoint strongly recommended
95% → 🚨 emergency — auto-checkpoint, output the resume prompt在紧急情况下,如果 auto_checkpoint: true (默认),它会自动持久化当前任务,因此即使是失控的循环也会产生可恢复的状态。
______________________________________________________________________
13. symbol_index
它解决的问题是: “在哪里 validateToken 定义?“--如果没有这个工具,Claude会读取多个文件来找出答案。有了它,答案就是一行文本。
它是如何工作的: 一个持续的项目范围索引。一次扫描将每个函数、类、方法、接口、类型和枚举提取到 { name, kind, file, line, signature } 记录。后续查找是本地和免费的。每个文件的哈希值都会被存储,因此重新索引会跳过未更改的文件。
标识符感知匹配意味着部分查询词命中camelCase/snake_case组件-- auth 发现 AuthService, validate 发现 validateToken.
// One-time (or after large refactors)
symbol_index({ action: "rebuild" })
// "Where is X defined?" — ~30 tokens / hit
symbol_index({ action: "find", name: "TokenEstimator" })
// All symbols in one file — replaces a skim-read
symbol_index({ action: "outline", file_path: "/app/src/auth/AuthService.ts" })
// Stats / sanity check
symbol_index({ action: "stats" })
// Re-index a single file (e.g. after editing)
symbol_index({ action: "refresh", file_path: "/app/src/auth/AuthService.ts" })Input: "find TokenEstimator across the project"
Cost without symbol_index: ~18,000 tokens (read every .ts file)
Cost with symbol_index: ~44 tokens
Saved: ~100%______________________________________________________________________
技术选择
为什么是单个JSON存储(不是SQLite)?
我们从SQLite开始(通过 better-sqlite3)并放弃了它。原因:
- 本机编译中断安装 —
better-sqlite3每个节点版本都需要一个可用的C++工具链。在Apple Silicon、Windows和几个Linux发行版上,这就是用户陷入困境的地方。 - 对于我们的工作负载,JSON就足够了。 我们在每次工具调用时都会联系商店,但 *总计* 数据集很小,只有几十KB。完整解析大约需要1毫秒。
- 一个商店,一个真理。 这
JsonStore是一个由文件路径键控的进程级单例。每个引擎(FileCache、SessionMemory、SnapshotManager、TaskStore、SymbolIndex)共享相同的内存副本,因此写入永远不会相互干扰。
我们放弃了什么:索引查找和跨进程并发。对于每会话MCP服务器来说,这两个都不重要。
Old (SQLite, broken): New (JsonStore singleton):
FileCache ──► db.sqlite FileCache ─┐
Session ──► db.sqlite ├──► JsonStore (in-memory)
Tasks ──► db.sqlite Session ─┤ ├── flush to JSON
Snapshots ──► db.sqlite Tasks ─┤
Snapshots ─┤
Compilation breaks on: Symbols ─┘
• Apple Silicon (some)
• Windows (most) Zero native code. Works on Node 18 → 24.
• Alpine / musl为什么不 tiktoken?
tiktoken 虽然准确,但:
- 需要本机编译(某些系统中断)
- 向包中添加10+MB
- 首次使用时加载需要200毫秒
我们的 chars / 4 估算值为:
- 英语/代码内容准确率在10%以内(足以用于预算)
- 即时--零开销
- 零依赖
- 在所有平台上工作方式相同
为什么是基于正则表达式的AST解析,而不是真正的AST解析器?
一个真正的TypeScript AST解析器(@typescript-eslint/parser, ts-morph)会更准确。但是:
- 添加50-200MB的依赖项
- 解析大文件需要500ms–2s
- 存在语法错误的文件中断
- 每种语言需要单独的解析器
我们基于正则表达式/缩进的方法:
- ~ 0ms解析时间(单程行扫描)
- 使用一个模式表处理12种语言
- 优雅地处理语法错误(返回发现的内容)
- 添加零依赖项
对于用例(提取用于令牌优化的函数边界),这种精度是足够的。
为什么BM25+标识符感知标记化(不是嵌入)?
一个天真的关键字评分员会平等对待每个单词。所以像这样的查询 *“auth”* 从不匹配 AuthService --文字子字符串不是作为单独的单词存在的。一个提到50次通用术语的块将战胜一个提到一次罕见的、命名完美的标识符的块。
我们选择了最小的工具来解决这两个问题:
- BM25 --关键字搜索的事实基线。三个属性很重要:
- 以色列国防军:罕见术语(validateToken)超过普通(user) - TF饱和度 (k1):一个重复“user”50×的块不会击败5××10×的块 - 长度归一化 (b):长块不会自动占据主导地位
- 标识符标记化 --每个代码标识符都被拆分为组成词:
- loginUser → [loginuser, login, user] - AuthService → [authservice, auth, service] - HTTPSConnection → [httpsconnection, https, connection] - get_user_id → [get_user_id, get, user, id]
在此之后,查询 *“auth”* 点击 AuthService, *“验证”* 点击 validateToken,以及 *“刷新令牌”* 等级 refreshTokens() 上面不相关的块。
我们故意跳过本地嵌入(例如。 all-MiniLM-L6-v2 通过 @xenova/transformers).它们增加了约80 MB的权重、约3秒的启动时间和 onnxruntime 依赖性——在短标识符密集的代码查询中,BM25的边际收益。
______________________________________________________________________
它能节省多少钱?
| 场景 | 无 | 有 | 保存 |
|---|---|---|---|
| 读取一个函数的500行文件 | ~5000个标记 | ~200个标记 | 96% |
| 读取5000行日志 | ~50000个令牌 | ~500个令牌 | 99% |
| 重新读取未更改的文件 | ~5000个令牌 | 0个令牌 | 100% |
| 了解一个新项目(20个文件) | ~80000个代币 | ~500个代币 | 99% |
| 在30个文件中查找模式 | ~300000个令牌 | ~2000个令牌 | 99% |
| 典型的20轮工作会议 | 约500000个代币 | 约80000个代币 | 84% |
视觉:每回合代币消耗
Tokens/turn (typical session — 20 turns)
Without optimizer:
Turn 1 ████████████████████████████████ 32,000
Turn 2 ████████████████████████████████ 31,000
Turn 3 ████████████████████████████████ 33,000 ← re-reads same files
Turn 5 ████████████████████████████████ 35,000
Turn 10 ███████████████████████████████████████ 42,000
Turn 15 ████████████████████████████████████████████ 48,000 ← context filling
Turn 20 ██████████ 9,000 ← Claude starts forgetting, quality drops
With optimizer:
Turn 1 ████████ 8,000 ← first read + cache
Turn 2 ███ 3,000 ← recall_file: unchanged, 0 tokens
Turn 3 ████ 4,000
Turn 5 ███ 2,500 ← smart_read: only relevant chunk
Turn 10 ███ 3,000
Turn 15 ███ 3,500
Turn 20 ████ 4,000 ← context stays clean, quality stays high
Total: Without = ~520,000 With = ~82,000 Saved = 84%缓存命中率随时间的变化
Cache hits (%) as session progresses
100% ┤ ············
90% ┤ ·····
80% ┤ ·····
70% ┤ ·····
60% ┤ ·····
50% ┤ ·····
40% ┤ ·····
30% ┤ ·····
20% ┤·
0% ┼────────────────────────────────────────────────
Turn 1 Turn 5 Turn 10 Turn 15 Turn 20
Every turn, more files are cached.
By Turn 10, ~80% of file requests cost 0 tokens.______________________________________________________________________
决策树:使用哪种工具
You need to work with a file or codebase...
│
▼
┌─────────────────────────────────────┐
│ Have I read this file this session? │
└───────────────────┬─────────────────┘
│ │
yes no
│ │
▼ ▼
┌──────────────┐ ┌────────────────────────────────────┐
│ recall_file │ │ What do I need from the file? │
│ │ └────────────┬───────────────────────┘
│ unchanged? │ │
│ → 0 tokens │ ┌──────┴──────────┐
│ changed? │ │ │
│ → smart_read│ specific understand
└──────────────┘ function/class how it works
│ │
▼ ▼
function_extractor smart_read
(name: "login") (query: "...")
You need to understand the whole project...
│
▼
┌──────────────────────────────────────┐
│ project_map │
│ Get the full structure in ~300 tok │
└──────────────────────────────────────┘
│
▼ (then drill down with)
dependency_graph → function_extractor → smart_read
You need to find something across the codebase...
│
▼
┌──────────────────────────────────────────────────────┐
│ Looking for a SYMBOL definition (function, class)? │
│ → symbol_index({ action: "find", name: "..." }) │
│ ~30 tokens / hit. Try this FIRST. │
│ │
│ Looking for a free-text PATTERN or usage? │
│ → bulk_search({ pattern: "..." }) │
│ Snippets, never full files. 3-tier disclosure.│
└──────────────────────────────────────────────────────┘
You have a huge log file...
│
▼
┌──────────────────────────────────────┐
│ compress_logs │
│ 5,000 lines → 40 relevant entries │
│ deduplicates repeated errors │
└──────────────────────────────────────┘
You want to see what changed in a file...
│
▼
┌──────────────────────────────────────┐
│ file_diff_only │
│ git diff vs HEAD or any branch │
│ returns only changed lines │
└──────────────────────────────────────┘资源使用情况
此服务器旨在消费 几乎没有CPU或内存:
| 资源 | 使用情况 | 为什么 |
|---|---|---|
| 内存 | 约15 MB | Node.js基线+小内存存储 |
| CPU(空闲) | 0% | 无轮询,无监视器 |
| CPU(每次调用) | \ 怎么 claude-context-optimizer 与其他克劳德记忆/上下文工具相比? |
┌─────────────────────────────────────────────────────────────────────────────────┐
│ claude-context-optimizer vs claude-mem │
├─────────────────────────────────────────┬───────────────────────────────────────┤
│ claude-context-optimizer (this project)│ claude-mem (thedotmack) │
├─────────────────────────────────────────┼───────────────────────────────────────┤
│ PROBLEM: Token waste in current session│ PROBLEM: Forgetting past sessions │
│ WHEN: Right now, as you work │ WHEN: Next week, new conversation │
│ HOW: On-demand, zero background work │ HOW: Background HTTP server + DB │
│ DEPS: Node.js only │ DEPS: Bun + Python + uv + ChromaDB │
│ LICENSE: MIT │ LICENSE: AGPL-3.0 │
│ INSTALL: npx one-liner │ INSTALL: Plugin marketplace │
├─────────────────────────────────────────┴───────────────────────────────────────┤
│ │
│ They solve DIFFERENT problems. They are COMPLEMENTARY, not competing. │
│ │
│ claude-mem = long-term episodic memory ("what did we do last sprint?") │
│ this tool = real-time token efficiency ("don't re-read unchanged files") │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘我们从claude mem那里整合了什么
claude mem的三个想法适用于我们的建筑:
1. 标签剥离 — smart_read 现在自动编校 ... 在内容到达Claude的上下文之前。将API密钥、机密或PII放入任何源文件中的这些标记中。
// Any file can contain:
const config = {
apiKey:
sk-proj-real-key-here
, // redacted from context
endpoint: 'https://api.example.com',
};2.键入的意见 — task_manager 现在支持语义观察类型(bugfix | feature | decision | discovery | warning),使简历提示更加结构化和可扫描:
task_manager({
action: "checkpoint",
observations: [
{ type: "bugfix", content: "fixed JWT expiry race condition in auth middleware" },
{ type: "decision", content: "using bcrypt rounds=12 for password hashing" },
{ type: "discovery", content: "rate limiter was silently swallowing 429 errors" }
]
})恢复提示现在按类型和图标分组(🐛 Bug修复,✨ 特征,💡 决定,🔍 发现,⚠️ 警告)。
3.逐步披露 bulk_search --从低成本开始,只在需要时进行深入研究:
Layer 1 — detail_level: "files" → ~50 tokens (just file paths + match count)
Layer 2 — detail_level: "lines" → ~200 tokens (matching lines, no context)
Layer 3 — detail_level: "context" → full output (lines + surrounding code)// Step 1: find which files are relevant
bulk_search({ pattern: "useEffect", detail_level: "files" })
// Step 2: only if you need the lines
bulk_search({ pattern: "useEffect", file_extensions: [".tsx"], detail_level: "lines" })______________________________________________________________________
*建造是因为克劳德很强大,但象征性的浪费是真实的。该项目的存在是为了使克劳德代码在规模上可持续发展。*
