Project Development Methodology
Principles for identifying tasks suited to LLM processing, designing effective architectures, and iterating rapidly using agent-assisted development.
Task-Model Fit Recognition
LLM-Suited Tasks
| Characteristic | Why It Fits |
|---|---|
| Synthesis across sources | LLMs excel at combining information |
| Subjective judgment with rubrics | Grading, evaluation, classification |
| Natural language output | Human-readable text goals |
| Error tolerance | Individual failures don't break system |
| Batch processing | No conversational state needed |
| Domain knowledge in training | Model has relevant context |
LLM-Unsuited Tasks
| Characteristic | Why It Fails |
|---|---|
| Precise computation | Math, counting unreliable |
| Real-time requirements | Latency too high |
| Perfect accuracy requirements | Hallucination risk |
| Proprietary data dependence | Model lacks context |
| Sequential dependencies | Heavy step-by-step coupling |
| Deterministic output requirements | Same input ≠ identical output |
Manual Prototype Step
Before automation, validate with manual test:
- Copy one representative input into model interface
- Evaluate output quality
- This takes minutes, prevents hours of waste
Answers critical questions:
- Does model have required knowledge?
- Can it produce needed format?
- What quality level to expect at scale?
- What failure modes exist?
Pipeline Architecture
Canonical structure:
acquire → prepare → process → parse → render- Acquire: Fetch raw data (APIs, files, databases)
- Prepare: Transform to prompt format
- Process: Execute LLM calls (expensive, non-deterministic)
- Parse: Extract structured data from outputs
- Render: Generate final outputs
Stages 1, 2, 4, 5 are deterministic. Stage 3 is expensive.
File System as State Machine
Each processing unit gets a directory:
data/{id}/
├── raw.json # acquire complete
├── prompt.md # prepare complete
├── response.md # process complete
├── parsed.json # parse completeBenefits:
- Natural idempotency (file existence gates execution)
- Easy debugging (human-readable state)
- Simple parallelization (directories independent)
- Trivial caching (files persist)
Structured Output Design
Effective structure includes:
- Section markers for parsing
- Format examples showing exact output
- Rationale: "I will be parsing this programmatically"
- Constrained values (enums, ranges, formats)
Build robust parsers:
- Use flexible regex patterns
- Provide sensible defaults for missing sections
- Log failures instead of crashing
Cost Estimation
Total cost = (items × tokens_per_item × price_per_token) + overheadFor batch processing:
- Estimate input tokens (prompt + context)
- Estimate output tokens (typical response)
- Multiply by item count
- Add 20-30% buffer for retries
Single vs Multi-Agent
Single-agent works for:
- Batch processing with independent items
- Non-interacting items
- Simpler cost management
Multi-agent works for:
- Parallel exploration
- Tasks exceeding single context window
- Specialized sub-agents improving quality
Primary reason: context isolation, not role anthropomorphization.
Architectural Reduction
Start minimal. Add complexity only when proven necessary.
Vercel d0 case:
- Before: 17 specialized tools, 80% success, 274s execution
- After: 2 tools (bash + SQL), 100% success, 77s execution
When reduction wins:
- Well-documented data layer
- Sufficient model reasoning capability
- Specialized tools constraining rather than enabling
Guidelines
- Validate task-model fit with manual prototyping first
- Structure pipelines as discrete, idempotent, cacheable stages
- Use file system for state management
- Design prompts for structured, parseable outputs
- Start minimal; add complexity only when proven necessary
- Estimate costs early and track throughout
- Build robust parsers handling LLM output variations
- Expect and plan for multiple architectural iterations