Concourse CI Pipeline Development
Expert guidance for writing, refactoring, and optimizing Concourse CI pipelines (v8.0+).
When to Use
- Creating or modifying Concourse pipelines
- Configuring resources (git, registry-image, custom types)
- Building container images with
oci-build-task - Troubleshooting resource check failures or build issues
- Migrating from legacy patterns (docker-image, duplicate jobs)
Quick Reference
| Task | Modern (Recommended) | Legacy (Avoid) |
|---|---|---|
| Building images | oci-build-task + registry-image | docker-image resource |
| Multi-env deploys | across step modifier | Duplicate jobs per env |
| Dynamic pipelines | set_pipeline + instanced pipelines | Manual pipeline duplication |
| Notification symbols | UTF-8 characters (e.g. \u2714 for checkmark, \u274c for X) | HTML entities (e.g. ✓, ✗) |
| Resource styling | Always use icon: property | No icon |
Core Concepts
Pipelines consist of resources (external versioned artifacts), jobs (sequences of steps), and optional groups (UI organization). All execution runs in containers.
Key step types: get, put, task, set_pipeline, in_parallel, do, try, load_var. Job hooks: on_success, on_failure, on_error, on_abort, ensure. Note: on_failure (non-zero exit) differs from on_error (infrastructure crash/OOM) -- handle both. Use fly execute to test tasks locally.
See references/core-concepts.md for step types table, lifecycle hooks, and fly CLI essentials.
Critical Gotchas
- Git tag detection after force-push -- Escape regex dots, enable
clean_tags: true, separate read/write resources, force recheck withfly -t T check-resource -r pipeline/resource. Seereferences/resources-guide.md. - registry_mirror format mismatch --
registry-imageexpects an object (host: mirror),docker-imageexpects a URL string. Provide separate formats inCONCOURSE_BASE_RESOURCE_TYPE_DEFAULTS. Seereferences/resources-guide.md. - GitLab Container Registry JWT auth -- The JWT endpoint lives on the GitLab host, not the registry host. Discover via
Www-Authenticateheader. Seereferences/resources-guide.md.
References
references/pipeline-syntax.md-- Complete YAML schema for pipelines, jobs, resourcesreferences/core-concepts.md-- Step types, lifecycle hooks, fly CLI essentialsreferences/resources-guide.md-- Git-resource, registry-image, docker-image migration, gotcha detailsreferences/best-practices.md-- Optimization, troubleshooting, notifications, deployment patternsreferences/resource-types-catalog.md-- Available resource types (Ansible, Terraform, etc.)
Examples
Working examples in examples/:
basic-pipeline.yml-- Build-test-deploy with versioningmodern-ci-cd.yml-- oci-build-task, across, build_log_retentionmulti-branch.yml-- Dynamic branch pipelines with set_pipelinedocker-build.yml-- Container image build and pushvars-template.yml-- Variable file organization
Validation
Use scripts/validate-pipeline.sh to check pipeline syntax before deployment.