Analyze Selective Testing
Quick Start
- Run
tuist test list --jsonto find test runs. - Run
tuist test xcode target list <test-run-id> --jsonto see per-target selective testing status. - If effectiveness dropped, compare targets between a known-good run and the regressed run.
Step 1: Resolve Test Runs
Find the test run to analyze
List recent test runs, optionally filtering by branch:
tuist test list --json --page-size 10
tuist test list --git-branch feature-x --json --page-size 5Get basic info for a specific test run:
tuist test show <test-run-id> --jsonUse this to confirm the run's branch, status, scheme, and environment (xcode_version, macos_version).
Defaults
- If no test run is specified, use the most recent CI test run on the current branch.
- For comparison, use the most recent CI test run on the project's default branch.
Step 2: Drill Into Selective Testing Targets
List per-target selective testing data for a test run:
tuist test xcode target list <test-run-id> --jsonEach target includes:
- name: The test target name
- hit_status:
miss(ran),local(skipped via local cache), orremote(skipped via remote cache) - hash: The selective testing hash for this target
Filter by status to focus your analysis:
tuist test xcode target list <test-run-id> --hit-status miss --json
tuist test xcode target list <test-run-id> --hit-status local --json
tuist test xcode target list <test-run-id> --hit-status remote --jsonAssess effectiveness
Count the targets by status:
effectiveness = (local_hits + remote_hits) / total_targets * 100| Effectiveness | Verdict |
|---|---|
| 60-100% | Healthy — most unchanged tests are being skipped |
| 30-60% | Moderate — some cache invalidation occurring, worth investigating |
| 0-30% | Low — likely a core dependency changed or the cache was invalidated |
| 0% | Cold cache — first run, environment change, or full invalidation |
Step 3: Diagnose Regressions
If effectiveness dropped, investigate:
Check for global invalidation
If nearly all targets show as miss, the entire selective testing cache was invalidated. Common causes:
| Cause | How to verify |
|---|---|
| Xcode version change | Compare xcode_version between good and bad runs via tuist test show |
| CI environment change | Compare macos_version, model_identifier between runs |
| Project graph or dependency change | Check git history for changes to project manifests, dependency versions, or target configurations |
| Core dependency change | A widely-depended-on target changed, invalidating all its dependents |
Note: Tuist CLI version upgrades rarely cause hash invalidation — the hash version is not updated on every release.
Check for cascade invalidation
If only some targets show as miss, a dependency change likely cascaded:
- Run
tuist test xcode target list <good-run-id> --jsonandtuist test xcode target list <bad-run-id> --json. - Match targets by name — identify those that changed from
local/remotetomiss. - Compare hashes for changed targets — if a target's hash differs, its sources or dependencies changed.
- Look for a common dependency among the invalidated targets.
Compare two test runs
For each target:
- Match by
name - Compare
hit_status(hit -> miss = invalidated, miss -> hit = improved) - Compare
hash(different hash = target or its dependencies changed)
Step 4: Recommendations
Based on the diagnosis:
- Environment change: Ensure CI uses consistent Xcode/macOS versions. This is the most common cause of full cache invalidation.
- Cascade from dependency change: Consider if the change to the root target was necessary. If the root target is a widely-shared module, consider splitting it.
- Cold cache on new branch: Run tests once to warm the cache. Subsequent runs will benefit from selective testing.
Summary Format
Produce a summary with:
- Overall: effectiveness percentage, verdict (healthy/moderate/low/cold)
- Target breakdown: targets grouped by hit status
- Comparison (if applicable): targets that changed status between runs, with hash diffs
- Root cause: what caused the regression
- Recommendations: actionable next steps
Example:
Selective Testing Analysis: test run abc123 on feature-x
Effectiveness: 15% (3/20 targets skipped) -- LOW
Local hits: 2 targets (CoreTests, UtilTests)
Remote hits: 1 target (NetworkTests)
Misses: 17 targets
Comparison with baseline (run def456, 75% effectiveness):
14 targets changed from hit to miss
All changed targets have different hashes
Root cause: Xcode version changed from 15.2 to 16.0 between runs.
This changed all target hashes, invalidating the entire cache.
Recommendations:
- Align CI Xcode version with the version used in the baseline run.
- If the upgrade is intentional, run tests once to re-warm the cache.Done Checklist
- Identified the test run(s) to analyze
- Drilled into per-target selective testing status via
tuist test xcode target list - Diagnosed root cause if effectiveness is low
- Compared targets with baseline if regression detected
- Provided actionable recommendations