databricks-cli（databricks CLI 搜索）

Databricks

Core skill for Databricks CLI, authentication, and data exploration.

Product Skills

For specific products, use dedicated skills:

• databricks-jobs - Lakeflow Jobs development and deployment
• databricks-pipelines - Lakeflow Spark Declarative Pipelines (batch and streaming data pipelines)
• databricks-apps - Full-stack TypeScript app development and deployment
• databricks-lakebase - Lakebase Postgres Autoscaling project management

Prerequisites

1. CLI installed: Run databricks --version to check.

- If the CLI is missing or outdated (< v0.292.0): STOP. Do not proceed or work around a missing CLI. - Read the CLI Installation reference file and follow the instructions to guide the user through installation. - Note: In sandboxed environments (Cursor IDE, containers), install commands write outside the workspace and may be blocked. Present the install command to the user and ask them to run it in their own terminal.

1. Authenticated: databricks auth profiles

- If not: see CLI Authentication

Profile Selection - CRITICAL

NEVER auto-select a profile.

1. List profiles: databricks auth profiles
2. Present ALL profiles to user with workspace URLs
3. Let user choose (even if only one exists)
4. Offer to create new profile if needed

Claude Code - IMPORTANT

Each Bash command runs in a separate shell session.

# WORKS: --profile flag
databricks apps list --profile my-workspace

# WORKS: chained with &&
export DATABRICKS_CONFIG_PROFILE=my-workspace && databricks apps list

# DOES NOT WORK: separate commands
export DATABRICKS_CONFIG_PROFILE=my-workspace
databricks apps list  # profile not set!

Data Exploration — Use AI Tools

Use these instead of manually navigating catalogs/schemas/tables:

# discover table structure (columns, types, sample data, stats)
databricks experimental aitools tools discover-schema catalog.schema.table --profile <PROFILE>

# run ad-hoc SQL queries
databricks experimental aitools tools query "SELECT * FROM table LIMIT 10" --profile <PROFILE>

# find the default warehouse
databricks experimental aitools tools get-default-warehouse --profile <PROFILE>

See Data Exploration for details.

Quick Reference

⚠️ CRITICAL: Some commands use positional arguments, not flags

# current user
databricks current-user me --profile <PROFILE>

# list resources
databricks apps list --profile <PROFILE>
databricks jobs list --profile <PROFILE>
databricks clusters list --profile <PROFILE>
databricks warehouses list --profile <PROFILE>
databricks pipelines list --profile <PROFILE>
databricks serving-endpoints list --profile <PROFILE>

# ⚠️ Unity Catalog — POSITIONAL arguments (NOT flags!)
databricks catalogs list --profile <PROFILE>

# ✅ CORRECT: positional args
databricks schemas list <CATALOG> --profile <PROFILE>
databricks tables list <CATALOG> <SCHEMA> --profile <PROFILE>
databricks tables get <CATALOG>.<SCHEMA>.<TABLE> --profile <PROFILE>

# ❌ WRONG: these flags/commands DON'T EXIST
# databricks schemas list --catalog-name <CATALOG>    ← WILL FAIL
# databricks tables list --catalog <CATALOG>           ← WILL FAIL
# databricks sql-warehouses list                       ← doesn't exist, use `warehouses list`
# databricks execute-statement                         ← doesn't exist, use `experimental aitools tools query`
# databricks sql execute                               ← doesn't exist, use `experimental aitools tools query`

# When in doubt, check help:
# databricks schemas list --help

# get details
databricks apps get <NAME> --profile <PROFILE>
databricks jobs get --job-id <ID> --profile <PROFILE>
databricks clusters get --cluster-id <ID> --profile <PROFILE>

# bundles
databricks bundle init --profile <PROFILE>
databricks bundle validate --profile <PROFILE>
databricks bundle deploy -t <TARGET> --profile <PROFILE>
databricks bundle run <RESOURCE> -t <TARGET> --profile <PROFILE>

Clusters

Default: general-purpose

Use for all standard workloads (data exploration, SQL, non-GPU notebooks).

GPU/ML: machine-learning (gpu)

Use for GPU-accelerated training, inference, and embedding workloads.

Running Notebooks on the Cluster

Upload a local .py notebook to the Databricks workspace and execute it on the cluster via the generic runner job, with automatic error fixing and retry.

Default cluster: general-purpose — use machine-learning (gpu) only when GPU acceleration is needed.

Config

Steps

1. Derive workspace path — strip .py, keep filename:

local:     notebooks/3.1-sales-visit-scoring.py
workspace: /Users/dominik.peter@bmsuisse.ch/AIActionPlan/notebooks/3.1-sales-visit-scoring

2. Upload:

databricks workspace import "<WORKSPACE_PATH>" \
  --file "<LOCAL_PATH>" --format SOURCE --language PYTHON \
  --overwrite --profile premium

3. Trigger:

databricks jobs run-now --profile premium --json '{
  "job_id": 194658749253431,
  "notebook_params": {"notebook_path": "<WORKSPACE_PATH>"}
}'

Capture both the top-level run_id and tasks[0].run_id (task run ID) from the response.

4. Check result — look at state.result_state: SUCCESS → done, FAILED → fix and retry.

5. Fetch error using the task run_id:

databricks api get "/api/2.1/jobs/runs/get-output" \
  --profile premium --json '{"run_id": <TASK_RUN_ID>}'

Read error (short) and error_trace (full traceback).

6. Fix and retry — read the local notebook, fix the root cause, re-upload, rerun. After 3 failed attempts stop and explain what was tried.

Common error fixes

Ad-hoc Job Submit

Run a notebook (or Python script already uploaded to the workspace) directly as a one-time job — no runner job wrapper needed.

⚠️ The CLI uses a tasks array, not top-level existing_cluster_id / notebook_task.

databricks jobs submit --profile premium --json '{
  "run_name": "my_ad_hoc_run",
  "tasks": [{
    "task_key": "main",
    "existing_cluster_id": "<CLUSTER_ID>",
    "notebook_task": {
      "notebook_path": "<WORKSPACE_PATH>",
      "base_parameters": {"sample": "500", "epochs": "1"}
    }
  }]
}'

Capture the run_id from the response, then poll and fetch output the same way as the runner job (steps 4–6 above).

For Python scripts (not notebooks), use spark_python_task instead:

databricks jobs submit --profile premium --json '{
  "run_name": "my_script_run",
  "tasks": [{
    "task_key": "main",
    "existing_cluster_id": "<CLUSTER_ID>",
    "spark_python_task": {
      "python_file": "<WORKSPACE_PATH>.py",
      "parameters": ["--sample", "500"]
    }
  }]
}'

Scripts

All scripts are accessed through the single entrypoint scripts/cli.py. run.py and metadata.py are libraries — do not invoke them directly.

uv run scripts/cli.py <subcommand> [flags]

run — Execute code on a cluster

Opens an execution context, runs Python or SQL, polls for the result, prints output, and cleans up.

# Python — inline
uv run scripts/cli.py run \
  --profile premium \
  --cluster-id <CLUSTER_ID> \
  --lang python \
  --code "print(spark.version)"

# Python — from a local file (language inferred from .py extension)
uv run scripts/cli.py run \
  --profile premium \
  --cluster-id <CLUSTER_ID> \
  --file path/to/script.py

# SQL
uv run scripts/cli.py run \
  --profile premium \
  --cluster-id <CLUSTER_ID> \
  --lang sql \
  --code "SELECT * FROM catalog.schema.table LIMIT 10"

# Pass arguments to a Python script (injected as ARGS dict)
uv run scripts/cli.py run \
  --profile premium \
  --cluster-id <CLUSTER_ID> \
  --file migrate.py \
  --args '{"catalog": "prod", "dry_run": true}'

Inside the script, read from ARGS instead of sys.argv:

catalog = ARGS["catalog"]
dry_run = ARGS.get("dry_run", False)

Flags

Stdout goes to stdout; errors and tracebacks go to stderr with a non-zero exit code.

tables — List tables in a schema

uv run scripts/cli.py tables \
  --profile premium \
  --catalog <CATALOG> \
  --schema <SCHEMA>

Outputs a markdown table with fully qualified name, table type, and description. No cluster needed.

describe — Full metadata for a table

uv run scripts/cli.py describe \
  --profile premium \
  <CATALOG.SCHEMA.TABLE>

Outputs table type, owner, comment, and a column-level table with types, nullability, and per-column descriptions. No cluster needed.

search — Find tables by keyword

# Search across all catalogs
uv run scripts/cli.py search --profile premium <KEYWORD>

# Restrict to a specific catalog
uv run scripts/cli.py search --profile premium <KEYWORD> --catalog <CATALOG>

Queries system.information_schema.tables via experimental aitools tools query. No cluster needed.

Running SQL on the Cluster

Use the execution context API to run SQL directly on the cluster — full Spark + Unity Catalog access, no separate warehouse needed. Prefer this over the SQL warehouse when debugging notebook errors (same Spark session, sees temp views and uncommitted Delta writes).

Use general-purpose for standard SQL; use machine-learning (gpu) (0303-075313-aono4t6a) for GPU workloads.

# 1. Create context (use general-purpose cluster ID; swap for GPU cluster if needed)
databricks api post /api/1.2/contexts/create --profile premium \
  --json '{"clusterId": "<CLUSTER_ID>", "language": "python"}'

# 2. Execute (use spark.sql() for SQL, or any Python)
databricks api post /api/1.2/commands/execute --profile premium --json '{
  "clusterId": "<CLUSTER_ID>",
  "contextId": "<CONTEXT_ID>",
  "language": "python",
  "command": "spark.sql(\"SELECT * FROM transf.sales LIMIT 5\").show(truncate=False)"
}'

# 3. Poll until status = Finished
databricks api get /api/1.2/commands/status --profile premium --json '{
  "clusterId": "0303-075313-aono4t6a",
  "contextId": "<CONTEXT_ID>",
  "commandId": "<COMMAND_ID>"
}'

Useful diagnostic queries:

spark.sql("DESCRIBE EXTENDED <catalog>.<schema>.<table>").show(truncate=False)
spark.sql("SELECT * FROM <table> LIMIT 10").show(truncate=False)
spark.sql("SHOW TABLES IN <schema>").show()

Troubleshooting

Required Reading by Task

Reference Guides

• Databricks CLI docs — official CLI reference (commands, flags, auth, bundles)
• Databricks llms.txt — machine-readable index of all Databricks docs
• CLI Installation
• CLI Authentication
• Data Exploration
• Asset Bundles
• Databricks SQL CLI (Azure docs) — SQL CLI options, authentication methods, and output formats