Databricks Cost Leak Hunter

Audits a Databricks workspace for real-dollar cost leaks — idle compute, jobs on

the wrong SKU, overprovisioned clusters, and the Photon premium paid without the

speedup — then emits a CFO-grokkable, dollar-ranked FinOps remediation report.

Overview

This skill finds where Databricks money is leaking and how much, in dollars, per

month. Confirmed-spend figures come from the customer's own system.billing.usage

joined to system.billing.list_prices — never an estimate. Two of the four

categories (overprovisioning, Photon premium) are explicitly modeled/at-risk

amounts, labeled as such so a CFO never confuses them with recoverable spend. The

skill surfaces four named leak categories, ranks them by monthly dollar impact, and

explains each root cause in FinOps language a CFO can act on without an engineer to

translate.

It is architecturally distinct from the v1 databricks-cost-tuning skill: that one

AUTHORS policy (creates cluster policies, spot configs). This one DETECTS leaks and

reports them, dollarized and ranked. The math is deterministic — bundled scripts in

scripts/ do the arithmetic so the agent never eyeballs numbers — and deep domain

knowledge lives in references/ loaded only when a leak needs it.

The skill uses two data planes. Dollar figures come from the **Databricks CLI

Statement Execution API** (databricks api post /api/2.0/sql/statements) reading

system.* — authenticated by the CLI's own DATABRICKSHOST+DATABRICKSTOKEN /

databricks auth login. The live config/event evidence that explains why a leak

exists (auto-termination setting, node type, autoscale floor, pool min_idle) comes

from the custom databricks-workspace-mcp control-plane tools — the one MCP

dependency. The SQL produces the number; the workspace MCP turns it into a verified,

single-config-change fix.

Prerequisites

Read access to the billing system tables is the hard dependency and the most common

failure. system.billing.usage requires a metastore-admin grant chain — the

skill detects a missing grant upfront and reports it, rather than failing mid-flow.

• Databricks Premium or Enterprise workspace with Unity Catalog (system tables

are UC-governed; not available on Standard).

• Metastore-admin grant chain on the billing schema, granted by a metastore

admin to the running principal:

• GRANT USE CATALOG ON CATALOG system TO
• GRANT USE SCHEMA ON SCHEMA system.billing TO

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is being cut in the v2 rebuild — no direct replacement. Too vague to map to a pain cluster — cut.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Data Handling

Overview

Implement GDPR compliance, PII masking, data retention, and row-level security in Delta Lake with Unity Catalog. Covers data classification tagging, right-to-deletion workflows, automated retention enforcement, column-level masking functions, and subject access request (SAR) reporting.

Prerequisites

• Unity Catalog enabled
• Understanding of data classification requirements (GDPR, CCPA, HIPAA)
• Admin access for tags and masking functions

Instructions

Step 1: Classify and Tag Data

Use Unity Catalog tags to classify tables and columns for automated compliance enforcement.

-- Tag tables with classification and retention
ALTER TABLE prod_catalog.silver.customers
SET TAGS ('data_classification' = 'PII', 'retention_days' = '730');

ALTER TABLE prod_catalog.silver.orders
SET TAGS ('data_classification' = 'CONFIDENTIAL', 'retention_days' = '365');

ALTER TABLE prod_catalog.gold.metrics
SET TAGS ('data_classification' = 'INTERNAL', 'retention_days' = '1825');

-- Tag PII columns
ALTER TABLE prod_catalog.silver.customers
ALTER COLUMN email SET TAGS ('pii_type' = 'email');

ALTER TABLE prod_catalog.silver.customers
ALTER COLUMN phone SET TAGS ('pii_type' = 'phone');

ALTER TABLE prod_catalog.silver.customers
ALTER COLUMN full_name SET TAGS ('pii_type' = 'name');

Step 2: GDPR Right-to-Deletion

Delete all user data across PII-tagged tables with audit logging.

from pyspark.sql import SparkSession
from datetime import datetime

spark = SparkSession.builder.getOrCreate()

class GDPRHandler:
    """Handle GDPR deletion requests across all PII-tagged tables."""

    def __init__(self, catalog: str):
        self.catalog = catalog

    def find_pii_tables(self) -> list[str]:
        """Find all tables tagged as PII."""
        result = spark.sql(f"""
            SELECT table_catalog, table_schema, table_name
            FROM {self.catalog}.information_schema.table_tags
            WHERE tag_name = 'data_classification' AND tag_value = 'PII'
        """).collect()
        return [f"{r.table_catalog}.{r.table_schema}.{

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-bundle-medic.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Debug Bundle

Current State

!databricks --version 2>/dev/null || echo 'CLI not installed'

!python3 -c "import databricks.sdk; print(f'SDK {databricks.sdk.version}')" 2>/dev/null || echo 'SDK not installed'

Overview

Collect all diagnostic information needed for Databricks support tickets: environment info, cluster state, cluster events, job run details, Spark driver logs, and Delta table history. Produces a redacted tar.gz bundle safe to share with support.

Prerequisites

• Databricks CLI installed and configured
• Access to cluster logs (admin or cluster owner)
• Permission to access job run details

Instructions

Step 1: Create Debug Collection Script

#!/bin/bash
set -euo pipefail
# databricks-debug-bundle.sh [cluster_id] [run_id] [table_name]

BUNDLE_DIR="databricks-debug-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BUNDLE_DIR"

CLUSTER_ID="${1:-}"
RUN_ID="${2:-}"
TABLE_NAME="${3:-}"

echo "=== Databricks Debug Bundle ===" | tee "$BUNDLE_DIR/summary.txt"
echo "Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$BUNDLE_DIR/summary.txt"
echo "Workspace: ${DATABRICKS_HOST:-unset}" >> "$BUNDLE_DIR/summary.txt"

Step 2: Collect Environment Info

{
    echo ""
    echo "--- Environment ---"
    echo "CLI: $(databricks --version 2>&1)"
    echo "SDK: $(pip show databricks-sdk 2>/dev/null | grep Version || echo 'not installed')"
    echo "Python: $(python3 --version 2>&1)"
    echo "OS: $(uname -srm)"
    echo ""
    echo "--- Current User ---"
    databricks current-user me --output json 2>&1 | jq '{userName, active}' || echo "Auth failed"
} >> "$BUNDLE_DIR/summary.txt"

Step 3: Collect Cluster Information

if [ -n "$CLUSTER_ID" ]; then
    echo "" >> "$BUNDLE_DIR/summary.txt"
    echo "--- Cluster: $CLUSTER_ID ---" >> "$BUNDLE_DIR/summary.txt"

    # Full cluster config
    databricks clusters get --cluster-id "$CLUSTER_ID" --output json \
        > "$BUNDLE_DIR/cluster_conf

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-bundle-medic.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Deploy Integration

Overview

Deploy Databricks jobs, DLT pipelines, and ML models using Declarative Automation Bundles (DABs, formerly Asset Bundles). Bundles provide infrastructure-as-code with databricks.yml defining resources, targets (dev/staging/prod), variables, and permissions. The CLI handles validation, deployment, and lifecycle management.

Prerequisites

• Databricks CLI v0.200+ (databricks --version)
• Workspace access with service principal for automated deploys
• databricks.yml bundle configuration at project root

Instructions

Step 1: Initialize a Bundle

# Create from a template
databricks bundle init

# Available templates:
# - default-python: Python notebook project
# - default-sql: SQL project
# - mlops-stacks: Full MLOps template with feature engineering

Step 2: Configure databricks.yml

# databricks.yml — single source of truth for project deployment
bundle:
  name: sales-etl-pipeline

workspace:
  host: ${DATABRICKS_HOST}

variables:
  catalog:
    description: Unity Catalog name
    default: dev_catalog
  alert_email:
    description: Alert notification email
    default: dev@company.com
  warehouse_size:
    default: "2X-Small"

include:
  - resources/*.yml

targets:
  dev:
    default: true
    mode: development
    # dev mode auto-prefixes resources with [username] and enables debug
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev
    variables:
      catalog: dev_catalog

  staging:
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/staging
    variables:
      catalog: staging_catalog
      alert_email: staging-alerts@company.com

  prod:
    mode: production
    # production mode prevents accidental destruction
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/prod
    variables:
      catalog: prod_catalog
      alert_email: oncall@company.com
      warehouse_size: "Medium"

Step 3: Define Resources

# resources/jobs.yml
resources:
  jobs:
    daily_etl:
      name: "daily-etl-${bundle.target}"
      max_concurrent_runs: 1
      timeout_seconds: 14400

      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "UTC"

      email_notif

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-uc-migration-pilot.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Enterprise RBAC

Overview

Implement enterprise access control using Unity Catalog privileges, SCIM-provisioned groups, workspace entitlements, cluster policies, and audit logging. Unity Catalog uses a three-level namespace (catalog.schema.object) with privilege inheritance: granting USAGE on a catalog cascades to schemas. Account-level SCIM syncs groups from your IdP (Okta, Azure AD, Google Workspace).

Prerequisites

• Databricks Premium or Enterprise with Unity Catalog enabled
• Account admin access for SCIM and group management
• Identity Provider supporting SAML 2.0 and SCIM 2.0

Instructions

Step 1: Provision Groups via SCIM API

Sync groups from your IdP at the account level. Max 10,000 users + service principals and 5,000 groups per account.

# Create account-level groups that map to IdP teams
databricks account groups create --json '{
  "displayName": "data-engineers",
  "entitlements": [
    {"value": "workspace-access"},
    {"value": "databricks-sql-access"}
  ]
}'

databricks account groups create --json '{
  "displayName": "data-analysts",
  "entitlements": [
    {"value": "workspace-access"},
    {"value": "databricks-sql-access"}
  ]
}'

databricks account groups create --json '{
  "displayName": "ml-engineers",
  "entitlements": [
    {"value": "workspace-access"},
    {"value": "databricks-sql-access"},
    {"value": "allow-cluster-create"}
  ]
}'

# Assign groups to workspaces
from databricks.sdk import AccountClient

acct = AccountClient()

# Get workspace ID
workspaces = list(acct.workspaces.list())
prod_ws = next(ws for ws in workspaces if ws.workspace_name == "production")

# Assign group to workspace with permissions
acct.workspace_assignment.update(
    workspace_id=prod_ws.workspace_id,
    principal_id=group_id,
    permissions=["USER"],
)

Step 2: Unity Catalog Privilege Hierarchy

-- Privilege model: CATALOG > SCHEMA > TABLE/VIEW/FUNCTION
-- USAGE grants must cascade from catalog to schema

-- Data Engineers: full ETL access
GRANT 

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is being cut in the v2 rebuild — no direct replacement. Documentation cosplay — cut, not absorbed.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Hello World

Overview

Create your first Databricks cluster and notebook via the REST API and Python SDK. Covers single-node dev clusters, SQL warehouses, notebook upload, one-time job runs, and Delta Lake smoke tests.

Prerequisites

• Completed databricks-install-auth setup
• Workspace access with cluster creation permissions
• Valid API credentials in env vars or ~/.databrickscfg

Instructions

Step 1: Create a Single-Node Dev Cluster

# POST /api/2.0/clusters/create
databricks clusters create --json '{
  "cluster_name": "hello-world-dev",
  "spark_version": "14.3.x-scala2.12",
  "node_type_id": "i3.xlarge",
  "autotermination_minutes": 30,
  "num_workers": 0,
  "spark_conf": {
    "spark.databricks.cluster.profile": "singleNode",
    "spark.master": "local[*]"
  },
  "custom_tags": {
    "ResourceClass": "SingleNode"
  }
}'
# Returns: {"cluster_id": "0123-456789-abcde123"}

Or via Python SDK:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.compute import AutoScale

w = WorkspaceClient()

# create_and_wait blocks until cluster reaches RUNNING state
cluster = w.clusters.create_and_wait(
    cluster_name="hello-world-dev",
    spark_version="14.3.x-scala2.12",
    node_type_id="i3.xlarge",
    num_workers=0,
    autotermination_minutes=30,
    spark_conf={
        "spark.databricks.cluster.profile": "singleNode",
        "spark.master": "local[*]",
    },
)
print(f"Cluster ready: {cluster.cluster_id} ({cluster.state})")

Step 2: Create and Upload a Notebook

import base64
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.workspace import ImportFormat, Language

w = WorkspaceClient()

notebook_source = """
# Databricks notebook source
# COMMAND ----------

# Simple DataFrame
data = [("Alice", 28), ("Bob", 35), ("Charlie", 42)]
df = spark.createDataFrame(data, ["name", "age"])
display(df)

# COMMAND ----------

# Write as Delta table
df.write.format(&

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-cluster-forensics + databricks-streaming-guardian.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Incident Runbook

Overview

Rapid incident response for Databricks: triage script, decision tree, immediate actions by error type, communication templates, evidence collection, and postmortem template. Designed for on-call engineers to follow during live incidents.

Severity Levels

Prerequisites

Before this runbook runs, the responder must have:

• Databricks CLI v2 installed and on PATH (databricks --version returns ≥ 0.200.0). Install: pip install databricks-cli or brew install databricks/tap/databricks-cli.
• Authentication to the affected workspace via one of:
• PAT (Personal Access Token) — databricks configure --token, paste a token from the workspace User Settings → Developer → Access Tokens page. Fastest for on-call; OK for short-lived incident sessions.
• OAuth U2M — databricks auth login --host https://.cloud.databricks.com for human-in-the-loop sessions with auto-refresh.
• OAuth M2M — service principal client-credentials grant, for automated incident bots; env vars DATABRICKSCLIENTID + DATABRICKSCLIENTSECRET.
• jq on PATH (used to parse JSON from CLI output and status APIs). Install: apt install jq or brew install jq.
• curl on PATH (used to hit status.databricks.com and internal status pages). Comes standard on every modern Linux/macOS install.
• Read permission on the workspace's job runs + cluster events (granted by default to anyone with workspace access; not all workspaces enable s

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-bundle-medic.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Local Dev Loop

Overview

Set up a fast local development workflow using Databricks Connect v2, Asset Bundles, and VS Code. Databricks Connect lets you run PySpark code locally while executing on a remote Databricks cluster, giving you IDE debugging, fast iteration, and proper test isolation.

Prerequisites

• Completed databricks-install-auth setup
• Python 3.10+ (must match cluster's Python version)
• A running Databricks cluster (DBR 13.3 LTS+)
• VS Code or PyCharm

Instructions

Step 1: Project Structure

my-databricks-project/
├── src/
│   ├── __init__.py
│   ├── pipelines/
│   │   ├── __init__.py
│   │   ├── bronze.py          # Raw ingestion
│   │   ├── silver.py          # Cleansing transforms
│   │   └── gold.py            # Business aggregations
│   └── utils/
│       ├── __init__.py
│       └── helpers.py
├── tests/
│   ├── conftest.py            # Spark fixtures
│   ├── unit/
│   │   └── test_transforms.py # Local Spark tests
│   └── integration/
│       └── test_pipeline.py   # Databricks Connect tests
├── notebooks/
│   └── exploration.py
├── resources/
│   └── daily_etl.yml          # Job resource definitions
├── databricks.yml             # Asset Bundle root config
├── pyproject.toml
└── requirements.txt

Step 2: Install Development Tools

set -euo pipefail

# Create virtual environment
python -m venv .venv && source .venv/bin/activate

# Databricks Connect v2 — version MUST match cluster DBR
pip install "databricks-connect==14.3.*"

# SDK and CLI
pip install databricks-sdk

# Testing
pip install pytest pytest-cov

# Verify Connect installation
databricks-connect test

Step 3: Configure Databricks Connect

Databricks Connect v2 reads from standard SDK auth (env vars, ~/.databrickscfg, or DATABRICKSCLUSTERID).

# Set cluster for Connect to use
export DATABRICKS_HOST="https://adb-1234567890123456.7.azuredatabricks.net"
export DATABRICKS_TOKEN="dapi..."
export DATABRICKS_CLUSTER_ID="0123-456789-abcde123"

# src/utils/spark_session.py
from databricks.connect import DatabricksSession

def get_spark():
    """Get a DatabricksSession — runs Spark on the remote cluster."""
    return DatabricksS

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-uc-migration-pilot.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Migration Deep Dive

Overview

Comprehensive migration strategies for moving to Databricks from Hadoop, Snowflake, Redshift, Synapse, or legacy data warehouses. Covers discovery and assessment, schema conversion, data migration with batching and validation, ETL/pipeline conversion, and cutover planning with rollback procedures.

Prerequisites

• Access to source and target systems
• Databricks workspace with Unity Catalog enabled
• Understanding of current data architecture and dependencies
• Stakeholder alignment on migration timeline

Migration Patterns

Instructions

Step 1: Discovery and Assessment

Inventory all source tables with metadata for migration planning.

from pyspark.sql import SparkSession
from dataclasses import dataclass

spark = SparkSession.builder.getOrCreate()

@dataclass
class TableInventory:
    database: str
    table: str
    table_type: str
    format: str
    row_count: int
    size_mb: float
    columns: int
    partitions: list[str]

def assess_hive_metastore() -> list[TableInventory]:
    """Inventory all Hive Metastore tables for migration planning."""
    inventory = []
    databases = [r.databaseName for r in spark.sql("SHOW DATABASES").collect()]

    for db in databases:
        tables = spark.sql(f"SHOW TABLES IN hive_metastore.{db}").collect()
        for t in tables:
            table_name = f"hive_metastore.{db}.{t.tableName}"
            try:
                detail = spark.sql(f"DESCRIBE DETAIL {table_name}").first()
                schema = spark.table(table_name).schema

                inventory.append(TableInventory(
                    data

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-uc-migration-pilot.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Multi-Environment Setup

Overview

Configure Databricks across dev, staging, and production with isolated workspaces (or catalog-level isolation), per-environment secrets, Asset Bundle targets, and Terraform for workspace provisioning. Each environment gets its own credentials, Unity Catalog namespace, and compute policies.

Prerequisites

• Databricks account with multiple workspaces (or Premium for catalog-level isolation)
• Service principals per environment
• Secret management (Databricks Secret Scopes, AWS Secrets Manager, or GCP Secret Manager)
• CI/CD pipeline (GitHub Actions, Azure DevOps, etc.)

Environment Strategy

Instructions

Step 1: CLI Profiles per Environment

# ~/.databrickscfg
[dev]
host = https://adb-dev-workspace.7.azuredatabricks.net
token = dapi_dev_token

[staging]
host = https://adb-staging-workspace.7.azuredatabricks.net
client_id = staging-sp-client-id
client_secret = staging-sp-secret

[production]
host = https://adb-prod-workspace.7.azuredatabricks.net
client_id = prod-sp-client-id
client_secret = prod-sp-secret

# Use a specific environment
databricks workspace list / --profile staging
databricks clusters list --profile production

Step 2: Asset Bundle Targets

# databricks.yml — single project, multiple targets
bundle:
  name: data-platform

variables:
  catalog:
    description: Unity Catalog for this environment
    default: dev_catalog
  alert_email:
    default: dev@company.com
  cluster_size:
    default: "2X-Small"

targets:
  dev:
    default: true
    mode: development
    workspace:
      host: https://adb-dev-workspace.7.azuredatabricks.net
      root_path: /Users/${workspace.current_user.userName}/.bun

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-streaming-guardian.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Observability

Overview

Monitor Databricks jobs, clusters, SQL warehouses, and costs using system tables in the system catalog. System tables provide queryable observability data: system.lakeflow (job runs), system.billing (costs), system.query (SQL history), system.access (audit logs), and system.compute (cluster metrics). Data updates throughout the day, not real-time.

Prerequisites

• Databricks Premium or Enterprise with Unity Catalog enabled
• Access to system.billing, system.lakeflow, system.query, and system.access schemas
• SQL warehouse for running monitoring queries

Instructions

Step 1: Job Health Monitoring

-- Job success/failure over last 24 hours
SELECT
    COUNT(CASE WHEN result_state = 'SUCCESS' THEN 1 END) AS succeeded,
    COUNT(CASE WHEN result_state = 'FAILED' THEN 1 END) AS failed,
    COUNT(CASE WHEN result_state = 'TIMED_OUT' THEN 1 END) AS timed_out,
    ROUND(100.0 * COUNT(CASE WHEN result_state = 'SUCCESS' THEN 1 END) / COUNT(*), 1) AS success_rate_pct,
    ROUND(AVG(TIMESTAMPDIFF(MINUTE, start_time, end_time)), 1) AS avg_duration_min
FROM system.lakeflow.job_run_timeline
WHERE start_time > current_timestamp() - INTERVAL 24 HOURS;

-- Failed jobs with error details
SELECT job_id, run_name, result_state, start_time, end_time,
       TIMESTAMPDIFF(MINUTE, start_time, end_time) AS duration_min,
       error_message
FROM system.lakeflow.job_run_timeline
WHERE result_state = 'FAILED'
  AND start_time > current_timestamp() - INTERVAL 24 HOURS
ORDER BY start_time DESC;

Step 2: Cluster Utilization and Costs

-- DBU consumption by cluster (last 7 days)
SELECT usage_metadata.cluster_id,
       COALESCE(usage_metadata.cluster_name, 'unnamed') AS cluster_name,
       sku_name,
       SUM(usage_quantity) AS total_dbus,
       ROUND(SUM(usage_quantity * p.pricing.default), 2) AS cost_usd
FROM system.billing.usage u
LEFT JOIN system.billing.list_prices p ON u.sku_name = p.sku_name
WHERE u.usage_date >= current_date() - INTERVAL 7 DAYS
GROUP BY usage_metadata.cluster_id, cluster_name, u.sku_name
ORDER BY cost_usd DESC
LIMIT 20;

Step 3: SQL Warehouse Performance

-

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-cost-leak-hunter + databricks-cluster-forensics.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Performance Tuning

Overview

Optimize Databricks cluster sizing, Spark configuration, and Delta Lake query performance. Covers workload-specific Spark configs, Adaptive Query Execution (AQE), Liquid Clustering, Z-ordering, OPTIMIZE/VACUUM maintenance, query plan analysis, and caching strategies.

Prerequisites

• Access to cluster configuration (admin or cluster owner)
• Understanding of workload type (ETL batch, ML training, streaming, interactive)
• Query history access for identifying slow queries

Instructions

Step 1: Cluster Sizing by Workload

def recommend_cluster(data_size_gb: float, workload: str) -> dict:
    """Recommend cluster config based on data size and workload type."""
    configs = {
        "etl_batch": {"node": "c5.2xlarge", "memory_gb": 16, "multiplier": 1.5},
        "ml_training": {"node": "r5.2xlarge", "memory_gb": 64, "multiplier": 2.0},
        "streaming": {"node": "c5.xlarge", "memory_gb": 8, "multiplier": 1.0},
        "interactive": {"node": "m5.xlarge", "memory_gb": 16, "multiplier": 1.0},
    }
    cfg = configs.get(workload, configs["etl_batch"])
    workers = max(1, int(data_size_gb / cfg["memory_gb"] * cfg["multiplier"]))

    return {
        "node_type_id": cfg["node"],
        "num_workers": workers,
        "autoscale": {"min_workers": max(1, workers // 2), "max_workers&qu

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is being cut in the v2 rebuild — no direct replacement. Checklists move into v2 references/, not a top-level skill.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Production Checklist

Overview

Complete checklist for deploying Databricks jobs and pipelines to production. Covers security hardening, infrastructure validation, code quality gates, job configuration, deployment commands, monitoring setup, and rollback procedures.

Prerequisites

• Staging environment tested and verified
• Production workspace access with service principal
• Unity Catalog configured with prod catalogs
• Monitoring and alerting ready (see databricks-observability)

Instructions

Step 1: Pre-Deployment Security

• [ ] Service principal configured for automated runs (not personal PAT)
• [ ] Secrets in Databricks Secret Scopes (not env vars or hardcoded)
• [ ] Token expiration set (max 90 days)
• [ ] Unity Catalog grants follow least privilege
• [ ] Cluster policies enforced for cost/security guardrails
• [ ] IP access lists configured in Admin Console
• [ ] Audit logging verified via system.access.audit

Step 2: Infrastructure Validation

• [ ] Instance pool created for fast cluster startup
• [ ] Node types validated for workload (compute-optimized for streaming, memory-optimized for ML)
• [ ] Autoscaling configured with sensible min/max workers
• [ ] Spot instances enabled for worker nodes (on-demand for driver)
• [ ] Auto-termination disabled for job clusters (they terminate on completion)

# Verify infrastructure
databricks clusters list-node-types --output json | jq '.[0:5] | .[].node_type_id'
databricks instance-pools list --output json | jq '.[] | {id: .instance_pool_id, name: .instance_pool_name}'

Step 3: Code Quality Gates

• [ ] Unit tests passing locally (pytest tests/unit/)
• [ ] Integration tests passing on staging data
• [ ] No .collect() on large datasets
• [ ] No hardcoded credentials or paths
• [ ] Error handling covers all failure modes
• [ ] Delta Lake best practices: MERGE for upserts, OPTIMIZE scheduled
• [ ] Logging is production-appropriate (structured, no PII)

# Run tests and validate bundle
pytest tests/ -v --tb=short
databricks bundle validate -t prod

Step 4: Job Con

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is being cut in the v2 rebuild — no direct replacement. Architecture doc, not a skill — moves to shared references/.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Reference Architecture

Overview

Production-ready lakehouse architecture with Unity Catalog, Delta Lake, and the medallion pattern. Covers workspace organization, three-level namespace governance, compute strategy, CI/CD with Asset Bundles, and project structure for team collaboration.

Prerequisites

• Databricks workspace with Unity Catalog enabled
• Understanding of medallion architecture (bronze/silver/gold)
• Databricks CLI configured
• Terraform or Asset Bundles for infrastructure

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    UNITY CATALOG                                  │
│                                                                   │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌───────────┐  │
│  │  Bronze    │  │  Silver    │  │   Gold     │  │ ML Models │  │
│  │  Catalog   │─▶│  Catalog   │─▶│  Catalog   │  │ (MLflow)  │  │
│  │  (raw)     │  │  (clean)   │  │  (curated) │  │           │  │
│  └────────────┘  └────────────┘  └────────────┘  └───────────┘  │
│       ▲                                    │                      │
│  ┌────────────┐                   ┌────────────────┐             │
│  │ Auto Loader│                   │ Model Serving  │             │
│  │ Ingestion  │                   │ Endpoints      │             │
│  └────────────┘                   └────────────────┘             │
├─────────────────────────────────────────────────────────────────┤
│  Compute: Job Clusters │ SQL Warehouses │ Instance Pools        │
├─────────────────────────────────────────────────────────────────┤
│  Security: Row Filters │ Column Masks │ Secret Scopes │ SCIM   │
├─────────────────────────────────────────────────────────────────┤
│  CI/CD: Asset Bundles │ GitHub Actions │ dev/staging/prod       │
└─────────────────────────────────────────────────────────────────┘

Project Structure

databricks-platform/
├── src/
│   ├── ingestion/
│   │   ├── bronze_raw_events.py       # Auto Loader streaming
│   │   ├── bronze_api_data.py         # REST API batch ingestion
│   │   └── bronze_file_uploads.py     # Manual file uploads
│   ├── transformation/
│   │   ├── silver_clean_events.py     # Cleansing + dedup
│   │   ├── silver_schema_enforce.py   # Schema validation
│   │   └── silver_scd2.py            # Slowly changing dimensio

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is being cut in the v2 rebuild — no direct replacement. SDK ergonomics is library documentation, not a skill.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks SDK Patterns

Overview

Production-ready patterns for the Databricks Python SDK (databricks-sdk). Covers singleton client initialization, typed error handling, cluster lifecycle management, type-safe job construction, and pagination. Uses real SDK exception classes and API shapes.

Prerequisites

• databricks-sdk>=0.20.0 installed
• Authentication configured (see databricks-install-auth)
• Python 3.10+

Instructions

Step 1: Singleton Client with Profile Support

Each WorkspaceClient holds an HTTP session and re-authenticates. Cache instances.

from databricks.sdk import WorkspaceClient, AccountClient
from functools import lru_cache

@lru_cache(maxsize=4)
def get_client(profile: str = "DEFAULT") -> WorkspaceClient:
    """Cached WorkspaceClient — one per profile."""
    return WorkspaceClient(profile=profile)

@lru_cache(maxsize=1)
def get_account_client() -> AccountClient:
    """Account-level client for multi-workspace operations."""
    return AccountClient(
        host="https://accounts.cloud.databricks.com",
        account_id="00000000-0000-0000-0000-000000000000",
    )

# Usage
w = get_client()
w_prod = get_client("production")

Step 2: Structured Error Handling

The SDK raises typed exceptions from databricks.sdk.errors. Distinguish transient (retryable) from permanent failures.

from dataclasses import dataclass
from typing import TypeVar, Generic, Optional, Callable
from databricks.sdk.errors import (
    NotFound,
    PermissionDenied,
    TooManyRequests,
    TemporarilyUnavailable,
    ResourceConflict,
    InvalidParameterValue,
    ResourceAlreadyExists,
)

T = TypeVar("T")

@dataclass
class Result(Generic[T]):
    value: Optional[T] = None
    error: Optional[str] = None
    retryable: bool = False

    @property
    def ok(self) -> bool:
        return self.error is None

def safe_call(func: Callable, *args, **kwargs) -> Result:
    """Execute a Databricks API call with structured error classification."""
    try:
        return Result(value=func(*args, **kwargs))
    except NotFound as e:
        return Result(error=f"Not found: {e.m

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-uc-migration-pilot + databricks-bundle-medic (identity/secrets ops).

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Security Basics

Overview

Implement Databricks security: secret scopes for credential storage, token rotation, least-privilege access via Unity Catalog grants, and security auditing via system tables. Secrets API uses PUT /api/2.0/secrets/put and values are automatically redacted in notebook output.

Prerequisites

• Databricks CLI configured
• Workspace admin access (for secret scope creation)
• Unity Catalog enabled

Instructions

Step 1: Create and Manage Secret Scopes

# Create a Databricks-backed secret scope
databricks secrets create-scope my-app-secrets

# Create Azure Key Vault-backed scope (Azure only)
databricks secrets create-scope azure-kv \
  --scope-backend-type AZURE_KEYVAULT \
  --resource-id "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault>" \
  --dns-name "https://<vault>.vault.azure.net/"

# List all scopes
databricks secrets list-scopes

Step 2: Store and Access Secrets

# Store a secret (prompts for value interactively)
databricks secrets put-secret my-app-secrets db-password

# Store from CLI argument
databricks secrets put-secret my-app-secrets api-key --string-value "sk_live_abc123"

# List secrets (values always hidden)
databricks secrets list-secrets my-app-secrets

# Access secrets in notebooks and jobs — values auto-redacted in output
db_password = dbutils.secrets.get(scope="my-app-secrets", key="db-password")
api_key = dbutils.secrets.get(scope="my-app-secrets", key="api-key")

# Printing shows [REDACTED] — Databricks prevents accidental exposure
print(f"Password: {db_password}")  # Output: Password: [REDACTED]

# Use in JDBC connections
jdbc_url = f"jdbc:postgresql://host:5432/db?user=app&password={db_password}"
df = spark.read.format("jdbc").option("url", jdbc_url).load()

Step 3: Secret Scope Access Control

# Grant READ to a user
databricks secrets put-acl my-app-secrets user@company.com READ

# Grant MANAGE to a group (full control)
databricks secrets put-acl my-app-secrets data-engineers MANAGE

# List ACLs for a scope
databricks secrets

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-cluster-forensics (DBR-upgrade triage folds into cluster lifecycle).

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Upgrade & Migration

Overview

Upgrade Databricks Runtime versions and migrate from Hive Metastore to Unity Catalog. Covers version compatibility, deprecated config removal, table migration via SYNC/CTAS, API endpoint updates, and Delta protocol upgrades.

Prerequisites

• Admin access to workspace
• Test environment (dev/staging) for validation before prod
• Inventory of current workloads and dependencies

Instructions

Step 1: Runtime Version Upgrade

Version Compatibility Matrix

Automated Upgrade Script

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

def plan_cluster_upgrade(
    cluster_id: str,
    target_version: str = "14.3.x-scala2.12",
    dry_run: bool = True,
) -> dict:
    """Plan and optionally execute a DBR version upgrade."""
    cluster = w.clusters.get(cluster_id)
    plan = {
        "cluster_id": cluster_id,
        "cluster_name": cluster.cluster_name,
        "current_version": cluster.spark_version,
        "target_version": target_version,
        "removals": [],
        "warnings": [],
    }

    # Check for deprecated Spark configs
    deprecated = {
        "spark.databricks.delta.preview.enabled": "GA in 13.x+",
        "spark.sql.legacy.createHiveTableByDefault": "Removed in 14.x+",
        "spark.databricks.passthrough.enabled": "Removed in 15.x+",
        "spark.sql.legacy.allowNonEmptyLocationInCTAS": "Removed in 14.x+",
    }

    for key, reason in deprecated.items():
        if cluster.spark_conf and key in cluster.spark_conf:
            plan["removals"].append({"config": key, "reason": reason})

    # Check Python version compatibility
    if "13." in target_v

> [!WARNING]

> DEPRECATED — to be removed in databricks-pack@2.0.0.

>

> This v1 skill is being cut in the v2 rebuild — no direct replacement. No corresponding pain in the v2 catalog.

> See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Webhooks & Events

Overview

Configure notifications and event-driven workflows for Databricks jobs. Covers notification destinations (Slack, Teams, PagerDuty, email, generic webhooks), job lifecycle events, SQL alerts with automated triggers, and system table queries for event auditing.

Prerequisites

• Databricks workspace admin access (for notification destinations)
• Webhook endpoint URL (Slack incoming webhook, Teams connector, etc.)
• Job permissions for notification configuration

Instructions

Step 1: Create Notification Destinations

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.settings import (
    CreateNotificationDestinationRequest,
    SlackConfig, EmailConfig, GenericWebhookConfig,
)

w = WorkspaceClient()

# Slack destination
slack = w.notification_destinations.create(
    display_name="Engineering Slack",
    config=SlackConfig(url="https://hooks.slack.com/services/T00/B00/xxxx"),
)

# Email destination
email = w.notification_destinations.create(
    display_name="Oncall Email",
    config=EmailConfig(addresses=["oncall@company.com", "data-team@company.com"]),
)

# Generic webhook (PagerDuty, custom endpoint)
pagerduty = w.notification_destinations.create(
    display_name="PagerDuty",
    config=GenericWebhookConfig(
        url="https://events.pagerduty.com/integration/YOUR_KEY/enqueue",
    ),
)

print(f"Slack: {slack.id}, Email: {email.id}, PD: {pagerduty.id}")

Step 2: Attach Notifications to Jobs

from databricks.sdk.service.jobs import (
    JobEmailNotifications, WebhookNotifications, Webhook,
)

# Update existing job with notifications
w.jobs.update(
    job_id=123,
    new_settings={
        "email_notifications": JobEmailNotifications(
            on_start=["team@company.com"],
            on_success=["team@company.com"],
            on_failure=["oncall@company.com", "team@company.com"],
            no_alert_for_skipped_runs=True,
        ),
        "webhook_notifications": WebhookNotifications(
            on_start=[Webhook(id=slack.id)],
            on_success=[Webhook(id=slack.id)],
            on_failure=[Webhook(id=slack.id), Webhook(id=pagerduty.id)],
        ),
    },
)

Or declara