AGENT SKILL

databricks-multi-env-setup

Configure Databricks across development, staging, and production environments. Use when setting up multi-environment deployments, configuring per-environment secrets, or implementing environment-specific Databricks configurations. Trigger with phrases like "databricks environments", "databricks staging", "databricks dev prod", "databricks environment setup", "databricks config by env".

使用场景运维与部署
License 许可
MIT
最近更新
2026-11-15
Stars
★ 2.0k
Forks
233

01 前言 Front Matter

name
databricks-multi-env-setup
author
Jeremy Longshore <jeremy@intentsolutions.io>
license
MIT
version
1.0.0
description
Configure Databricks across development, staging, and production environments. Use when setting up multi-environment deployments, configuring per-environment secrets, or implementing environment-specific Databricks configurations. Trigger with phrases like "databricks environments", "databricks staging", "databricks dev prod", "databricks environment setup", "databricks config by env".
allowed-tools
Read, Write, Edit, Bash(databricks:*), Bash(terraform:*)

03 SKILL.md 正文

Databricks Multi-Environment Setup

Overview

Configure Databricks across development, staging, and production environments.

Prerequisites

  • Multiple Databricks workspaces (or single workspace with isolation)
  • Secret management solution (Databricks Secrets, Azure Key Vault, etc.)
  • CI/CD pipeline configured
  • Unity Catalog for cross-workspace governance

Environment Strategy

Environment Purpose Workspace Data Permissions
Development Local dev, experimentation Shared dev workspace Sample/synthetic Broad access
Staging Integration testing, UAT Dedicated staging Prod clone (masked) Team access
Production Live workloads Dedicated prod Real data Restricted

Instructions

Step 1: Asset Bundle Environment Configuration

# databricks.yml
bundle:
  name: data-platform

variables:
  catalog:
    description: Unity Catalog name
    default: dev_catalog
  schema_prefix:
    description: Schema prefix for isolation
    default: dev
  warehouse_size:
    description: SQL Warehouse size
    default: "2X-Small"
  cluster_size:
    description: Default cluster workers
    default: 1

include:
  - resources/*.yml

workspace:
  host: ${DATABRICKS_HOST}

targets:
  # Development - personal workspaces
  dev:
    default: true
    mode: development
    variables:
      catalog: dev_catalog
      schema_prefix: "${workspace.current_user.short_name}"
      warehouse_size: "2X-Small"
      cluster_size: 1
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev

  # Staging - shared test environment
  staging:
    mode: development
    variables:
      catalog: staging_catalog
      schema_prefix: staging
      warehouse_size: "Small"
      cluster_size: 2
    workspace:
      host: ${DATABRICKS_HOST_STAGING}
      root_path: /Shared/.bundle/${bundle.name}/staging
    run_as:
      service_principal_name: ${STAGING_SERVICE_PRINCIPAL}
    permissions:
      - level: CAN_VIEW
        group_name: developers
      - level: CAN_MANAGE_RUN
        group_name: data-engineers

  # Production - locked down
  prod:
    mode: production
    variables:
      catalog: prod_catalog
      schema_prefix: prod
      warehouse_size: "Medium"
      cluster_size: 4
    workspace:
      host: ${DATABRICKS_HOST_PROD}
      root_path: /Shared/.bundle/${bundle.name}/prod
    run_as:
      service_principal_name: ${PROD_SERVICE_PRINCIPAL}
    permissions:
      - level: CAN_VIEW
        group_name: data-consumers
      - level: CAN_MANAGE_RUN
        group_name: data-engineers
      - level: CAN_MANAGE
        service_principal_name: ${PROD_SERVICE_PRINCIPAL}

Step 2: Environment-Specific Resources

# resources/jobs.yml
resources:
  jobs:
    etl_pipeline:
      name: "${bundle.name}-etl-${bundle.target}"

      # Environment-specific schedule
      schedule:
        quartz_cron_expression: >-
          ${bundle.target == "prod" ? "0 0 6 * * ?" :
            bundle.target == "staging" ? "0 0 8 * * ?" : null}
        timezone_id: "America/New_York"
        pause_status: ${bundle.target == "dev" ? "PAUSED" : "UNPAUSED"}

      # Environment-specific notifications
      email_notifications:
        on_failure: ${bundle.target == "prod" ?
          ["oncall@company.com", "pagerduty@company.pagerduty.com"] :
          ["team@company.com"]}

      # Environment-specific cluster sizing
      job_clusters:
        - job_cluster_key: etl_cluster
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            node_type_id: >-
              ${bundle.target == "prod" ? "Standard_DS4_v2" : "Standard_DS3_v2"}
            num_workers: ${var.cluster_size}
            autoscale:
              min_workers: ${bundle.target == "prod" ? 2 : 1}
              max_workers: ${bundle.target == "prod" ? 10 : 4}

            # Spot instances for non-prod
            azure_attributes:
              availability: >-
                ${bundle.target == "prod" ? "ON_DEMAND_AZURE" : "SPOT_AZURE"}
              first_on_demand: 1

Step 3: Unity Catalog Cross-Environment Setup

-- Create environment-specific catalogs
CREATE CATALOG IF NOT EXISTS dev_catalog;
CREATE CATALOG IF NOT EXISTS staging_catalog;
CREATE CATALOG IF NOT EXISTS prod_catalog;

-- Grant cross-environment read access for data lineage
GRANT USAGE ON CATALOG prod_catalog TO `staging-service-principal`;
GRANT SELECT ON CATALOG prod_catalog TO `staging-service-principal`;

-- Set up data sharing between environments
CREATE SHARE IF NOT EXISTS prod_to_staging;
ALTER SHARE prod_to_staging ADD SCHEMA prod_catalog.reference;

-- Create recipient for staging workspace
CREATE RECIPIENT IF NOT EXISTS staging_workspace
  USING IDENTITY ('staging-workspace-identity');

GRANT SELECT ON SHARE prod_to_staging TO RECIPIENT staging_workspace;

Step 4: Secret Management by Environment

# src/config/secrets.py
from databricks.sdk import WorkspaceClient
import os

class EnvironmentSecrets:
    """Environment-aware secret management."""

    def __init__(self, environment: str = None):
        self.environment = environment or os.getenv("ENVIRONMENT", "dev")
        self.w = WorkspaceClient()
        self._secret_scope = f"{self.environment}-secrets"

    def get_secret(self, key: str) -> str:
        """Get secret for current environment."""
        # In notebooks, use dbutils
        # return dbutils.secrets.get(scope=self._secret_scope, key=key)

        # Via API (for testing)
        return self.w.secrets.get_secret(
            scope=self._secret_scope,
            key=key
        ).value

    def get_database_url(self) -> str:
        """Get environment-specific database URL."""
        return self.get_secret("database_url")

    def get_api_key(self, service: str) -> str:
        """Get API key for service."""
        return self.get_secret(f"{service}_api_key")

# Usage in notebooks
# secrets = EnvironmentSecrets()
# db_url = secrets.get_database_url()

Step 5: Environment Detection

# src/config/environment.py
import os
from dataclasses import dataclass
from typing import Optional

@dataclass
class EnvironmentConfig:
    """Environment-specific configuration."""
    name: str
    catalog: str
    schema_prefix: str
    is_production: bool
    debug_enabled: bool
    max_cluster_size: int

def detect_environment() -> EnvironmentConfig:
    """Detect current environment from context."""
    # From environment variable (set by Asset Bundles)
    env = os.getenv("ENVIRONMENT", "dev")

    # Or from Databricks tags
    # spark.conf.get("spark.databricks.tags.Environment")

    configs = {
        "dev": EnvironmentConfig(
            name="dev",
            catalog="dev_catalog",
            schema_prefix="dev",
            is_production=False,
            debug_enabled=True,
            max_cluster_size=4,
        ),
        "staging": EnvironmentConfig(
            name="staging",
            catalog="staging_catalog",
            schema_prefix="staging",
            is_production=False,
            debug_enabled=True,
            max_cluster_size=8,
        ),
        "prod": EnvironmentConfig(
            name="prod",
            catalog="prod_catalog",
            schema_prefix="prod",
            is_production=True,
            debug_enabled=False,
            max_cluster_size=20,
        ),
    }

    return configs.get(env, configs["dev"])

# Usage
env = detect_environment()
full_table_name = f"{env.catalog}.{env.schema_prefix}_sales.orders"

Step 6: Environment Promotion Pipeline

# scripts/promote_to_prod.py
from databricks.sdk import WorkspaceClient
import subprocess

def promote_to_production(
    version_tag: str,
    dry_run: bool = True,
) -> dict:
    """
    Promote code from staging to production.

    Steps:
    1. Verify staging tests passed
    2. Tag release in git
    3. Deploy to production
    4. Run smoke tests
    5. Enable schedules
    """
    results = {"steps": []}

    # 1. Verify staging tests
    print("Verifying staging tests...")
    staging_result = subprocess.run(
        ["databricks", "bundle", "run", "-t", "staging", "integration-tests"],
        capture_output=True
    )
    if staging_result.returncode != 0:
        raise Exception("Staging tests failed")
    results["steps"].append({"stage": "verify_staging", "status": "passed"})

    # 2. Tag release
    print(f"Tagging release {version_tag}...")
    if not dry_run:
        subprocess.run(["git", "tag", version_tag])
        subprocess.run(["git", "push", "origin", version_tag])
    results["steps"].append({"stage": "tag_release", "status": "done" if not dry_run else "skipped"})

    # 3. Deploy to production
    print("Deploying to production...")
    if not dry_run:
        subprocess.run(["databricks", "bundle", "deploy", "-t", "prod"])
    results["steps"].append({"stage": "deploy_prod", "status": "done" if not dry_run else "skipped"})

    # 4. Run smoke tests
    print("Running smoke tests...")
    if not dry_run:
        subprocess.run(["databricks", "bundle", "run", "-t", "prod", "smoke-tests"])
    results["steps"].append({"stage": "smoke_tests", "status": "done" if not dry_run else "skipped"})

    return results

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--version", required=True)
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args()

    result = promote_to_production(args.version, args.dry_run)
    print(result)

Output

  • Multi-environment Asset Bundle configuration
  • Environment-specific job settings
  • Cross-environment Unity Catalog setup
  • Secret management by environment
  • Promotion pipeline

Error Handling

Issue Cause Solution
Wrong environment Missing env var Check ENVIRONMENT variable
Secret not found Wrong scope Verify scope name matches environment
Permission denied Missing grants Add Unity Catalog grants
Config mismatch Target override issue Check bundle target syntax

Examples

Quick Environment Check

env = detect_environment()
print(f"Running in {env.name} environment")
print(f"Using catalog: {env.catalog}")
print(f"Production mode: {env.is_production}")

Terraform Multi-Workspace

# infrastructure/terraform/main.tf
locals {
  environments = {
    dev = {
      workspace_name = "data-platform-dev"
      sku           = "premium"
    }
    staging = {
      workspace_name = "data-platform-staging"
      sku           = "premium"
    }
    prod = {
      workspace_name = "data-platform-prod"
      sku           = "premium"
    }
  }
}

resource "azurerm_databricks_workspace" "workspace" {
  for_each = local.environments

  name                = each.value.workspace_name
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  sku                 = each.value.sku

  tags = {
    environment = each.key
    managed_by  = "terraform"
  }
}

Resources

Next Steps

For observability setup, see databricks-observability.

04 同仓库更多 Skills

Dicklesworthstone
Dicklesworthstone/pi_agent_rust

theme-factory

Toolkit for styling artifacts with a theme. These artifacts can be slides, docs, reportings, HTML landing pages, etc. There are 10 pre-set themes with colors/fonts that you can apply to any artifact that has been creating, or can generate a new theme on-the-fly.

视觉设计
★ 2.0k
Dicklesworthstone
Dicklesworthstone/pi_agent_rust

web-artifacts-builder

Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn/ui). Use for complex artifacts requiring state management, routing, or shadcn/ui components - not for simple single-file HTML/JSX artifacts.

前端开发
★ 2.0k
Dicklesworthstone
Dicklesworthstone/pi_agent_rust

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

测试与质量前端开发
★ 2.0k
Dicklesworthstone
Dicklesworthstone/pi_agent_rust

neuropixels-analysis

Neuropixels neural recording analysis. Load SpikeGLX/OpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen/IBL curation, AI-assisted visual analysis, for Neuropixels 1.0/2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.

★ 2.0k
Dicklesworthstone
Dicklesworthstone/pi_agent_rust

skill-creator

Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.

Skill与智能体开发
★ 2.0k
Dicklesworthstone
Dicklesworthstone/pi_agent_rust

notebooklm

Use this skill to query your Google NotebookLM notebooks directly from Claude Code for source-grounded, citation-backed answers from Gemini. Browser automation, library management, persistent auth. Drastically reduced hallucinations through document-only responses.

Skill与智能体开发
★ 2.0k