databricks-local-dev-loop
Configure Databricks local development with dbx, Databricks Connect, and IDE. Use when setting up a local dev environment, configuring test workflows, or establishing a fast iteration cycle with Databricks. Trigger with phrases like "databricks dev setup", "databricks local", "databricks IDE", "develop with databricks", "databricks connect".
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 2.0k
- Forks
- 233
01 前言 Front Matter
- name
- databricks-local-dev-loop
- author
- Jeremy Longshore <jeremy@intentsolutions.io>
- license
- MIT
- version
- 1.0.0
- description
- Configure Databricks local development with dbx, Databricks Connect, and IDE. Use when setting up a local dev environment, configuring test workflows, or establishing a fast iteration cycle with Databricks. Trigger with phrases like "databricks dev setup", "databricks local", "databricks IDE", "develop with databricks", "databricks connect".
- allowed-tools
- Read, Write, Edit, Bash(pip:*), Bash(databricks:*), Bash(dbx:*), Grep
03 SKILL.md 正文
Databricks Local Dev Loop
Overview
Set up a fast, reproducible local development workflow for Databricks.
Prerequisites
- Completed
databricks-install-authsetup - Python 3.8+ with pip
- VS Code or PyCharm IDE
- Access to a running cluster
Instructions
Step 1: Project Structure
my-databricks-project/
├── src/
│ ├── __init__.py
│ ├── pipelines/
│ │ ├── __init__.py
│ │ ├── bronze.py # Raw data ingestion
│ │ ├── silver.py # Data cleansing
│ │ └── gold.py # Business aggregations
│ └── utils/
│ ├── __init__.py
│ └── helpers.py
├── tests/
│ ├── __init__.py
│ ├── unit/
│ │ └── test_helpers.py
│ └── integration/
│ └── test_pipelines.py
├── notebooks/ # Databricks notebooks
│ └── exploration.py
├── resources/ # Asset Bundle configs
│ └── jobs.yml
├── databricks.yml # Asset Bundle project config
├── .env.local # Local secrets (git-ignored)
├── .env.example # Template for team
├── pyproject.toml
└── requirements.txt
Step 2: Install Development Tools
# Install Databricks SDK and CLI
pip install databricks-sdk databricks-cli
# Install dbx for deployment
pip install dbx
# Install Databricks Connect v2 (for local Spark)
pip install databricks-connect==14.3.*
# Install testing tools
pip install pytest pytest-cov
Step 3: Configure Databricks Connect
# Configure Databricks Connect for local development
databricks-connect configure
# Or set environment variables
export DATABRICKS_HOST="https://adb-1234567890.1.azuredatabricks.net"
export DATABRICKS_TOKEN="dapi..."
export DATABRICKS_CLUSTER_ID="1234-567890-abcde123"
Step 4: Create databricks.yml (Asset Bundle)
# databricks.yml
bundle:
name: my-databricks-project
workspace:
host: ${DATABRICKS_HOST}
variables:
catalog:
description: Unity Catalog name
default: main
schema:
description: Schema name
default: default
targets:
dev:
default: true
mode: development
workspace:
root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev
staging:
mode: development
workspace:
root_path: /Shared/.bundle/${bundle.name}/staging
prod:
mode: production
workspace:
root_path: /Shared/.bundle/${bundle.name}/prod
Step 5: Local Testing Setup
# tests/conftest.py
import pytest
from pyspark.sql import SparkSession
@pytest.fixture(scope="session")
def spark():
"""Create local SparkSession for unit tests."""
return SparkSession.builder \
.master("local[*]") \
.appName("unit-tests") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.getOrCreate()
@pytest.fixture(scope="session")
def dbx_spark():
"""Connect to Databricks cluster for integration tests."""
from databricks.connect import DatabricksSession
return DatabricksSession.builder.getOrCreate()
Step 6: VS Code Configuration
// .vscode/settings.json
{
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": ["tests"],
"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
"editor.formatOnSave": true,
"[python]": {
"editor.defaultFormatter": "ms-python.black-formatter"
},
"databricks.python.envFile": "${workspaceFolder}/.env.local"
}
// .vscode/launch.json
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File (Databricks Connect)",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"env": {
"DATABRICKS_HOST": "${env:DATABRICKS_HOST}",
"DATABRICKS_TOKEN": "${env:DATABRICKS_TOKEN}",
"DATABRICKS_CLUSTER_ID": "${env:DATABRICKS_CLUSTER_ID}"
}
}
]
}
Output
- Working local development environment
- Databricks Connect configured for remote execution
- Unit and integration test setup
- VS Code/PyCharm integration ready
Error Handling
| Error | Cause | Solution |
|---|---|---|
Cluster not running |
Auto-terminated | Start cluster first |
Version mismatch |
DBR vs Connect version | Match databricks-connect version to DBR |
Module not found |
Missing local install | Run pip install -e . |
Connection timeout |
Network/firewall | Check VPN and firewall rules |
SparkSession already exists |
Multiple sessions | Use getOrCreate() pattern |
Examples
Run Tests Locally
# Unit tests (local Spark)
pytest tests/unit/ -v
# Integration tests (Databricks Connect)
pytest tests/integration/ -v --tb=short
# With coverage
pytest tests/ --cov=src --cov-report=html
Deploy with Asset Bundles
# Validate bundle
databricks bundle validate
# Deploy to dev
databricks bundle deploy -t dev
# Run job
databricks bundle run -t dev my-job
Interactive Development
# src/pipelines/bronze.py
from pyspark.sql import SparkSession, DataFrame
def ingest_raw_data(spark: SparkSession, source_path: str) -> DataFrame:
"""Ingest raw data from source."""
return spark.read.format("json").load(source_path)
if __name__ == "__main__":
# Works locally with Databricks Connect
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
df = ingest_raw_data(spark, "/mnt/raw/events")
df.show()
Hot Reload with dbx
# Watch for changes and sync
dbx sync --watch
# Or use Asset Bundles
databricks bundle sync -t dev --watch
Resources
Next Steps
See databricks-sdk-patterns for production-ready code patterns.
04 同仓库更多 Skills
theme-factory
Toolkit for styling artifacts with a theme. These artifacts can be slides, docs, reportings, HTML landing pages, etc. There are 10 pre-set themes with colors/fonts that you can apply to any artifact that has been creating, or can generate a new theme on-the-fly.
web-artifacts-builder
Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn/ui). Use for complex artifacts requiring state management, routing, or shadcn/ui components - not for simple single-file HTML/JSX artifacts.
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
neuropixels-analysis
Neuropixels neural recording analysis. Load SpikeGLX/OpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen/IBL curation, AI-assisted visual analysis, for Neuropixels 1.0/2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.
skill-creator
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.
notebooklm
Use this skill to query your Google NotebookLM notebooks directly from Claude Code for source-grounded, citation-backed answers from Gemini. Browser automation, library management, persistent auth. Drastically reduced hallucinations through document-only responses.