The Databricks GitHub map: repos you need to know
The Databricks ecosystem moves fast. Every Data + AI Summit brings an avalanche of announcements, and between summits new repos keep popping up that sometimes solve problems you’ve been fighting for months. The thing is, if you’re not actively watching, you miss them — and you end up reinventing the wheel or using deprecated tools.
How do you deal with this without going crazy? You don’t need to follow everything. What you do need is to know where to look and what’s worth it. The good news is that Databricks builds in the open — their GitHub repos are the most direct window into what’s coming. Spark Declarative Pipelines, Lakebase, Omnigent — all of it showed up on GitHub first, before any corporate blog or keynote. Following the right repos, following the people who give back to the community, and trying the tools in your own environment is the most practical way to stay current without depending on someone else telling you about it.
Databricks has over 500 public repos, scattered across four organizations of its own plus several open-source projects that live in independent foundations. Each org has a different purpose and a different level of support, and if you don’t know which is which, you can end up running something in production that’s an experiment from an internal hackathon. If you’ve ever searched for something and landed on a deprecated repo from 2019, this post is for you.
I put together this curated guide organizing everything by topic and maturity level. You can read it straight through as a blog post — it has context, reviews and demos for each tool — or use it as a reference when you need to find a repo for a specific problem. It’s not an exhaustive list: it’s what I consider most useful if you work with Databricks day to day.
- Databricks has four orgs of its own on GitHub — all maintained by employees, but with different roles:
databricks(product),databrickslabs(experimental),databricks-solutions(field toolkits),databricks-industry-solutions(end-to-end industry solutions). Plus OSS projects in independent foundations. - For development with coding agents (Claude Code, Cursor, Copilot): the AI Dev Kit is the most complete repo, with 75+ tools.
- For Databricks Apps: AppKit is the new React/Node.js SDK with plugins for Genie, Lakebase, SQL and Files.
- For Data Engineering:
dbt-databricks,terraform-provider-databricks,mlops-stacksand the bundle examples are still essential. - Community MVPs (MrPowers, Jacek Laskowski) maintain tools that many teams use in production without knowing they’re community-driven.
Color guide
Throughout the post, each card has a colored border indicating which organization it comes from:
databricks — Official product
databrickslabs — Experimental
databricks-solutions — Field Engineering
databricks-industry-solutions — Industry solutions
OSS foundations — Linux Foundation / Apache
Community — MVPs and third parties
The Databricks organizations on GitHub
They all belong to Databricks, but they’re not the same thing. All four organizations are owned by Databricks and maintained by employees — none of them is an external community. The difference lies in who inside Databricks maintains them and with what level of commitment:
databricks Official
The product team. Official SDKs, CLI, drivers, connectors, educational material. Official support with triaged issues, versioning and SLAs.
databrickslabs Labs
Employees, experimental projects. Prototypes, incubated tools, internal side projects. Best-effort, no guarantees. Some graduate into the product.
databricks-solutions Solutions
The Field Engineering team. Horizontal toolkits, app templates, tool integrations (AI Dev Kit, AppKit cookbook, Lakebase dev kit). Reference code for developers and platform engineers.
databricks-industry-solutions Industry
213 repos of end-to-end solutions by industry. Complete notebooks, end-to-end pipelines, sample data. “Softer” than the other orgs — there are no SDKs or CLIs here, there are business solutions ready to clone and adapt: fraud detection for banking, demand forecasting for retail, medical imaging for healthcare. It’s what Databricks calls Solution Accelerators.
So what’s the difference between the four? databricks is the product. databrickslabs is experiments that may disappear. databricks-solutions is horizontal tooling for developers. databricks-industry-solutions is vertical business solutions — the highest level of abstraction, ready to show a stakeholder.
On top of that, there’s a fifth category: projects that Databricks created or championed but that live in independent organizations under the Linux Foundation or Apache (delta-io, mlflow, unitycatalog, apache/spark). These have community governance and are the most stable of all.
Several historical repos in databricks are archived or deprecated: koalas (now pyspark.pandas), spark-csv (merged into Spark core), the Python databricks-cli (replaced by the Go CLI). If a README says “deprecated”, believe it.
Developer Experience Official Solutions
This is the category that grew the most over the last year. Databricks bet big on letting you develop from your favorite editor, with coding agents, and with a centralized portal.
AI Dev Kit
The star repo of databricks-solutions. Certified as a Gold Project by Databricks.
It’s not a library — it’s a context kit that teaches your coding agent how Databricks works. 75+ executable tools plus patterns curated by Databricks field engineers. Your agent “knows” how to build Spark Pipelines, Jobs, Unity Catalog, dashboards, Model Serving, MLflow and Apps without you having to explain the conventions.
Supports Claude Code, Cursor, Codex, Copilot, Gemini CLI and Windsurf.
vs. copying prompts by hand: the AI Dev Kit is executable skills, not just documentation text. The agent can do things in your workspace, not just suggest code.
# Install into an existing project (Mac/Linux)
bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh)
# Or directly as an MCP server for Claude Code
claude mcp add databricks-ai-dev-kit \
-- uvx databricks-ai-dev-kit@latestRequires uv and the Databricks CLI installed. Installation is project-level — you have to run the agent from the directory where you installed it.
databricks-agent-skills
The lightest-weight approach: skills that install as Markdown files into your coding agent. It auto-detects which agent you have installed and writes the SKILL.md files to the right location. Includes databricks-setup and databricks-doctor for diagnostics.
databricks aitools installcli
The modern Go CLI that replaced the old Python CLI. It’s the core of Databricks Asset Bundles (DABs) — if you use DABs, you already have it installed.
databricks-sdk-py
The official SDK. Replaces the old pattern of requests plus a manual token. Typed, with retry logic, and covering the entire REST API.
databricks-vscode
For local development connected to remote clusters. Includes notebook execution, debugging, and catalog autocompletion.
DevHub
Not a repo, but it’s the portal that ties everything together. Templates, AppKit documentation, setup guides. If you’re going to build anything on Databricks, start here.
Databricks Apps Official Solutions
Databricks Apps is the platform for deploying web applications inside the workspace, with authentication, governance and networking taken care of.
AppKit
The only TypeScript SDK designed for Databricks Apps. It gives you a server with production-ready plugins: Analytics (SQL queries with caching), Genie (conversational AI/BI), Files (Unity Catalog Volumes) and Lakebase (returns a standard pg.Pool compatible with Prisma, Drizzle, TypeORM).
vs. calling the REST API directly: AppKit handles OAuth authentication (on-behalf-of user), retry logic, caching, OpenTelemetry telemetry and remote hot reload. Without AppKit, you have to implement all of that by hand.
import { createApp, analytics, lakebase, genie }
from '@databricks/appkit';
createApp({
plugins: [
analytics(), // SQL with caching and automatic type-gen
lakebase(), // Native OLTP Postgres (standard pg.Pool)
genie(), // Natural language questions
],
async onPluginsReady(appkit) {
await setupRoutes(appkit);
},
}).catch(console.error);# Scaffold a new app
npx create-databricks-app my-appTypeScript/Node.js only — there’s no Python version. Currently at v0, API still evolving.
databricks-apps-cookbook
Ready-to-copy-and-paste snippets for your Databricks App.
apx
Toolkit for building apps on Databricks.
app-templates
Official Databricks templates to get started fast.
claude-databricks-app-template
Template for a native Databricks App with Claude AI.
caspers
Complete simulated business demo: streaming, AI agents, apps — all deployable.
Databricks Lakebase Solutions
Lakebase is Databricks’ serverless Postgres that writes directly to Delta/Iceberg (the LTAP promise). The repo ecosystem around it is growing fast.
Lakebase App Dev Kit
Opinionated git + Lakebase workflows: branch pairing, schema diff, PR flow. Includes an MCP server and a development methodology (Spec-First Test-Driven Development). Supports Claude Code, Cursor and Databricks Genie Code.
bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/lakebase-app-dev-kit/main/install.sh)lakebase-fastapi-app
FastAPI app with Lakebase as the backend.
brickhouse-brands-demo
Full-stack demo: React + FastAPI + Lakebase. Deployable inventory management.
Agents & AI Official
lilac
Curate and clean datasets for LLM fine-tuning.
judges
A library of LLM judges for model evaluation.
genai-cookbook
A GenAI-on-Databricks cookbook: patterns, examples, best practices.
databricks-ai-bridge
Bridge between Databricks and AI frameworks: LangChain, LlamaIndex.
megablocks
Mixture-of-Experts for efficient LLM training.
omnigent
An open-source meta-harness for orchestrating AI agents. It sits on top of Claude Code, Codex, Cursor, Pi — and turns them into interchangeable pieces. Built by the Databricks AI team together with Neon.
vs. using a single agent directly: Omnigent lets you compose agents without rewriting code, with spending/rate-limiting policies, sandboxing, and real-time collaboration via URL. An agent is a YAML file with a prompt, harness, tools and sub-agents.
# omnigent.yaml — define a composite agent
agents:
- name: data-engineer
harness: claude-code
prompt: "You are a DE who's an expert in Databricks..."
tools: [databricks-cli, dbt]
policies:
spend_cap: 10.00
require_approval: [shell]I wrote a full post about Omnigent with installation, configuration and a real test drive.
Databricks acquired MosaicML in 2023. Its repos are still active and are what’s used internally to train Databricks’ foundation models (DBRX, MPT).
llm-foundry
Foundation model training code. This is the actual code used to train DBRX and the models running behind AI/BI Genie. One of the most complete LLM training repos in open source.
composer
Framework for efficient model training.
streaming
Optimized data loading for model training.
Data Engineering & Platform Official
These are the repos you probably already know if you work with Databricks, but it’s worth having them all in one place.
terraform-provider-databricks
IaC for workspaces, clusters, jobs, Unity Catalog. Essential for any serious setup.
terraform-databricks-examples
Multi-cloud Terraform templates (AWS, Azure, GCP).
terraform-databricks-sra
Security Reference Architecture. Security best practices with Terraform.
mlops-stacks
Production MLOps template with DABs and CI/CD. The best starting point if you’re starting a project from scratch.
dbt-databricks
The official dbt adapter for Databricks.
bundle-examples
Databricks Asset Bundles examples. Templates for jobs, pipelines, MLOps.
delta-live-tables-notebooks
Reference notebooks for Delta Live Tables.
databricks-sql-python
The official SQL connector for Python.
notebook-best-practices
Software engineering patterns for Databricks notebooks.
Solution Accelerators by industry Industry
This is Databricks’ “softest” org — and probably the least known among developers. You won’t find SDKs or CLIs here. What you’ll find are complete business solutions: notebooks with end-to-end pipelines, sample data, and use-case-oriented documentation. If your boss says “I need this for yesterday”, these repos are your best friend.
The databricks-industry-solutions org has 213 repos organized by vertical. Solution Accelerators are meant to be cloned, adapted to your data, and get you something working fast. Databricks lists them in its official catalog.
pixels — Medical imaging at scale
The most popular repo in the org (418 stars). Massive-scale processing of DICOM images, documents and ZIPs. Includes OHIF Viewer integration and segmentation models. If you work with healthcare data, this one is a must.
smolder — HL7 as a Spark DataSource
A native Spark datasource for HL7 messages (the healthcare interoperability standard). You read HL7 files as if they were Parquet.
hls-llm-doc-qa — QA over medical documents
RAG over clinical documentation with open-source LLMs. A good starting point if you want to build an assistant for healthcare professionals.
esg-scoring — ESG scoring with NLP
Sustainable finance: analyzes news with NLP to generate automatic ESG scores. Full pipeline for ingestion, processing and scoring.
smart-claims — Smart claims management
DLT pipeline for insurance claims management with fraud detection. Includes ML models and dashboards.
fine-grained-demand-forecasting — Distributed demand forecasting
Store-item level forecasting with distributed models on Spark. The classic retail accelerator — one of the most cloned.
product-search — Semantic product search
Embeddings + vector store for semantic search over product catalogs. Ideal for e-commerce.
security-analysis-tool — Security auditing
SAT analyzes your Databricks workspace’s security configurations and gives you best-practice recommendations. Useful for any company, regardless of industry.
many-model-forecasting — Forecasting at large scale
A framework for training thousands of forecasting models in parallel (MMF). Applies to retail, finance, logistics — any domain with massive time series.
auto-data-linkage — Automated entity resolution
Record linkage and deduplication with Databricks ARC. Solves the classic “is this the same customer?” problem at scale.
Clone the repo, import the notebooks into your workspace, and adapt the queries to your data. The accelerators come with sample data so you can try them without wiring anything up. The official catalog lets you filter by industry and use case.
Databricks Labs Labs
Labs is where the interesting stuff happens. Projects built by Databricks employees that have no official support but often end up becoming indispensable tools.
dolly
Databricks’ first open-source instruction-tuned LLM. Historic — it marked a before and after in the democratization of LLMs.
dbldatagen
Synthetic data generator that scales to billions of rows using native Spark. I use it in my pipelines to generate test data before touching real data. Supports consistent primary/foreign keys across tables, statistical distributions, weights on discrete values and Faker integration.
vs. Faker in Python: Faker is single-threaded and doesn’t scale. dbldatagen generates distributed data on Spark while keeping referential consistency across tables.
import dbldatagen as dg
df = (dg.DataGenerator(spark, name="test_data",
rows=1_000_000, partitions=4)
.withIdOutput()
.withColumn("status", "string",
values=['active', 'inactive', 'pending'],
random=True, weights=[9, 1, 1])
.withColumn("score", "float",
expr="floor(rand() * 100)")
).build()dqx
A Data Quality framework for PySpark DataFrames. I use it in production to validate data between medallion layers — if something comes in wrong from bronze, you find out before it reaches gold.
vs. Great Expectations / Soda Core: dqx is PySpark-native with no external dependencies, with a built-in profiler that automatically generates quality rules from statistical profiles of your data. GE requires a lot of manual config; Soda needs its own cloud. dqx also generates expectations for DLT/Lakeflow Pipelines.
from databricks.labs.dqx.profiler.profiler import DQProfiler
from databricks.labs.dqx.engine import DQEngine
from databricks.sdk import WorkspaceClient
ws = WorkspaceClient()
profiler = DQProfiler(ws)
_, profiles = profiler.profile(input_df)
# Generates checks automatically — review them before applying
checks = DQGenerator(ws).generate_dq_rules(profiles)
# Split valid rows from quarantined ones
valid_df, quarantine_df = DQEngine(ws) \
.apply_checks_by_metadata_and_split(input_df, checks)ucx
The only tool for migrating entire workspaces from Hive Metastore to Unity Catalog in an automated way. It handles groups, tables, views, dashboards, jobs, notebooks, DLT pipelines and permissions in bulk.
vs. manual migration: without UCX you need weeks of manual scripting with CREATE TABLE ... AS SELECT. UCX generates an upfront assessment that identifies incompatibilities and then migrates in batch with post-migration data reconciliation.
# 1. Install
databricks labs install ucx
# 2. Assessment of the current workspace
databricks labs ucx assessment
# 3. Hive → Unity Catalog table mapping
databricks labs ucx create-table-mapping
# 4. Migrate tables and code
databricks labs ucx migrate-tables
databricks labs ucx migrate-local-codeRequires Workspace Administrator, a UC metastore already created, and a PRO/Serverless SQL Warehouse for the assessment report.
tempo
Time series API on top of Spark: lagged values, rolling stats, AS OF joins, downsampling.
mosaic
Large-scale geospatial processing on Spark.
dlt-meta
Metadata-driven ETL with DLT. You define bronze/silver in config, not in code.
ontos
Business Catalog: a business-context layer on top of Unity Catalog.
lakebridge
A migration accelerator from other data warehouses to Databricks.
discoverx
A swiss-army knife for exploring and administering Unity Catalog.
mcp
Model Context Protocol for Databricks.
pylint-plugin
A PyLint plugin with rules specific to Databricks code.
No Labs repo has official Databricks support. Before using one in production, check the repo’s recent activity (commits, answered issues) and have a plan B. Some popular repos like dbx and overwatch are already deprecated.
Open-Source Ecosystem OSS
There’s a point here worth clarifying: not all of these projects were “created by Databricks”. The relationship is more nuanced than that, and understanding it helps you gauge how independent each one really is.
Apache Spark was born in 2009 at UC Berkeley’s AMPLab as an academic research project. The founders of that project (Matei Zaharia and others) later founded Databricks in 2013 to commercialize it, but Spark already existed before the company did. Today it’s an Apache Software Foundation project, community-governed, with contributors from dozens of companies. Databricks built its entire platform around Spark, not the other way around.
Delta Lake was indeed created by Databricks and open-sourced in 2019. But it lives as an independent project in the Linux Foundation under the delta-io org. It has its own governance and external contributors. Databricks uses it as the core table format of its platform, but Delta Lake works perfectly well outside Databricks — with standalone Spark, with Flink, or directly from Python via delta-rs.
MLflow was created by Databricks (led by Matei Zaharia) and open-sourced in 2018. It also lives in the Linux Foundation with independent governance. Databricks integrated it deeply into its platform (Model Registry, Experiment Tracking, Model Serving), but MLflow runs anywhere — it’s platform-agnostic.
Unity Catalog is the most recent. It was a proprietary Databricks product until 2024, when the OSS version was open-sourced. It’s still the project most dependent on the Databricks ecosystem, but the intent is for it to work as an open multi-engine catalog.
The key point: these are independent projects that Databricks fit into its platform, not the other way around. Spark existed before Databricks, and Delta Lake, MLflow and Unity Catalog work outside Databricks. If tomorrow you migrate to another vendor, these projects are still yours.
delta-rs
Delta Lake native in Rust, with Python bindings. Living proof that Delta Lake is an independent project: this implementation doesn’t contain a single line of Databricks or Spark code.
vs. delta-spark: delta-spark requires JVM + Spark. delta-rs runs in pure Python — perfect for CI/CD, Lambda functions, validation scripts, or integrations with Polars and pandas. It’s the only option for Delta Lake access in JVM-less environments.
from deltalake import DeltaTable, write_deltalake
import pandas as pd
# Write Delta without Spark (also accepts s3://, abfss://)
df = pd.DataFrame({"id": [1, 2, 3], "value": ["a", "b", "c"]})
write_deltalake("./data/mi_tabla", df)
# Read with filters and time travel
dt = DeltaTable("./data/mi_tabla")
df = dt.to_pandas(filters=[("id", ">", 1)])
# Native DML: merge, delete, update
dt.merge(source=new_df, predicate="s.id = t.id",
source_alias="s", target_alias="t") \
.when_matched_update_all() \
.when_not_matched_insert_all() \
.execute()delta-sharing
An open protocol for securely sharing data across organizations. Works cross-platform — you don’t need both parties to be on Databricks.
delta-kernel-rs
A low-level Rust implementation for reading/writing Delta from any engine. It’s the kernel that lets any query engine (not just Spark) work with Delta Lake natively.
Community and Developer Advocates Community
Not every useful repo in the Databricks ecosystem comes from the official orgs. Some come from Databricks Developer Advocates (employees who build open-source tools for the community) and others from external MVPs recognized by the MVP program.
Matthew Powers is a Staff Developer Advocate at Databricks (not an MVP — he’s an employee). But his repos are open-source, live outside the Databricks orgs, and he maintains them as personal projects. Many teams use them in production without knowing they come from someone at Databricks. Fun fact: most of his repos are named after characters from Daria — the MTV animated series. spark-daria, quinn (Daria’s sister), mack, jodie… If you see a PySpark repo named after a nineties character, it’s probably MrPowers’.
chispa
The only PySpark testing library that shows you error messages with colors and readable diffs — differing rows in red, matching ones in blue. Without chispa, a failing test throws an unreadable dump of Row objects at you. With chispa, you see exactly which row failed and why.
vs. the alternative (manual asserts with .collect()): no contest. chispa saves hours of debugging.
from chispa import assert_column_equality
import pyspark.sql.functions as F
data = [("jo&&se", "jose"), ("**li**", "li"), (None, None)]
df = (spark.createDataFrame(data, ["nombre", "esperado"])
.withColumn("limpio",
F.regexp_replace(F.col("nombre"), "[^\\w\\s]+", "")))
# If it fails, it shows you exactly which row differs
assert_column_equality(df, "limpio", "esperado")spark-daria
Essential Spark extensions in Scala. The one that gave the saga its name.
quinn
PySpark productivity utilities. Column helpers, DataFrame validations, transformations.
mack
Helpers for Delta Lake: SCD Type 2, upserts, deduplication, kill duplicates.
Databricks MVP. He maintains “The Internals Of”, a series of open-source books that are probably the best free resource out there for understanding how Spark, Delta Lake and the ecosystem work under the hood. These aren’t shallow tutorials — they’re real deep dives into the source code, the algorithms and the design decisions. If you’ve ever wanted to understand why Spark does what it does (and not just how to use it), this series is a must. All free at books.japila.pl.
apache-spark-internals
The main book in the series. Covers the scheduler, memory management, shuffle, storage — everything the official documentation doesn’t explain.
spark-sql-internals
Catalyst optimizer, Tungsten execution, query planning, AQE. If you optimize queries in Spark, this book changes your perspective.
spark-structured-streaming-internals
Micro-batching, watermarks, state management — Structured Streaming inside out.
delta-lake-internals
Transaction log, checkpointing, optimistic concurrency, VACUUM — everything that happens underneath a Delta table.
sqlglot
SQL parser and transpiler in Python. Supports the Databricks/Spark SQL dialect. Used internally by many tools in the ecosystem.
dataflint/spark
Drop-in replacement for the Spark UI. Much better for debugging and profiling jobs.
spark-style-guide
A community style guide for Spark projects. A good reference for teams that want to standardize.
Final thoughts: open source as a way to stay current
If there’s one thing that’s clear to me after putting this map together, it’s that the Databricks ecosystem moves fast because it moves in the open. The repos listed here aren’t static projects — they have issues, PRs, discussions, releases. Each one is a window into how real problems are being solved today.
Keeping a good relationship with the open source community isn’t just about “giving back” — it’s one of the most effective ways to stay up to date. When you follow a repo, read the changelogs, try a new version or report a bug, you’re learning how the platform evolves before it shows up in a corporate blog or a keynote.
You don’t have to contribute to everything. Just by following the repos you care about, trying the tools in your environment and sharing what you learn, you’re already participating. Open source works because there are people who choose to stay current — and share it. As long as you want to keep learning, the community will keep you in the loop.
The stars and last-commit badges update automatically — if you’re reading this months after publication, the numbers you see are today’s.
Know a repo I missed? Leave me a comment below — I’m always looking for new tools.