Blog

Advanced Databricks tips, Data Engineering book reviews, and episodes of the Spark de Ideas podcast.

Databricks Tips #16: Data Quality — dynamic expectations, quarantine, and DQX
Databricks Tips
Data Engineering

Warn is the default and filters nothing: how to go from scattered checks to a quality system in Databricks, with rules loaded dynamically from a Delta table, a quarantine…

Aug 5, 2026

Databricks Tips #15: Query Federation — querying Postgres and MySQL without moving data
Databricks Tips
Data Architecture

How to query Postgres and MySQL from Databricks without moving data: connections and foreign catalogs in Unity Catalog, which part of the query gets pushed down to the…

Jul 18, 2026

Databricks Tips #14: Liquid Clustering — the replacement for partitions and Z-ORDER
Databricks Tips
Delta Lake

What Liquid Clustering is, why it replaces partitioning and Z-ORDER, the CLUSTER BY syntax (including AUTO), how it gets triggered by OPTIMIZE, and a reproducible lab where…

Jul 10, 2026

Databricks Tips #13: OpenSharing — sharing without copying
Databricks Tips
Data Engineering
Delta Lake

What OpenSharing inherits from Delta Sharing and what it adds: true zero-copy, shares to Iceberg clients like Snowflake and Trino, models and agent skills as shareable…

Jul 7, 2026

Databricks Tips #12: Photon — the C++ engine that speeds up your queries without changing code
Databricks Tips
Data Engineering

What Databricks’ vectorized engine is, where it runs, what it speeds up and what it doesn’t, how to measure how much Photon your query uses, a with/without Photon benchmark…

Jul 2, 2026

From YAML to production: deploying AI Agents with Declarative Automation Bundles
Databricks Tips
Data Engineering
MLOps

How to deploy a complete RAG agent with memory, guardrails and CI/CD using a single databricks.yml — with real code, step by step.

Jun 27, 2026

Genie ZeroOps: Databricks wants your pipelines to fix themselves
Data Engineering
Databricks Tips
MLOps

Detect, Assess, Remediate, Verify — how the Databricks agent that monitors, diagnoses, and proposes fixes for your production pipelines works.

Jun 27, 2026

The Databricks GitHub map: repos you need to know
Data Engineering
Databricks Tips

A curated guide to the most useful repositories in the Databricks GitHub ecosystem: from the four official orgs to community tools and MVPs. With context on what each thing…

Jun 21, 2026

LTAP: Databricks wants to eliminate the ETL between your database and your warehouse
Data Engineering
Data Architecture
Databricks Tips

LTAP unifies OLTP and OLAP on a single copy of data over Delta and Iceberg. What it is, how it works, how it differs from HTAP and Zero ETL, and a concrete example of how it…

Jun 19, 2026

DAIS 2026 Day 3: Genie Code for ML, Lakeflow Designer and Nadella x Ghodsi
Data Engineering
Databricks Tips
MLOps

Genie Code specializes in ML engineering, Lakeflow Designer leaves preview, AI Runtime adds multi-node training, and Nadella records a fireside chat with Ghodsi. Recap of…

Jun 17, 2026

DAIS 2026: everything Databricks announced (and what it means for your stack)
Data Engineering
Data Architecture
Databricks Tips
MLOps

Lakehouse//RT, LTAP, Genie One, CustomerLake, Lakewatch, OpenSharing and more. A complete recap of Data + AI Summit 2026 from afar, with links to every announcement.

Jun 16, 2026

Omnigent: the open source meta-harness to orchestrate all your AI agents
AI Agents
Developer Tools
Open Source

Compose, govern and share: Omnigent is the missing layer on top of Claude Code, Codex and Pi. Installation, YAML configuration, policies and a real test.

Jun 14, 2026

Databricks Tips #11: Lakeflow Declarative Pipelines (ex DLT) — declarative pipelines with built-in quality
Databricks Tips
Data Engineering
Streaming

Streaming tables, materialized views, expectations, AUTO CDC with SCD Type 1/2, triggered vs continuous modes, serverless, and the gotchas that aren’t in the tutorial.

Jun 11, 2026

Databricks Tips #10: Unity AI Gateway — centralized governance for LLMs in production
Databricks Tips
Data Engineering
MLOps

Rate limiting, guardrails, usage tracking, cost attribution, fallbacks and traffic splitting. Everything you need to govern LLMs across your organization.

Jun 9, 2026

Databricks Tips #9: SQL Warehouses — the compute that turns itself on
Databricks Tips
Data Engineering
Serverless

Classic, Pro or Serverless: which one to pick, how to size it, and the gotchas that will save you money. With Photon, Query Federation, AI Functions and real costs.

Jun 6, 2026

Databricks Tips #8: Jobs & Workflows — streaming and triggers that start on their own
Databricks Tips
Data Engineering
Streaming

File arrival triggers, table update triggers, Trigger.AvailableNow, Job Clusters and the limitations that aren’t in the tutorial.

Jun 4, 2026

Databricks Tips #7: Docker on Databricks — custom containers for environments that don’t break
Databricks Tips
Data Engineering
Delta Lake

Databricks Container Services, custom images, golden containers, CI/CD with Docker, and the mistakes that will cost you hours.

May 30, 2026

Data Mesh in practice: what works, what doesn’t, and what nobody tells you
Data Architecture
Data Engineering
Podcast

Data Mesh beyond the hype. The 4 principles, a real implementation with Unity Catalog, and why most companies get it wrong.

Apr 21, 2026

Data Contracts: how to design a framework from scratch
Data Architecture
Data Engineering
Podcast

What Data Contracts are, why you need them, and how I implemented a framework in production with Databricks and PySpark.

Apr 14, 2026

Medallion vs Data Vault vs Kimball: when to use each one and why it matters
Data Architecture
Data Engineering
Podcast

A practical comparison of the three most widely used data models in the industry. With real examples, trade-offs, and a guide for choosing.

Apr 7, 2026

Databricks Tips #6: Feature Engineering — designing features that survive production
Databricks Tips
Machine Learning
Feature Engineering

Feature Store, point-in-time lookups, online features, feature freshness, and the classic data leakage mistakes.

Mar 31, 2026

Databricks Tips #5: MLflow + Unity Catalog — from experiment to model in production
Databricks Tips
Machine Learning
MLflow
Unity Catalog

Model registration in UC, aliases instead of stages, data-to-model lineage, and model serving with AI Gateway.

Mar 24, 2026

Databricks Tips #4: Structured Streaming — watermarks, triggers, and the micro-batch traps
Databricks Tips
Data Engineering
Streaming

Trigger modes, watermarks, foreachBatch, Auto Loader, and the traps that make you lose data or money.

Mar 17, 2026

Databricks Tips #3: Unity Catalog — the governance model nobody implements well
Databricks Tips
Data Engineering
Unity Catalog

Three-level namespace, inherited GRANTS, automatic lineage, row/column security and common governance mistakes.

Mar 10, 2026

Databricks Tips #2: 7 things about Delta Lake I wish someone had told me sooner
Databricks Tips
Data Engineering
Delta Lake

Liquid clustering, OPTIMIZE, Z-ORDER, vacuum, time travel and tricks that change how you work with Delta.

Mar 3, 2026

Databricks Tips #1: Databricks Asset Bundles — what the documentation doesn’t tell you
Databricks Tips
Data Engineering
DevOps

Advanced use cases, deployment patterns, complex variables and tricks I picked up working with DABs in production.

Feb 24, 2026

If I were starting from scratch… what I’d prioritize learning in Data Engineering
Podcast
Data Engineering
Career

Python, SQL, Spark, dbt, Databricks — the order matters. What I learned after years of working with data and what I’d do differently if I had to start over.

Feb 17, 2026

Data Engineering Design Patterns: 8 ingestion patterns you need to know
Podcast
Data Engineering
Book Review

The 8 ingestion patterns from Bartosz Konieczny’s book. Examples in PySpark/SQL and how they map to the Medallion architecture.

Sep 15, 2025

DataOps: how to take your pipelines to the next level
Podcast
Data Engineering
DataOps

What DataOps really is, how it differs from DevOps, the three pillars (automation, observability, collaboration), and common mistakes in production.

Sep 1, 2025

Fundamentals of Data Engineering: the foundations every data engineer needs to know
Podcast
Data Engineering
Book Review

Review of the book by Joe Reis and Matt Housley (O’Reilly, 2nd ed.). The data lifecycle, key architectures, undercurrents and what changed in this edition.

Aug 26, 2025

Spark de Ideas: the spark of data engineering in Spanish
Podcast
Data Engineering

Why we need a Data Engineering podcast in Spanish. Databricks, data architecture, MLOps and career — from the region, for the region.

Aug 18, 2025
No matching items