The Lakehouse at the Edge: A Million Readings a Minute Is Easy. Knowing Which Ones to Believe Is the Job. [2026]
The predictive maintenance model flagged the compressor on Line 4 for imminent bearing failure. Maintenance pulled it during the Sunday window. The bearing was fine. What was not fine was a vibration sensor that had been reporting in the wrong units since a firmware update three weeks earlier — and a pipeline that landed every one of those readings, in good faith, into the table the model trained on.
Nobody built a bad model. Somebody built a pipeline that trusted the sensor. The first IoTunderground article named this as the reason the 2017–2019 platform generation failed: intelligence was being built on a foundation of noise, and the data infrastructure to validate telemetry at scale did not exist. It exists now. This is what it looks like.
First, the Name
If you built pipelines on Databricks between 2022 and mid-2025, you built them with Delta Live Tables. At the 2025 Data + AI Summit, Databricks rebranded the product as Lakeflow Declarative Pipelines under a broader Lakeflow umbrella and open-sourced the core engine as Apache Spark Declarative Pipelines. [QueryPlane, Jun 2026] Existing pipelines did not need to change: all DLT pipelines continue to run within Lakeflow with no upgrade or code modifications, and all capabilities — streaming tables, materialized views, data quality expectations — remain available. [Databricks]
The import dlt still works. The concepts are the same. This article uses the current name and the code you already know.
Three Layers, One Job Each
The medallion architecture is not a Databricks invention, but the declarative framework makes it nearly free to implement correctly: Bronze streaming tables land raw data, Silver tables clean, deduplicate, join to dimensions, and enforce row-level rules through expectations, and Gold materialized views aggregate to the granularity each downstream consumer needs. [QueryPlane, Jun 2026]
For telemetry, each layer has exactly one responsibility, and the failure mode of every troubled IIoT deployment is a layer doing a job that belongs to a different one.
| Layer | Its One Job | What It Must Not Do | What Breaks If It Does |
|---|---|---|---|
| Bronze | Land every byte, exactly as received, with ingest metadata | Transform, filter, or cast | You lose the ability to replay from raw when the Silver rules turn out to be wrong |
| Silver | Type, deduplicate, and enforce every rule about what a valid reading is | Aggregate or compute business metrics | Validation gets tangled with analytics and nobody can say which rows the model actually saw |
| Gold | Aggregate to the shape one consumer needs | Validate or clean | A single wide table tries to serve everyone and serves no one; one view per dashboard family is the right granularity [QueryPlane] |
The Pipeline
This is a working pipeline for a fleet of industrial sensors landing JSON to cloud storage — roughly 10,000 devices reporting every six seconds, which is a million readings a minute. It is Python, it runs as-is in a Lakeflow pipeline, and the Silver layer is where you should spend your attention.
import dlt
from pyspark.sql import functions as F
# ─────────────────────────────────────────────────────────────
# BRONZE — land everything. Validate nothing. Keep the raw payload.
# ─────────────────────────────────────────────────────────────
@dlt.table(
name="telemetry_bronze",
comment="Raw sensor readings exactly as landed. Replay source of truth.",
table_properties={"quality": "bronze"}
)
def telemetry_bronze():
return (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/mnt/telemetry/_schema/bronze")
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.load("/mnt/telemetry/landing/")
.withColumn("_ingest_ts", F.current_timestamp())
.withColumn("_source_file", F.col("_metadata.file_path"))
)
# ─────────────────────────────────────────────────────────────
# SILVER — the validation layer. Every expectation is a control limit.
# ─────────────────────────────────────────────────────────────
@dlt.table(
name="telemetry_silver",
comment="Typed, deduplicated, validated. The rules below are the spec.",
table_properties={"quality": "silver"}
)
# Structural: a reading with no identity or no time is not a reading.
@dlt.expect_or_drop("has_device_id", "device_id IS NOT NULL")
@dlt.expect_or_drop("has_event_ts", "event_ts IS NOT NULL")
# Temporal: reject the far past (replayed backlog) and the future (clock skew).
@dlt.expect_or_drop("ts_is_plausible",
"event_ts BETWEEN current_timestamp() - INTERVAL 7 DAYS "
"AND current_timestamp() + INTERVAL 5 MINUTES")
# Physical: the sensor's rated range. Outside it is a sensor fault, not a process event.
# WARN, don't drop — you want these rows counted and visible, not silently gone.
@dlt.expect("temp_in_rated_range", "temp_c BETWEEN -40 AND 125")
@dlt.expect("vibration_in_rated_range", "vibration_mm_s BETWEEN 0 AND 50")
# Contract: an unknown schema version means firmware shipped something you haven't reviewed.
# FAIL the update. Stop the line. Look at it.
@dlt.expect_or_fail("schema_version_known", "schema_version IN ('1.0', '1.1', '2.0')")
def telemetry_silver():
return (
dlt.read_stream("telemetry_bronze")
.withColumn("event_ts", F.to_timestamp("event_ts"))
.withColumn("temp_c", F.col("temp_c").cast("double"))
.withColumn("vibration_mm_s", F.col("vibration_mm_s").cast("double"))
.withWatermark("event_ts", "10 minutes")
.dropDuplicates(["device_id", "event_ts"])
)
# ─────────────────────────────────────────────────────────────
# GOLD — one view per consumer. This one feeds the maintenance dashboard.
# ─────────────────────────────────────────────────────────────
@dlt.table(
name="machine_health_5m",
comment="Per-device 5-minute windows. Mean and SD per channel for SPC charting.",
table_properties={"quality": "gold"}
)
def machine_health_5m():
return (
dlt.read_stream("telemetry_silver")
.withWatermark("event_ts", "10 minutes")
.groupBy(F.window("event_ts", "5 minutes"), "device_id")
.agg(
F.avg("temp_c").alias("temp_mean"),
F.stddev("temp_c").alias("temp_sd"),
F.avg("vibration_mm_s").alias("vib_mean"),
F.stddev("vibration_mm_s").alias("vib_sd"),
F.count("*").alias("n_readings")
)
)Three tables. About sixty lines. The framework handles the orchestration, the cluster, the checkpointing, and the dependency graph — Databricks reports nearly 4x price/performance over its own baseline for pipelines built this way. [Databricks] What it does not handle is deciding what a valid reading is. That is the six expectation lines in the middle, and it is the entire job.
Warn, Drop, or Fail: The Decision That Is Actually the Design
Every expectation in Lakeflow has three modes, and which one you choose for each rule is the most consequential decision in the pipeline. The code above uses all three deliberately.
| Mode | What Happens | Use It When | In the Code Above |
|---|---|---|---|
expect |
Row passes through. Violation is logged and counted. | The violation is information — you want to see how often it happens and where | Rated-range checks. An out-of-range temperature is a sensor fault you need to count, not a row you want to hide. |
expect_or_drop |
Row is removed. Violation is logged and counted. | The row cannot be a valid reading under any interpretation | Null identity, null time, timestamp from next year. Not a reading. Gone — but counted. |
expect_or_fail |
Pipeline update stops. | Continuing would poison everything downstream and a human needs to look | Unknown schema version. Firmware shipped a payload you have never reviewed. That is the Line 4 compressor. |
The Line 4 failure was a expect_or_fail problem handled as an expect problem — or more precisely, handled as nothing, because there was no expectation at all. The firmware update changed the units. A schema-version gate would have stopped the pipeline on the first new payload. Instead, three weeks of wrong-unit vibration data went into the model, and the model did exactly what it was trained to do with it.
Expectations Are Control Limits. Treat Them That Way.
Walter Shewhart’s insight in 1924 was that a process has natural variation, and the job of quality control is to distinguish that variation from a genuine signal that something changed. The tool was the control chart: compute the limits, plot the data, and every point either falls inside the limits (noise) or outside them (signal). Deming spent fifty years teaching manufacturers to trust the chart over their instincts.
The Silver expectations are that chart, applied to the pipe instead of the product. temp_c BETWEEN -40 AND 125 is a control limit. The expect mode is the chart — it lets the point through and marks it. And the pipeline’s own metrics on how many rows hit each expectation are the process-behavior data Deming would have wanted on the wall: if the rated-range violation count on one device jumps from 0.01% to 4% between Tuesday and Wednesday, something changed on that device Tuesday night, and you know it before the model does.
This is why the layer goes first. Not because validation is a best practice, but because the expectation violation rates are themselves the earliest telemetry you have about the health of the fleet. A pipeline that validates last treats those signals as cleanup. A pipeline that validates first treats them as the first thing worth looking at.
Where It Breaks at a Million a Minute
Late data and the watermark. The withWatermark("event_ts", "10 minutes") line tells the stream to stop waiting for readings older than ten minutes. Readings that arrive later than that are dropped from the windowed aggregates. For a fleet on reliable connectivity, ten minutes is generous. For a fleet on NB-IoT in basements — see the previous article — a device that repeats transmissions for battery reasons can deliver a reading twenty minutes late, and a ten-minute watermark silently discards it. The watermark is a physics decision, not a default.
Schema drift is the normal case, not the edge case. Auto Loader’s addNewColumns mode is the safe default: it lets new top-level columns appear without breaking downstream consumers. [QueryPlane, Jun 2026] That handles a firmware update that adds a field. It does not handle one that changes the meaning of an existing field — which is the Line 4 case, and why the schema-version gate exists as a separate expectation.
Replay is the whole reason Bronze exists. Auto Loader tracks which files have been processed in a transactional log under the schema location, so a re-run never double-processes a file, and a backfill is a single FULL REFRESH away. [QueryPlane, Jun 2026] When you discover a Silver rule was wrong — and you will — you fix the rule and refresh Silver from Bronze. If Bronze had been transforming data, there would be nothing clean to refresh from.
Deduplication needs a real key. dropDuplicates(["device_id", "event_ts"]) assumes a device never sends two different readings with the same timestamp. NB-IoT repetition means it can send the same reading twice; that is what this catches. If your firmware batches readings and reuses a batch timestamp, you need a sequence number in the key, and that is a conversation with the firmware team, not a pipeline fix.
I spent a long stretch of my career in audit, and audit has one rule that data engineering keeps relearning at cost: you cannot sample your way to confidence when the population is the thing that matters. Taleb makes the same argument about tail risk. The two percent of readings you did not check are where the bearing that was fine gets pulled and the bearing that was failing does not.
The lakehouse layer changed one thing and it is the thing this whole site is about: a hundred percent validation on a million readings a minute is now cheap. Six expectation lines. The framework does the rest. There is no longer a cost argument for validating a sample and trusting the remainder. The only argument left is habit — the habit of treating validation as the cleanup step after the interesting work, instead of the interesting work itself.
Shewhart would recognize the Silver table immediately. It is a control chart with the plotting done by a cluster. The expectation violation counts are the process telling you it changed. The model gets the credit. The six lines in the middle do the work. Write them first.
Sources
QueryPlane, “Databricks Lakeflow Declarative Pipelines (Delta Live Tables) in Practice,” June 2026 (2025 Data + AI Summit rebrand; Apache Spark Declarative Pipelines open-source; medallion layer responsibilities; Auto Loader transactional log and FULL REFRESH; addNewColumns schema evolution; one Gold view per dashboard family) · Databricks, “Getting Started with DLT” (existing pipelines run unchanged under Lakeflow; all capabilities retained; avionics IoT tutorial) · Databricks, Delta Live Tables product page (Auto Loader and streaming tables to Bronze; expectations; ~4x price/performance vs. baseline) · B EYE, “Databricks Delta Live Tables: Best Practices and Advanced Techniques,” June 2025 (manufacturing IoT medallion case: Bronze streaming from Event Hub, Silver expectations on sensor bounds, Gold anomaly flags per machine) · Flexera, “Delta Live Table 101,” July 2026 (expectation syntax; ON VIOLATION FAIL UPDATE) · IoTunderground, “The Intelligent Edge” (the data-quality gap) and “Subterranean Telemetry” (NB-IoT repetition and late arrival) · W.A. Shewhart, Economic Control of Quality of Manufactured Product, 1931.
The Line 4 compressor is a composite drawn from incidents of the same shape; the failure pattern is real and common, the specific plant is not identified. The code has been reviewed for syntax against current Lakeflow documentation; if you run it and something breaks, that is exactly the kind of correction this site is built on. Scott@IoTunderground.com.
