The Pipe
Pre-Crime Telemetry: Why Autopsies Are for Amateurs and Margins Die in the Drift
A failed test is the last place a problem shows up. A control chart is the first. On a fixed-price contract, the time between them comes out of margin.
Written by Scot Free
Here is a composite, not a case file. A flight unit fails thermal vacuum testing. Within a day there is a tiger team, a standing meeting, and a slide template. Six weeks later the root cause lands: a cure oven on a bonding process had been running a few degrees warm since spring. Every panel it produced passed inspection. By the time anyone knows that, the milestone is gone, the penalty clause has fired, and the rework is priced against a contract value that was fixed two years ago.
That is an autopsy. It is thorough, it is documented, and it is performed on something that is already dead.
The failed test did not find the problem. It found the bill. The problem had been sitting in the process data for weeks, inside the spec limits, in a stream nobody was charting.
Keep the post-mortem. Stop using it as the detection system.
The receipt is sixteen years old
In April 2010 the Government Accountability Office compared how defense programs and leading commercial manufacturers get a product into production (GAO-10-439). It visited GE Aviation, GE Healthcare, Honeywell and Siemens. Its finding reads like the premise of this article. The commercial firms design quality in and bring their manufacturing processes into statistical control before they commit to production. At many defense contractors, in GAO's words, "problems are identified and corrected after a product is produced."
The same report put numbers on what that costs:
- Nearly two-thirds of major programs that entered production after 2000 reported unit cost growth above 5 percent.
- More than half of the programs in production had slipped after the production decision, the point where processes are supposed to be under control.
- On one missile program, the prime relied on sub-tier suppliers to self-report their capabilities. Defective parts followed.
- The readiness criteria DOD was proposing at the time did not require process control until after the production decision.
GAO had been writing versions of this since 1996, when it named culture as the biggest reason the fixes were not adopted.
In July 2026 GAO published its 24th annual weapon systems assessment: 104 programs, more than $2.4 trillion, and an average of over 12 years to deliver a capability. It found, again, that programs do not consistently follow leading practices. Trade coverage of the same assessment describes a hypersonic missile line held below its planned rate by manufacturing and quality-control problems, including work standards that were incomplete or inconsistent. You cannot chart a process that has no standard.
The limit of this receipt: GAO does not say a control chart would have saved any of these programs. Production maturity is one of three knowledge gaps it tracks, alongside technology and design, and cost growth has more than one parent. What GAO does establish is the pattern: the knowledge was available earlier than the program used it.
What fixed-price does to a late finding
On a cost-plus contract, a late finding is mostly the customer's schedule problem. On a fixed-price contract it is the contractor's money.
Boeing's defense unit is the public ledger for this. Its charges on five fixed-price development programs ran to more than $4.4 billion in 2022, nearly $1.6 billion in 2023, and $4.9 billion in 2024. That is close to $11 billion in three years. The KC-46 tanker alone had passed $7 billion in cumulative losses by late 2023, and a July 2026 charge pushed the Air Force One replacement past $3 billion.
The limit of this receipt: Boeing attributes those charges to supply chain, inflation, labor, a strike, and bids that were priced to win. A chart on a torque station does not fix a lowball bid, and I am not claiming it would. The claim is narrower. Fixed-price converts time-to-detect directly into dollars, and there is nobody to send the invoice to. Boeing's then-CEO said as much in 2023: under fixed price, an unplanned hurdle becomes a cost you do not get back.
The mechanism is simple. A drifted process keeps producing. Every unit built between the drift and the detection carries it, and every assembly step stacked on top has to come back off. GAO's helicopter case from 2010 is the picture: problems found in test had to be corrected on aircraft already on the line.
What Shewhart actually built
In 1924 Walter Shewhart, a physicist at Western Electric, sketched the first control chart in a one-page memo. It fit on a page because it is one idea. Every process varies, and the variation comes in two kinds. Common-cause variation is the noise the process always makes. Special-cause variation (Shewhart called it assignable) means something changed.
A control chart separates the two and does nothing else. It does not know why. It has no idea what the process intends. It says: this is not the variation this process normally makes, go look.
Two sets of limits get confused here, and the confusion is the whole article.
- Specification limits come from the drawing. They answer: will this part work?
- Control limits come from the process's own history. They answer: is this the same process that made the last thousand good parts?
Inspection checks the first. A process can walk a long way inside the spec limits before a single part fails, and every step of that walk is visible against control limits. That gap is the pre-crime window.
The drift also shows up before anything crosses three sigma. Western Electric's 1956 handbook added run rules for exactly this. A chart signals on any of four patterns:
- One point beyond three sigma.
- Two of three consecutive points beyond two sigma, same side.
- Four of five consecutive points beyond one sigma, same side.
- Eight consecutive points on the same side of the centerline.
The first rule catches a break. The other three catch a slow mover.
The pipe version
The Lakehouse at the Edge ended with a Gold table: five-minute means and standard deviations per device. That table is the input to a control chart. The four rules are about twenty lines on top of it.
from pyspark.sql import functions as F, Window as W
# gold: station_id, characteristic, window_start, mean_value
# baseline: station_id, characteristic, cl, sigma
# frozen from a known-good run; sigma is of the window means
z = (gold.join(baseline, ["station_id", "characteristic"])
.withColumn("z", (F.col("mean_value") - F.col("cl")) / F.col("sigma")))
w = W.partitionBy("station_id", "characteristic").orderBy("window_start")
def run(k, n, zone):
"""k of the last n points beyond `zone` sigma, same side."""
last_n = w.rowsBetween(-(n - 1), 0)
hi = F.sum(F.when(F.col("z") > zone, 1).otherwise(0)).over(last_n) >= k
lo = F.sum(F.when(F.col("z") < -zone, 1).otherwise(0)).over(last_n) >= k
return hi | lo
signals = (z
.withColumn("rule_1", F.abs("z") > 3)
.withColumn("rule_2", run(2, 3, 2))
.withColumn("rule_3", run(4, 5, 1))
.withColumn("rule_4", run(8, 8, 0)))
Three things in that block matter more than the syntax.
- The baseline is frozen. Centerline and sigma come from a period you have reason to believe was in control. Compute them on a rolling window and the limits follow the drift, which means the chart never fires.
- Sigma is the sigma of the window means. Use the standard deviation of raw readings and the limits are far too wide to see anything.
- This is a batch step over Gold. Row-based windows do not run on a streaming DataFrame. At five-minute grain that costs you nothing.
To check the code I ran it on one synthetic series: a stable process for 60 windows, then a slow ramp of 0.04 sigma per window. Rule 3 fired 21 windows into the drift. The three-sigma rule took 37. Across all 120 windows, not one value left a six-sigma spec band. That is one seed and an illustration, not a benchmark, but it is the shape of the argument: the run rules bought 80 minutes, and inspection would have passed every part.
| Autopsy | Pre-crime | |
|---|---|---|
| Question asked | Why did it fail? | Is this still the process we qualified? |
| Measured against | Spec limits, at test | Control limits, at the station |
| Signal arrives | After the unit is built and stacked | While the assembly is on the jig |
| Units affected | Everything since the drift began | Everything since the last clean window |
| Output | A root-cause deck | A stop, and someone walking to the station |
Where it breaks
The name is borrowed from Minority Report, and it is worth remembering that the story is about the prediction system being wrong. This one can be wrong too.
- False alarms scale with the pipe. On a stable process with well-behaved data, the three-sigma rule alone gives a false signal about once every 370 points. Turn on all four rules and it is about once every 92. At five-minute grain that is roughly three false signals a day per chart. Chart 200 characteristics and you have more than 600 a day, every one of them asking a person to walk somewhere. Chart the critical few. A million readings a minute does not mean a million charts.
- Telemetry is not independent. Those false-alarm rates assume each point is unrelated to the last. Fast sensor data is not, and correlated data makes the limits too tight. Widen the windows until the points behave, or chart the residuals.
- Aerospace builds twelve of things. There is no baseline in twelve units. So chart the process, not the product. A program that ships twelve vehicles still runs thousands of fasteners through the same torque tool and hundreds of cycles through the same cure oven.
- A signal is not a diagnosis. The chart says something changed. It takes a person at the station to find what. Without a written reaction plan that names who can stop the line, this is a dashboard.
- The drift is usually at the sub-tier, and you do not have their data. GAO's missile case was a prime trusting supplier self-reports. If the process data is not in the purchase order, the status slide is all you are entitled to.
Why it gets resisted
A chart is also a record. Once it exists, "nobody could have known" is gone. The stream shows the day the process moved and every status report filed after it, including the supplier's green slide. An autopsy culture can spread a failure across luck, supply chain and the weather. A timestamp cannot be spread.
That is the real reason this gets slow-walked, and it is also the way to ruin it. Deming's warning applies: use the chart to hunt operators and the operators will learn to manage the chart. They will adjust a stable process after every point, which makes the variation worse, or the data will stop being honest. The arrest is of the process. If it becomes an arrest of the person, the pipe goes dark.
The Underground Take
I ran audits for eleven years, and an audit is an autopsy with better paperwork. The period is closed, the sample is pulled, and the finding describes something that already happened to somebody. We sampled because looking at everything cost too much. On a shop floor that excuse expired the day the torque wrench got a radio.
What is left is not a technology problem. The statistics are a century old and the code above is twenty lines. What is left is whether anyone has the authority to stop a jig on the strength of eight points on one side of a line, with every part still in spec and a milestone payment two weeks out.
Deming put most of the trouble in the system and not in the worker. A control chart is how you prove it, in time to do something. Pick the critical few. Freeze the baseline. Write down who walks to the station. Do it before the test campaign, because afterward all you can do is describe the body. The model gets the credit. The pipe makes the arrest.
Sources
- U.S. Government Accountability Office, Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed, GAO-10-439, April 2010: commercial versus defense practice on statistical control; unit cost growth and post-decision delays for programs in production; supplier self-reporting on the missile case; the helicopter rework case; the 1996 finding on culture.
- U.S. Government Accountability Office, Weapon Systems Annual Assessment, GAO-26-108457, July 2, 2026: 104 programs, over $2.4 trillion, average delivery time over 12 years, inconsistent use of leading practices. Hypersonic production findings as reported by DefenseScoop, July 2026.
- Boeing fixed-price charges by year as reported from company disclosures by Defense News (Feb. 2024) and Breaking Defense (Jan. 2025, July 2026); KC-46 cumulative figure from Air & Space Forces Magazine (Oct. 2023).
- NIST/SEMATECH e-Handbook of Statistical Methods, section 6.3.2: control limits versus specification limits; Western Electric rules; false-alarm rates of about 1 in 371 and 1 in 91.75, citing Champ and Woodall, Technometrics, 1987.
- Western Electric Company, Statistical Quality Control Handbook, 1956: the four zone rules.
- Note on the arithmetic: the alarms-per-day figures are my multiplication of the NIST rates by 288 five-minute windows. They assume a stable process and independent, normally distributed points. Real telemetry will do worse.
The Gold table this chart reads from is built in The Lakehouse at the Edge. The security side of the same pipe is The Drainage Gate at Helm's Deep.
Corrections and field data: Scott@IoTunderground.com
