Software engineering settled the testing argument decades ago. Code without tests is understood to be code you cannot safely change. Data pipelines, which are code, are still routinely shipped with nothing beyond a successful run as evidence of correctness.
The reason is partly historical and partly genuine. Data is harder to test than application logic, because correctness depends on inputs you do not control and cannot fully anticipate. But the difficulty has been used to justify doing very little, and the gap between “hard” and “not attempted” is where most data incidents live.
A green run proves almost nothing
The default signal in most data platforms is job status. The pipeline ran. It did not error. Therefore things are fine.
This confuses execution with correctness. A pipeline that reads an empty source table, transforms nothing and writes zero rows completes successfully. A pipeline whose upstream schema changed so a join now matches nothing completes successfully and produces a table full of nulls. A pipeline processing a partially-loaded source completes successfully with quietly understated figures.
These are the failure modes that matter, because they are the ones that do not announce themselves. Loud failures get fixed within the hour. Silent ones get reported to a board.
Four layers worth having
Schema contracts at the boundary
The boundary between a source system and your platform is where most breakage originates, and it is a boundary you do not control. Someone else’s team ships a change and your pipeline is downstream of a decision it was not consulted on.
A contract states the expected shape: columns present, types, nullability, and any enumerated values that logic depends on. Violations should stop ingestion rather than propagate. Failing loudly at the boundary is enormously cheaper than discovering a type coercion six transformations downstream.
Structural assertions on every model
Uniqueness on keys. Not-null on required fields. Referential integrity where it is assumed. Accepted values on categoricals.
These are cheap to declare and catch a surprising proportion of real defects, particularly duplicate keys arising from a join fanning out after an upstream change. If a team is doing no testing at all, this is where to start, because the cost is close to zero and the coverage is immediate.
Volume and distribution checks
Structural tests will not catch a source that delivered sixty percent of its usual rows. Everything is unique, nothing is null, and the numbers are wrong.
Row counts against a rolling baseline catch this. So do distribution checks on key fields: if the share of transactions in a given currency moves sharply overnight, something upstream changed. These need tuning to avoid alert fatigue, and they need to account for genuine seasonality, which is real work. They also catch the class of failure that structural tests fundamentally cannot.
Business invariants
The highest-value and least-implemented layer. These are statements that must hold if the business logic is correct. The sum of segment revenue equals total revenue. No account has a negative balance. Every completed order has a fulfilment record.
They require domain knowledge, which is why they are usually missing: the engineer does not know them and the analyst does not write tests. Getting them written is an organisational problem more than a technical one, and it is worth the effort because they catch logic errors that no generic check will ever see.
Failing loudly, proportionately
A test that fires into an unmonitored channel is not a test. Neither is one that fires so often that the channel is muted.
Severity has to be part of the definition. A failed uniqueness check on a key feeding regulatory reporting should halt the pipeline and page someone. A distribution shift on a minor dimension should post a notice for review. Treating both identically guarantees that either the pipeline stops constantly or the alerts get ignored, and both outcomes end with the tests being disabled.
The second requirement is that a failure blocks downstream consumption. Detecting bad data and publishing it anyway, with an alert nobody reads before the report goes out, is a more elaborate way of shipping bad data.
Where to start
Blanket coverage is the wrong goal and the usual reason these initiatives stall. Instead, take the outputs that leave the building, meaning anything reported externally, submitted to a regulator or shown to a board. Trace their dependency paths. Instrument those paths properly across all four layers.
That will be a minority of the estate carrying the large majority of the consequence. Everything else gets structural assertions and nothing more until there is a reason to do otherwise. Uneven coverage that reflects actual risk beats even coverage that reflects a desire for tidiness.
The organisational problem underneath
Most of what we have described is technically straightforward. The reason it does not happen is rarely technical.
Data engineers can write structural tests without help, and generally do. The layer that catches the expensive failures, business invariants, requires knowing what the business logic is supposed to guarantee. That knowledge sits with analysts and domain experts, who usually do not write tests and often do not know that a test is the appropriate place to put what they know.
The teams that close this gap do it with a conversation rather than a tool. Sit an engineer with a domain expert and ask a single question: what would be obviously wrong if you saw it. The answers come quickly and they are exactly the invariants worth encoding. An hour of that per domain produces more useful coverage than a quarter of generic test expansion.
It also has a side effect worth mentioning. Domain experts who have watched their knowledge become an automated check tend to volunteer more of it, because they can see where it went and what it caught.
Testing the tests
A test that has never failed is not necessarily a test that is working. It may be a test that cannot fail, because the assertion is trivially satisfied or the query has a defect that returns no rows regardless.
This sounds like a marginal concern and it is not. We have reviewed suites where a meaningful proportion of assertions were structurally incapable of firing, and the team’s confidence was proportional to the number of tests rather than to their effectiveness. That is a worse position than having no tests, because it is a false signal that discourages further investigation.
Periodically prove that critical tests can fail. Introduce the defect deliberately in a development environment and confirm the alert arrives where it is supposed to and reaches a person. This is a half-day exercise a couple of times a year, and it is the only way to know whether the safety net has a hole in it.
The same applies to the alerting path itself, which frequently breaks silently when a channel is renamed, a rota changes, or an integration token expires. The test fires correctly into nowhere.
