dbt is genuinely good software and it scales further than most teams need. What does not scale automatically is the way teams use it. Almost every project we inherit was set up sensibly for thirty models and is now carrying three hundred, and the practices that were fine at the first number are actively harmful at the second.
The symptoms are consistent enough to be diagnostic. Builds take an hour. Nobody is confident about what a change will affect. The staging layer has models that nothing consumes. Two models compute the same thing slightly differently and both are in use.
The flat structure stops working around a hundred models
Most projects begin with the standard staging, intermediate and marts layering. It is good guidance and it holds well for a while.
What it does not encode is domain. At a hundred models, a directory of marts is a list, and finding the right one requires either knowing already or reading several. At three hundred, nobody knows already, and the practical consequence is that engineers build new models rather than search for existing ones. Duplication compounds from there.
Reorganising by domain first and layer second fixes navigation and, more importantly, creates a natural unit of ownership. A domain directory can have an owner in a way that a layer directory cannot. It also makes selector syntax useful, because building everything in a domain becomes a single sensible command.
This refactor is disruptive and gets postponed indefinitely. Doing it at a hundred and fifty models takes a week. Doing it at four hundred takes a month and requires coordinating with more people.
Full-refresh habits become full-refresh problems
Early on, rebuilding everything is fine. It takes minutes and it is simple. Teams get used to it and carry the habit forward.
Past a certain size it becomes the dominant cost of the platform, and it degrades development experience in a way that changes behaviour. When a local build takes forty minutes, engineers stop building locally, which means they test in a shared environment, which means they interfere with each other, which means everyone slows down.
Incrementalisation is the answer and it is a real engineering exercise, not a configuration flag. It requires deciding what late-arriving data means for each model, how to handle restatement of prior periods, and what the recovery procedure is when an incremental model is found to be wrong. Teams that set the flag without answering those questions get subtly wrong data and a harder debugging problem than they started with.
Do the highest-cost models properly rather than all of them carelessly. The distribution of build cost is usually steep, so a small number of models account for most of the time.
Tests grow without pruning
Testing discipline in dbt projects tends to be good, which produces its own problem. Tests accumulate. Nobody removes them. Eventually the test suite takes longer than the build and a proportion of it fails routinely.
Routinely-failing tests are worse than absent ones, because they train the team to ignore red. Once a suite has a known-failing subset that everyone mentally filters, the signal is gone and a genuine failure hides in the noise.
Two practices help. Severity levels, so warnings and errors are distinguished and only errors block. And a standing rule that a test failing for more than a defined period is either fixed or deleted, with deletion being an acceptable outcome. A test nobody will act on is documentation of an aspiration, not a control.
Everything is a view, until it should not be
Materialisation decisions made early rarely get revisited. A staging layer defined as views is elegant and cheap when the layer is thin. When there are eighty staging models and marts querying through several of them, the same source scans are being repeated many times per run, and the cost shows up in the warehouse bill rather than in the dbt logs, which is why it goes unnoticed for so long.
Reviewing materialisation across the project periodically is worth an afternoon a quarter. The rule of thumb we use is that anything queried by more than a handful of downstream models, or scanned repeatedly within a run, should be a table.
Ownership at the directory level
Below roughly a hundred models a single team can hold the whole project in its head. Above that, they cannot, and a project with no internal ownership boundaries becomes a place where changes are made nervously or not at all.
Once the domain structure exists, ownership can be attached to it mechanically. Code owners on directories mean changes route to the right reviewer without anyone deciding to involve them. Combined with exposure definitions marking which models feed which consumers, the review conversation becomes concrete: this change affects these three reports, owned by these people.
What we would fix first
- Reorganise by domain before the project passes two hundred models. It only gets more expensive.
- Incrementalise the ten most expensive models properly, with a documented restatement procedure.
- Clear every persistently failing test, by fixing or deleting, and introduce severity levels.
- Audit materialisation on anything with many downstream dependents.
- Attach owners to domains and wire review routing to them.
None of it is difficult. All of it is much easier before it becomes urgent, and the projects that stay pleasant to work in at five hundred models are the ones that did this work at a hundred and fifty.
What good looks like at scale
For a sense of the target rather than just the failure modes, the large dbt projects we consider healthy share a small number of properties.
A developer can build only what they need. Selector syntax, combined with a domain structure, means working on one area does not require rebuilding the project. This single property does more for development velocity than anything else on the list.
Changes come with a visible blast radius. Before merging, the reviewer can see which downstream models and which consumers are affected. Where exposures are defined properly, that includes named reports and named owners, which turns an abstract risk into a concrete one.
The default state of the test suite is green. Not mostly green with known exceptions. Green, such that red is genuinely informative and prompts action rather than a shrug.
And there is a routine for removal. Models get deprecated and deleted on a defined path rather than accumulating indefinitely. Every large project we have seen without a deletion habit carries somewhere between ten and thirty percent dead weight, and that weight is not merely wasted compute. It is models that appear in search results, get built on by mistake, and confuse everyone who joins.
None of these are exotic. They are ordinary software engineering practices applied to a codebase that happens to be written in SQL, and the projects that struggle are usually the ones where that framing never quite took hold.
