Factors of overengineering

2024 Aug 04

"Overengineering": I am not entirely sure what it is, but I am reliably told — and so I believed — it is a practice best avoided.

As a software developer, I think I have some kind of intuitive feeling about what is or is not overengineering. My intuition does not always align with other software developers' views, however. There seems to be significant subjectivity in the idea.

At worst, it boils down to: engineering for me, overengineering for thee. That is, engineering is when I understand my code, but overengineering is when I do not understand your code.

Then again, maybe a broad consensus around a subjective understanding of overengineering is just fine. A common-sense view — "I know it when I see it" — with room for negotiation may be enough for a working software developer to get by.

In general discourse, there are occasionally also some dissenting ideas. Sometimes the notion of "overdesigning" in engineering-engineering comes up (think hardhats, concrete, steel, wings, etc). There are safety margins, factors of safety, tolerances, capacity planning, or redundancy as a feature.

So when might erring on the side of overengineering or underengineering be favorable? The wisest accounts conclude that, "it depends". But depends on what? Leaving it to experience and intuition feels highly unsatisfactory.

Borrowing heavily from economics, here are some insights that might help to develop some kind of framework around the questions of what is or is not overengineering (or underengineering). It might begin to explain contradictory evidence of favorability of one or another approach to software development.

Non-optimality of overengineering or underengineering

At one level, overengineering and underengineering are the same: both non-optimal solutions.

Why does non-optimal engineering happen? Uncertainty and probablistic decision-making.

In real-world software development, it is rare that the optimal solution to a non-trivial problem is fully knowable ahead of time. If it were common, software estimation might be a generally solved problem. We are almost always dealing with an uncertain target — sometimes even a moving target, as requirements and circumstances evolve.

Uncertainty of the optimal solution means that any attempted solution is likely to be non-optimal. A non-optimal solution that builds too much might be described as overengineering. A non-optimal solution that builds too little might be underengineering.

Given those are the two most likely choices, a developer must decide which side to err on. It is not obvious that overengineering or underengineering is the better choice.

Intuitively, it might seem like overengineering must carry higher setup costs to develop (it takes more resources to build more stuff). But at least sometimes, the situation can be counterintuitive. Consider a web software project that can be overengineered by trivially adding a set of third-party software dependencies and rapidly bootstrapping an excessive frontend framework — low setup cost, compared to building up from scratch for a given estimated optimum. It might be that overengineering generally has higher setup costs, but that is just not universal.

The real difference emerges from iterating on the opening "bid". Software development is a meandering process, advancing toward the optimum in a series of moves starting from a non-optimal position.

Iterating through the elasticity of the medium

The question is which opening move is more advantageous for the iterative process? That is, whether overengineering or underengineering offers a superior path to reach an optimum.

But here we encounter a problem: elasticity.

If we are working in a medium that is elastic, then shifting toward the optimum is easier.

If our medium is highly elastic, all other things being equal, the setup costs may make the difference between overengineering and underengineering, and perhaps erring on the side of underengineering will be better.

But if the medium is highly inelastic, we may be better off with some kind of overengineering. (There, I said it!)

It may be too difficult to quickly iterate up from an underengineered solution, but it may be easier to draw upon some reserved potential stored in an overengineered solution.

Put on your hardhats for real engineering

Overdesign or overengineering in other fields of engineering can be a feature, not always a bug.

One hypothesis: fields like structural, civil, or aeronautical engineering commonly work with inelastic media, like soil, concrete, steel, water, wings, buildings, etc.

When building an urban stormwater drainage system, one might engineer for a 1-in-100 year overflow event. Probabilistically, "YAGNI" might imply you aren't going to need it. But when you need it, the inelasticity of the medium makes it impossible to quickly scale up — maximizing regret. The same can be said for increasing transport capacity, or maximum loading of an elevator, and so on.

The risks and impacts of failure also contribute to evaluations of elasticity. A structural or aeronautical failure may be difficult to recover from without the insurance of ahead-of-time redundancy and safety factors.

Software is soft

In software, there is entirely a different scale of elasticity: digital products are more elastic, as a rule.

However, there is relative elasticity within the field of software. Once upon a time, e-commerce web servers were provisioned pessimistically to handle peak loads like Black Friday or Christmas rushes. Server provisioning used to be inelastic. When a bet on anticipated high traffic fell through, over-provisioning could be costly. Ever since the emergence of the "elastic cloud", just-in-time scaling became viable. Servers can now be provisioned flexibly, on-demand, cutting the risk of underutilization.

So elasticity is not necessarily even a fixed property — an inelastic medium can become more elastic with new developments.

Nevertheless, there remains many media within software that are relatively inelastic. For example, database design and modeling: once large data sets get committed to a given database schema, migrations to change that schema become costly transactions. Barring some new invention to make databases more elastic, we have to cope another way.

Overengineering might be that way — at least some of the time.

Hedging options

Here lies another symmetry between underengineering and overengineering. Both are opening "bids" that approximate some uncertain optimum — indeed, a potentially evolving, moving target of an optimum. An overengineered solution or an underengineered solution should both be on the path to the optimum, even if the precise path is unknown.

In other words, non-optimal opening solutions are stores of path options toward the optimum target.

So at first glance, faced with uncertainty about the optimum, we might want to maximize options in solution design. Maybe more options are better — at least until we learn more about the optimum target. This is a kind of hedging play.

But hedging can be argued for either overengineering or underengineering.

The best case for overengineering is that it materializes a store of options (future bids) in the encoded solution. Then these realized options are already available by the time they need to be exercised. The best case for underengineering is that it reserves options by delaying decisions as late as possible.

When making decisions would exhaust options, underengineering can maximize preserved options (elastic scenaro). When non-decisions mean exhausting options, overengineering can maximize preserved options (inelastic scenario).

Stock and flow

Where have we seen this duality of options plays elsewhere? Consider the liquidity-inventory dichotomy from economics.

When we overengineer a solution, we materialize options as stock, building up our software inventory. When we underengineer a solution, we maintain liquidity to avoid investing in liabilities.

The elasticity of the medium can change the nature of timing: inelasticity favors early commitment because it naturally exhausts options in the course of development.

Assets and liabilities

So far we have presumed that building more stuff in stock gives more options toward an eventual optimum. But that may not be the case, because we could always build more of the wrong stuff — features that do not serve as options to any likely optimum. Our software in stock then accumulates as liabilities, one of the risks of overengineering. By analogy, this is like investing in poorly-performing stocks.

Somewhat counterintuitively, a similar problem can also occur when underengineering. Unrealized assets can amount to an opportunity cost. And when options naturally get exhausted (as in development in inelastic media) those opportunity costs can close out paths to an optimum. By analogy, this is like failing to invest in well-performing stocks.

Path dependency properties of any chosen approach can mean that iterating toward an optimum target may be easier or harder, or even impossible. In the face of uncertainty, an asset (liquid or fixed) would maximize options with actual paths to an eventual optimum.

Solving for "it depends"

So the choice of design strategy between overengineering and underengineering may rest on a number of factors.

1. The operating environment for software development begins with more-or-less high uncertainty, forcing non-optimal moves.
2. Overengineering or underengineering are inherently neutral strategies for non-optimal opening bids. (There is no universal choice.)
3. Setup costs may be higher or lower for either approach depending on circumstance. (There is still no universal choice.)
4. The elasticity of the medium is critical to the choice of strategy. (And elasticity itself may not be fixed over time.)
5. Faced with elasticity or inelasticity, the strategic choice that hedges by maximizing good options, may be best.
6. Optimizing for options essentially mirrors the liquidity-inventory problem (and solution models) in economics.

So just as sometimes the best bet might be to maintain liquidity or to invest in inventory — it depends! — a software developer's best bet might be to err on the side of underengineering or overengineering in any given situation.

Even so, we might be able to make some generalizations. Software being generally of an elastic nature might lead to a sensible default meta-strategy: YAGNI.

While that might be good default advice, for non-default cases, the considerations given here might add more depth when needed.

Working in web software stacks, I might be more inclined to hedge by overengineering a database schema decision — for example, denormalizing a table relationship tends to be easier than normalizing one. But I might err on the side of underengineering a web frontend architecture until the need for some dependency or other emerges more clearly, because it is easier to add than to remove.

While those may seem like common-sense choices to some, the method reflects the factors of overengineering that I hope might at least begin to standardize discussion when there is disagreement.

I suspect subjective uncertainty also helps to explain why sometimes developers might opt to overengineer in domains that are unfamiliar to them. Having uncertainty about the optimum solution is one thing, but also being uncertain about the elasticity of the medium might favor hedging by overengineering. A more seasoned practitioner equipped with greater certainty, ability to estimate, and comfort in the elasticity of the medium in that domain, will not rely so much on overengineering.

Is overengineering good or bad? Yes. (It depends.)