The Complexity Paradox in Software Engineering
Where do we put complexity in software? Does shifting it around help?
There's a pattern I've observed repeating throughout software development over the years. Two related patterns, actually:
- We continually attempt to move complexity around, convincing ourselves this somehow alleviates it.
- We strive to make things simple but inevitably end up creating equivalent complexity once our solutions address all the necessary requirements.
Shifting the Complexity Burden
A decade ago, I wrote an article titled "Microservices: Squeezing the Balloon." It explored how microservices seemed like an elegant solution, but in reality, they merely shifted complexity from individual engineers to systems administrators. While microservices made individual software components easier to maintain, someone still had to manage their deployment, monitoring, and interaction patterns—especially the critical interactions between services.
The Feature Bloat Cycle
On the other end of the spectrum, we see products that achieve tremendous success—even reaching unicorn status—because they initially do one thing exceptionally well. Then, customer by customer, they accumulate widgets, warts, dongles, and complexities. Eventually, these once-streamlined products become unrecognizable and unmanageable. Salesforce and various click-metrics tools exemplify this pattern.
The Cloud Native Paradox
Somewhere in the middle, we have technologies like serverless computing and containerized applications. Together, these approaches have spawned an entire industry called "cloud native."
Remember when you were excited about dropping some Lambda code into a web text box and instantly having a hosted service running in the cloud? Unless you've just done this for the first time, that excitement has likely faded. The complexity of orchestrating serverless functions to create a functional application is enormous!
Consider all the additional concerns: networking, API gateways, exposability, role-based access, environment configuration—all these challenges now exist on a per-function basis rather than a per-application basis.
Container Complexity
Perhaps you decide to containerize because functions are too granular to manage effectively. Now you're working with ECS or Kubernetes—entirely new conceptual frameworks to understand and maintain, complete with their own libraries and templating systems.
Both serverless and containerized approaches now require some form of service mesh to handle cross-cutting concerns.
Spaghetti Everywhere
While well-intentioned, these approaches have created a spaghetti nightmare in our operational and compute space, rather than confining the mess to our codebase. Except we still have code complexity because we must locate particular pieces of code that might live in entirely different repositories, not just different classes.
The New Vendor Lock-In
We used to worry about vendor lock-in and prioritize application portability. Containerization was supposed to solve this for us. Yet "cloud native" has become the new vendor lock-in. Even if you're using Kubernetes, which is similar across clouds, aspects like ingress and access control tend to be cloud-provider specific. And you're paying a premium for your compute resources on top of it all.
AI Frameworks: Kitchen-Sink Complexity
The latest evolution in complexity comes from AI frameworks. Many popular tools—like LangChain, Pinecone, and LangGraph—bundle dozens or even hundreds of integrations, implementations, and tools. They offer opinionated, fixed approaches that can sometimes require more workarounds than seamless integration. Often, these frameworks include features hidden behind paywalls, complicating budgeting and decision-making.
When integrating AI into engineering, the kitchen-sink approach of these frameworks might seem appealing initially, but the resulting overhead can quickly outweigh the benefits. Particularly in AI-augmented engineering, it is frequently faster, simpler, and ultimately more cost-effective to build precisely what you need and nothing more.
I don't claim to have all the answers. These are observations from years in the field. But I do have many ideas about potential paths forward.
And we haven't even discussed data yet... Perhaps that's a topic for next time.