In one of my earlier roles, a major pharma client handed my team a non-negotiable requirement: they needed sub-second, real-time streaming for their global supply chain dashboard.
We spent a huge amount of time engineering a masterpiece. We set up an event-driven architecture, orchestrated micro-batch streaming and spun up a hefty Databricks cluster that hummed 24/7 to process transactions the millisecond they occurred. It was technically flawless and it cost a small fortune in compute.
A month after launch, I sat down with the supply chain manager to see how our real-time marvel was transforming his day-to-day operations.
I watched as he opened the dashboard, waited for it to load, clicked “Export to CSV,” and saved it to his desktop. “How often do you check this?” I asked. “Every Monday morning at 9:00 AM,” he replied. “I need the weekend totals for my weekly sync meetings.”
I then realised that we had built a Ferrari just so the client could drive it to the end of the street once a week.
If you look at a modern data architecture diagram today, it probably looks a lot like one from five years ago. You will see pipelines, layered transformations, semantic models, and a data lake sitting in the middle. The vendor logos change, but the boxes and arrows stay exactly the same.
This raises an uncomfortable question: technology has fundamentally shifted, so why hasn’t our approach to system design?
Historically, our architecture was dictated by hardware limitations. Storage was incredibly expensive, compute was scarce, and moving data across a network introduced crippling latency. We built complex, heavily decoupled systems to tiptoe around those constraints. But today, with decoupled storage and compute becoming the default, many of those historical bottlenecks are gone. Yet, we are still building solutions as if they are there.
Are we solving actual business problems, or are we just preserving legacy patterns because they feel safe?
The Medallion Obsession
One of the most reflexively applied patterns in our industry is data duplication. Raw data lands, we copy it to a curated layer, and then we copy it again to an aggregated layer.
We do this because the “Medallion Architecture” playbook tells us to. But every hop introduces compute costs, pipeline fragility, and latency. If you are operating in a modern ecosystem like Microsoft Fabric where OneLake allows you to shortcut traditional data movement through direct shortcuts and unified storage – blindly building out physically duplicated layers often does more harm than good.
Before building another pipeline to move data from point A to point B, we need to ask if the same outcome could be achieved with better logical modeling and views on top of a single copy of the data.
The “Real-Time” Illusion
We have a habit of letting technical capabilities override actual business needs. Take ingestion, for example. The moment streaming technologies became accessible, near real-time ingestion became the default requirement for almost every project.
We build complex event-driven architectures to process data continuously in micro-batches. But then you look at how the business actually consumes that data: an operations manager checks a PBI dashboard once a day at 9 AM.
That second-level freshness isn’t providing strategic value; it is just introducing file fragmentation, driving up processing overhead, and complicating our recovery protocols. Good architecture requires making trade-offs explicit. We should be designing for how the data is consumed, not just what the ingestion engine is capable of.
The Silent Killer: Data Layout
We spend weeks debating orchestration tools, pipeline frameworks and DAG structures. Yet, when it comes to how the data is physically stored on disk, the decisions are often rushed and rarely revisited.
Anyone who has spent time optimizing a Databricks cluster knows exactly how this plays out. You can write the most elegant Pyspark transformations in the world, but if your pipeline is generating thousands of 2Bb JSON files, your queries will choke on the metadata overhead. Poor partitioning strategies lead to massive, unnecessary data scans.
Instead of obsessing over how data flows through the system, we need to spend just as much time engineering how it rests. File sizing, partitioning, and z-ordering are what actually dictate performance when the data volume scales.
Architecture as a Living System
Our workloads are dynamic, but we design our systems as if they are static. We fix our partitioning strategies on day one and leave them alone. We reactively tune queries only when the business complains about timeouts.
Performance is no longer something we can just solve by throwing a larger compute cluster at the problem. As platforms become more abstracted, performance becomes tightly coupled to our foundational data design.
The real evolution in data architecture won’t come from adopting the next shiny platform or framework. It will come from moving away from “system-centric” design and asking better questions about how the data is physically structured, how it is actually consumed, and what trade-offs we are actively choosing to make.
Conclusion: Closing the Architecture Gap
The gap in modern data strategy isn’t technological; it is cognitive. Ultimately, our role isn’t just to build pipelines or implement the latest framework, it is to solve actual problems through intentional system design.
For the seasoned data architects out there: it is time to challenge the defaults we have grown comfortable with and stop designing for constraints that no longer exist. The next time you sit down at the whiteboard to design a system, take a step back and challenge them.
To the budding architects entering the field: do not just memorize the tools or blindly copy reference architectures. Learn to ask why. Master the trade-offs.
The most elegant system is rarely the one with the most moving parts; it is the one built on a deep, practical understanding of how the data will actually be used.