Last time I walked through how we got from punch cards and mainframes to actual data warehouses — extract processing falling apart, OLTP systems never being built for analytics, all of that. This one picks up where that left off: you’ve got your warehouse built, data’s flowing in cleanly. Now what? That’s really where business intelligence and, eventually, big data come into the picture.
What it actually takes to make BI work
I’ve sat in enough planning meetings to know that “let’s do business intelligence” is where a lot of projects quietly go sideways, because nobody stops to define what that actually requires. A handful of things have to be in place before BI delivers anything useful.
First, and this sounds obvious but gets skipped constantly: an efficient data warehouse design that actually matches the business use case. Before anyone writes a single ETL job, somebody needs to answer why this data is being collected and what decisions it’s meant to support. Skip that step and you end up with a warehouse full of technically-correct data that answers none of the questions anyone’s actually asking.
Then there’s enterprise reporting — which tools, which applications, and critically, can users get at data ad hoc, or are they stuck waiting on someone to build a report every time a question comes up. Standard scheduled reports cover the routine stuff fine, but the moment someone needs “what happened yesterday” or “show me last week,” you need the ability to just go get that answer without submitting a ticket.
You also need proper support for relational and multidimensional analysis — this is where the star schema earns its keep, giving you multiple ways to slice the same data depending on what angle someone’s approaching it from. And you need integration figured out: are people accessing this through a dedicated application, through Oracle’s developer tooling, through something else entirely? That decision shapes a lot of what comes after it.
And then “easy administration” — a phrase I’ve never loved, honestly, because “easy” means something different to every stakeholder in the room. What it actually needs to mean in practice is that the warehouse runs efficiently without a mountain of manual maintenance steps, and that whichever business unit needs the data can get to it and start analyzing without a DBA holding their hand through every query.
Where big data forces its way into the conversation
At some point — and this conversation comes up with nearly every client I work with now — someone asks a version of: we’re launching a mobile app, we’re active on social media, can the warehouse actually handle all of that in something close to real time? And the honest answer is yes, most modern warehouses are built to handle unstructured, real-time data alongside the structured stuff. But it’s worth being specific about where that data is actually coming from, because it’s more sources than people initially assume.
Online shopping activity is the obvious one — and it’s grown enormously; I do plenty of my own shopping from my living room at this point, so it’s not exactly a surprising trend. But it goes well beyond that. Internet of Things devices are quietly generating a huge share of it: connected appliances tracking humidity or temperature in a warehouse, security systems, health monitors flagging something to a physician automatically, inventory systems adjusting stock counts the instant a register scans an item.
Logistics is another one people underestimate. I had a friend years ago working somewhere that received entire train cars of inventory — they’d scan the car, cross-check it against what was supposedly loaded, then scan the individual crates inside it against that manifest, across dozens of cars arriving from all over. Multiply that by how something like Amazon operates today — a handheld scanner registers a delivery and you get a notification before you’ve even checked your door. All of that is data hitting a warehouse somewhere, constantly.
Add social media activity — likes, clicks, shares — and you’ve landed squarely in what everyone calls big data. And the term isn’t just a buzzword; it describes a real set of characteristics: variety, meaning a mix of structured and unstructured or semi-structured formats like JSON stored directly rather than forced into rigid relational tables; volume and velocity, meaning genuinely large, fast-moving data streams; and veracity, which is the one people forget — not all of this data is reliable, and figuring out which of it is trustworthy enough to analyze is its own real problem.
Structured and unstructured data can absolutely coexist — a table can hold your clean structured records right alongside big data sources that don’t fit that mold at all. But that coexistence brings its own headaches: an unclear or unpredictable structure where you genuinely don’t know the shape of incoming data ahead of time, which is where tools like Oracle Big Data SQL earn their place — letting you query across structured and unstructured repositories together instead of treating them as two separate universes.
Then there’s the sheer speed requirement. Real-time capture means the data model has to be ready to store information the instant it’s generated — that package-scanned-to-email-notification pipeline only works because nothing in that chain is waiting on a batch job. And there’s volume at a genuinely different scale, warehouses running into multiple petabytes, which is partly why cheaper, lower-spec hardware for unstructured data storage makes financial sense — you’re not paying premium-tier storage costs for every byte of it.
Reliability adds a final layer of cost. Deciding whether to transform, filter, and authenticate data before it’s stored is a real tradeoff — doing that filtering properly is valuable, but it’s also additional workload and additional cost layered onto the environment.
Data lakes are not just warehouses with a different name
This is a distinction I still see people blur, so it’s worth being precise about it. Big data systems typically rely on data lakes — large-volume storage with no enforced model or schema. A data lake just holds data, structured, semi-structured, or fully unstructured, without imposing a shape on it up front.
A data warehouse works the opposite way. Structure gets applied before the data ever lands in a table — it’s already in strict, defined format by the time you’re querying it. A data lake, by contrast, applies structure at read time, when something actually goes to process it, which is a meaningfully different cost and performance tradeoff. Warehouses can absolutely store huge volumes too, but that tends to come at a real storage cost; lakes are generally the cheaper option for holding volume you’re not sure you’ll structure yet. In practice, most organizations I work with end up running both side by side, using each for what it’s actually good at rather than picking one and forcing everything through it.
Where this is all heading: the modern data warehouse
Here’s a question I get from clients constantly: they’ve got IT, data engineers, business analysts, and data science teams all pulling on the same warehouse with genuinely different needs, and they want to know if there’s a way to make all of that work together instead of constantly stepping on each other. That’s essentially what people mean now when they say “modern data warehouse” — one built with integrated machine learning capability and workflows designed around that mix of users from the start.
A modern data warehouse handles self-service data ingestion and transformation as data comes in, so it lands already consistent rather than needing cleanup after the fact. It supports SQL alongside machine learning, graph processing, and spatial queries — spatial being the “find me a gas station within five miles of where I’m standing” kind of question, which is a genuinely different query pattern than anything a traditional warehouse was built to handle well. It lets you run analytics against the data without physically moving it somewhere else first, and it automates a lot of what used to be manual provisioning and scaling work.
The practical upshot, from where I sit, is that a properly built modern warehouse removes a lot of the friction that used to eat entire sprints — analysts, engineers, and data scientists all get to spend their time on the actual analysis instead of fighting the plumbing to get at the data in the first place.
Tying it back together
Between this and the previous piece, you’ve now got the full arc: information management systems evolving from mainframes through decision support systems, OLTP’s real limitations for analytics, extract processing failing under its own weight, data warehousing solving that at the structural level, and now business intelligence and big data sitting on top of it all, pushing warehouses to handle volume, velocity, and variety that older architectures were never designed for. None of these pieces replaced the one before it entirely — they layered on top, each solving a problem the previous generation genuinely couldn’t.
