Can The Data Warehouse Survive?

Data warehouses are being deployed at an alarming rate. But, has anyone stopped to consider if they, too, are fast becoming — as dinosaurs — extinct?

No matter how well planned, the data warehouse will, without doubt, cease to evolve. Based on current models it will be unable to adapt and serve the ever increasing information demands that organizations require. Corporations acknowledge that information is their most valuable asset, but this asset, due to its exponential growth, is causing an ever-increasing burden on the organization itself.

Industry standard practices for data storage are showing its true limitations. Disk and disk server technology can no longer serve the business practice. Can the warehouse cope with these ever increasing demands? How will it manage the ever-varying type of data that corporations store against their customers? If disk and disk server technology has failed to meet organizations criteria, due to ever increasing volumes and the need for instant access; how then will the warehouse manage to provide the flexibility necessary to meet the differing needs of the organization?

Data warehouses, regardless of the database type or structure, suffer from one common problem: they get large. Statistics show that the typical growth of data used for warehousing will rise from 393 GB to 1.1 TB in less than three years. Warehouse spending is anticipated to grow from $37.4 billion to $148.5 billion by 2003. The growth of data deposits and the spiraling costs associated with managing them will continue to accelerate at new and alarming rates; so how will the deployment of these new dinosaurs manage to deliver all that they promise?

Had any of us envisioned the explosive growth and subsequent reliance on information? Five years ago, when we had Oracle 5 and were grateful, did we even have the notion of data warehouses with many hundreds of gigabytes of data? If we had taken data out of our database then, could we restore it now?

What we thought possible just five years ago has been superseded. Technology is everywhere; no one can escape it. There is no doubt that change is rapid within the field of technology, over the next five years there will be an ever increasing use and dependency on non-structured data, which will add a huge load to the data warehouse model in use today. With that certain knowledge, it is estimated that just 20% of stored data will come from traditional structured OSS databases.

As sources of data keep growing, the problem becomes exponential. If you're successfully using the warehouse - it will continue to grow, which will undoubtedly lead it to become the victim of its own success.

The warehouses principal function is to be the central point for long-term data storage, a method that is preferable to retaining it in fragmented operational systems. This single point enables users to go directly to where historic details about a customer and his transactions are stored. But, many organizations will find that the continued exponential growth of data and the associated data management costs will ensure that the warehouse will become unpractical. The warehouse will, because of these constant demands, be too expensive, too slow, too large - and like the dinosaurs before them, the warehouse will die, and with it the companies legacy information.

Warehouses will continue to be developed, but as history serves to remind us, in order to evolve and compete one must adapt to change.

So it is now not a question of to warehouse or not to warehouse – it is about an organizations ability to understand its business needs; recognizing that current practices, based on physical rules data management, can no longer serve the business practice if they are to meet their operational imperatives and retain their competitive edge.

This knowledge leads to the overwhelming need to adopt new practices, implement new, 'future proof ' data management strategies, that can safeguard an organizations most valuable asset. Offering optimized levels of service that provide the means to manage and contain escalating data volumes based on business rules logic.

Implementing business rules driven data management strategies have proven themselves as the only practicable solution that can meet corporations complex data management needs.