The data warehouse: not just for storage

The data warehouse: not just for storage

Approximately 70 percent of business data for large corporations still resides on mainframes

It's an undisputed fact that the growth and increased sophistication of data traffic in recent years has surpassed the predictions of early computer manufacturers. Many technology companies were surprised at the widespread acceptance of personal computers and the Internet as essential components for doing business. This acceptance made some quick to point a mocking finger at the big, solid mainframes that had made it all possible, and cry "Dinosaur!" In a rush to position themselves for an explosion of e-business activity, the PC advocates developed software that they claimed would enable businesses to archive and access data faster and less expensively than mainframes ever could.

The exponential growth of e-business and business-to-business (B2B) activity unfolded as expected—perhaps even more aggressively than anticipated—but migration from the use of mainframes for data management didn't. Approximately 70 percent of business data for large corporations still resides on mainframes, which have remained sound investments from their inception in terms of cost and dependability. For this reason, mainframes house the majority of today's data warehouses.

Any enterprise has a tremendous amount of activity taking place through its online systems, or operational systems. Once data has been used, it should be filed away for future reference. However, maintaining large amounts of such information on these active systems can seriously affect their performance. PC systems are quick and flexible, but have nowhere near the capability for storage of data that a mainframe offers. So a data warehouse structure is configured on the mainframe, designed to collect all the enterprise's transaction data and prepare it for analysis.

The database platform used by a data warehouse may be multidimensional, flat-file, object-related, hierarchical, or relational. Most data warehouses are deployed using a relational database, which derives its name from the term relations, applied to the two-dimensional tables in which data is displayed. Operational systems may also use a relational database model, but it's generally kept separate from that of the data warehouse because of normalization that takes place in the operational system. Normalization essentially breaks the relational database tables into smaller, independent tables, which provides flexibility for online operations but can make data management more complex for the data warehouse, affecting performance.

This concept of separation of data for business analysis from that in the operational systems is carried through in all aspects of the data warehouse. Though the data warehouse acts as a central repository for varied data about the enterprise's operations over time, the sources from which this information comes may be numerous, with differing systems, terms, and forms of data. It's therefore easier and safer to assemble this data in a place removed from the operational systems. As well, the data being entered into the warehouse is no longer dynamic. Data in the operational systems is being changed constantly as transactions take place, but the data warehouse stores research and analysis information that is no longer affected by the activities of the enterprise.

Because the data comes from such different sources, it needs to be adjusted for analysis purposes, using common terms and values. These sources may include such diverse areas of the business as sales, marketing, finance, and production. The adjustment is called "data scrubbing" or "data staging", and is basically a rewriting process to enable the data to be understood as though it has all come from the same source. This makes analysis much easier when the time comes to combine all the sources for an overall view of the business's activities.

Most activity involving the data warehouse, such as cross-referencing, is filtered according to time. One important advanced feature of data warehousing analysis is the ability to establish and understand how different organizational groups have performed at one time compared with another.

The use of data warehouses for query and reporting is still fairly new to many businesses, and at first most researchers use the very basic tools, perhaps to verify findings derived using older methods. The capabilities available through the data warehouse, however, are far-reaching and evolving rapidly. Multidimensional and graphical analytical tools are capable of powerful, in-depth analysis of complex data. These tools come in the form of desktop hardware and software. Almost all data warehouses are accessed by PC-based tools, in a client/server or multitiered computing architecture.

The data warehouse is often interfaced with other applications that use it as the source of operational system data

Here the capabilities promoted by PC developers as rivaling the mainframes come into play. Data warehouses can be made easily available across the Internet or company intranet, anywhere. Large-scale analysis can be performed using the data warehouse before being sent out to systems that would have had a hard time processing such information themselves. The data warehouse is often interfaced with other applications that use it in this way as the source of operational system data. The data warehouse may even feed information to smaller versions of itself, called data marts.

Personal computers certainly have their place in data analysis, and with the levels of performance available today can perform quick analysis of hundreds of gigabytes of information. However, after such analysis, the data is often fragmented and scattered, not having been used together as one source before. The data warehouse provides not just a powerful analytical tool, but a consistent source for maintaining enormous amounts of information as the business grows. And it is a certainty that enterprises using such technology will continue to grow.