Table of Contents Preface I: Introduction I: Introduction 1. The Enterprise IT Architecture The Past: Evolution of Enterprise Architectures The Present: The IT Professional's Responsibility Business Perspective Technology Perspective Architecture Migration Scenarios Migration Strategy: How Do We Move Forward? In Summary 2. Data Warehouse Concepts Gradual Changes in Computing Focus The Data Warehouse Defined The Dynamic, Ad Hoc Report The Purposes of a Data Warehouse A Word About Data Marts A Word About Operational Data Stores Data Warehouse Cost-Benefit Analysis / Return on Investment In Summary II: People II: People 3. The Project Sponsor How Will a Data Warehouse Affect our Decision-Making Processes? How Does a Data Warehouse Improve My Financial Processes? Marketing? Operations? When Is a Data Warehouse Project Justified? What Expenses Are Involved? What Are the Risks? Risk-Mitigating Approaches Is My Organization Ready for a Data Warehouse? How Do I Measure the Results? In Summary 4. The CIO How Do I Support the Data Warehouse? How Will My Data Warehouse Evolve? Who Should Be Involved in a Data Warehouse Project? What Is the Team Structure Like? What New Skills Will My People Need? How Does Data Warehousing Fit into My IT Architecture? How Many Vendors Do I Need to Talk to? What Should I Look for in a Data Warehouse Vendor? How Does Data Warehousing Affect My Existing Systems? Data Warehousing and Its Impact on Other Enterprise Initiatives When Is a Data Warehouse Not Appropriate? How Do I Manage or Control a Data Warehouse Initiative? In Summary 5. The Project Manager How Do I Roll Out a Data Warehouse Initiative? How Important Is the Hardware Platform? What Technologies Are Involved? Do I Still Use Relational Databases for Data Warehousing? How Long Does a Data Warehousing Project Last? How Is a Data Warehouse Different from Other IT Projects? What Are the Critical Success Factors of a Data Warehousing Project? In Summary III: Process III: Process 6. Warehousing Strategy Strategy Components Determine Organizational Context Conduct Preliminary Survey of Requirements Conduct Preliminary Source System Audit Identify External Data Sources (If Applicable) Define Warehouse Roolouts (Phased Implementation) Define Preliminary Data Warehouse Architecture Evaluate Development and Production Environment and Tools In Summary 7. Warehouse Management and Support Processes Define Issue Tracking and Resolution Process Perform Capacity Planning Define Warehouse Purging Rules Define Security Measures Define Backup and Recovery Strategy Set Up Collection of Warehouse Usage Statistics In Summary 8. Data Warehouse Planning Assemble and Orient Team Conduct Decisional Requirements Analysis Conduct Decisional Source System Audit Design Logical and Physical Warehouse Schema Produce Source-to-Target Field Mapping Select Development and Production Environment and Tools Create Prototype for This Rollout Create Implementation Plan of This Rollout Warehouse Planning Tips and Caveats In Summary 9. Data Warehouse Implementation Acquire and Set Up Development Environment Obtain Copies of Operational Tables Finalize Physical Warehouse Schema Design Build or Configure Extraction and Transformation Subsystems Build or Configure Data Quality Subsystem Build Warehouse Load Subsystem Set Up Warehouse Metadata Set Up Data Access and Retrieval Tools Perform the Production Warehouse Load Conduct User Training Conduct User Testing and Acceptance In Summary IV: Technology IV: Technology 10. Hardware and Operating Systems Parallel Hardware Technology Hardware Selection Criteria In summary 11. Warehousing Software Middleware and Connectivity Tools Extraction Tools Transformation Tools Data Quality Tools Data Loaders Database Management Systems Metadata Repository Data Access and Retrieval Tools Data Modeling Tools Warehouse Management Tools Source Systems In Summary 12. Warehouse Schema Design OLTP Systems Use Normalized Data Structures Dimensional Modeling for Decisional Systems Two Types of Tables: Facts and Dimensions A Schema Is a Fact Table Plus Its Related Dimension Tables Facts Are Fully Normalized, Dimensions Are Denormalized Dimensional Hierarchies and Hierarchical Drilling The Time Dimension The Granularity of the Fact Table The Fact Table Key Concatenates Dimension Keys Aggregates or Summaries Dimensional Attributes Multiple Star Schemas Core and Custom Tables In Summary 13. Warehouse Metadata Metadata Are a Form of Abstration Why Are Metadata Important? Metadata Types Versioning Metadata as the Basis for Automating Warehousing Tasks In Summary 14. Warehousing Applications The Early Adopters Types of Warehousing Applications Financial Analysis and Management Specialized Applications of Warehousing Technology In Summary V: Where to Now? V: Where to Now? 15. Warehouse Maintenance and Evolution Regular Warehous Loads Warehouse Statistics Collection Warehouse User Profiles Security and Access Profiles Data Quality Data Growth Updates to Warehouse Subsystems Database Optimization and Tuning Data Warehouse Staffing Warehouse Staff and User Training Subsequent Warehouse Rollouts Chargeback Schemes Disaster Recovery In Summary 16. Warehousing Trends Continued Growth of the Data Warehouse Industry Increased Adoption of Warehousing Technology by More Industries Increased Maturity of Data Mining Technologies Emergence and Use of Metadata Interchange Standards Increased Availability of Web-Enabled Solutions Popularity of Windows NT for Data Mart Projects Availability of Warehousing Modules for Application Packages More Mergers and Acquisitions Among Warehouse Players In Summary VI: Appendices VI: Appendices A. R/ OLAP XL® User's Manual Welcome to R/ OLAP XL! Installation Tutorial User's Guide Working with R/ OLAP XL Columns Setting R/ OLAP XL Options The R/ OLAP XL Toolbars Macro Programming R/ OLAP XL Messages B. Warehouse Designer® User's Manual Welcome to Warehouse Designer! Basic Consepts The Warehouse Designer Toolbars Applications Dimensions Schemas Custom Dimensions Custom Schemas Aggregate Dimensions Aggregate Schemas C. Online Data Warehousing Resources C. Online Data Warehousing Resources D. Tool and Vendor Inventory D. Tool and Vendor Inventory E. Software License Agreement Preface This book is intended for Information Technology (IT) professionals who have been hearing about or have been tasked to evaluate, learn or implement data warehousing technologies. Far from being just a passing fad, data warehousing technology has grown much in scale and reputation in the past few years, as evidenced by the increasing number of products, vendors, organizations, and yes, even books, devoted to the subject. Enterprises that have successfully implemented data warehouses find it strategic and often wonder how they ever managed to survive without it in the past. As early as 1995, a Gartner Group survey of Fortune 500 IT managers found that 90 percent of all organizations had planned to implement data warehouses by 1998. Virtually all Top-100 US banks will actively use a data warehouse-based profitability application by 1998. Nearly 30 percent of companies that actively pursue this technology have created a permanent or semipermanent unit to plan, create, maintain, promote, and support the data warehouse. If you are an IT professional who has been tasked with planning, managing, designing, implementing, supporting, or maintaining your organization's data warehouse, then this book is intended for you. The first section introduces the Enterprise Architecture and Data Warehouse concepts, the basis of the reasons for writing this book. The second section of this book focuses on three of the key People in any data warehousing initiative: the Project Sponsor, the CIO, and the Project Manager. This section is devoted to addressing the primary concerns of these individuals. The third section presents a Process for planning and implementing a data warehouse and provides guidelines that will prove extremely helpful for both first-time and experienced warehouse developers. The fourth section of this book focuses on the Technology aspect of data warehousing. It lends order to the dizzying array of technology components that you may use to build your data warehouse. The fifth section of this book opens a window to the future of data warehousing. This book also comes with a CD-ROM that contains two software products. Please refer to the readme.txt file on the CD-ROM for any last minute changes and updates. The enclosed software products are: • R/olapXL®. R/OLAPXL is a powerful query and reporting tool that allows users to draw data directly into Microsoft Excel spreadsheets from any dimensional data mart or data warehouse that resides on an ODBC-compliant database. Once the data are in Microsoft Excel, you are free to use any of Excel's standard features to analyze, report, or graph the retrieved data. • Warehouse Designer®. Warehouse Designer is a tool that generates DDL statements for creating dimensional data warehouse or data mart tables. Users specify the required data structure through a GUI front-end. The tool generates statements to create primary keys, foreign keys, indexes, constraints, and table structures. It recognizes key dimensional modeling concepts such as fact and dimension tables, core and custom schemas, as well as base and aggregate schemas. Also enclosed is a License Agreement that you must read and agree to before using any of the software provided on the disk. Manuals for both products are included as appendices in this book. The latest information on these products is available at the website of Intranet Business Systems, Inc. The URL is http://www.intranetsys.com. Part I: Introduction The term Enterprise Architecture refers to a collection of technology components and their interrelationships, which are integrated to meet the information requirements of an enterprise. This section introduces the concept of Enterprise IT Architectures with the intention of providing a framework for the various types of technologies used to meet an enterprise's computing needs. Data warehousing technologies belong to just one of the many components in an IT architecture. This chapter aims to define how data warehousing fits within the overall IT architecture, in the hope that IT professionals will be better positioned to use and integrate data warehousing technologies with the other IT components used by the enterprise. [...]... in an orderly fashion (see Figure 1- 11) The workflow management system alerts users through the automatic generation of notification messages or reminders and routes work so that the desired business result is achieved in an expedited manner Figure 1- 11 Process Implementation Figure 1- 12 highlights how this approach fits into the Enterprise Architecture Figure 1- 12 Process Implementation: Architectural... applications (or OLAP) that obtain data from the data warehouse are recommended for this particular need The data warehouse holds transformed and integrated enterprise-wide operational data appropriate for strategic decision-making, as shown in Figure 1- 13 The data warehouse also contains data obtained from external sources, whenever this data is relevant to decision-making Figure 1- 13 Decision Support Decision... information (see Figure 1- 15) Figure 1- 15 Hyperdata Distribution Web technology allows users to display charts and figures; navigate through large amounts of data; visualize the contents of database files; seamlessly navigate across charts, data, and annotation; and organize charts and figures in a hierarchical manner Users are therefore able to locate information with relative ease Figure 1- 16 highlights how... legacy systems with the rest of the architecture is best achieved through the Operational Data Store and/or the data warehouse Figure 1- 7 modifies Figure 1- 5 to show the integration of legacy systems Figure 1- 7 Legacy Integration Legacy programs that produce and maintain summary information are migrated to the data warehouse Historical data are likewise migrated to the data warehouse Reporting functionality... Decision support applications analyze and make data warehouse information available in formats that are readily understandable by decision-makers Figure 1- 14 highlights how this approach fits into the Enterprise Architecture Figure 1- 14 Decision Support: Architectural View Hyperdata Distribution The Need Past informational requirements were met by making data available in physical form through reports,... modern Enterprise Architectures The two orthogonal perspectives of business and technology are merged to form one unified framework, as shown in Figure 1- 3 Figure 1- 3 The InfoMotion Enterprise Architecture Business Perspective From the business perspective, the requirements of the enterprise fall into categories illustrated in Figure 1- 4 and described below Figure 1- 4 The Enterprise Architecture (Business... frames, therefore requiring reconciliation Table 1. 1 Migration of Legacy Functionality to the Appropriate Architectural Component Functionality in Legacy Systems Should be Migrated to Summary Information Data Warehouse Historical Data Data Warehouse Operational Reporting Flash Monitoring and Reporting Tools Data for Operational Monitoring Operational Data Store Decisional Reporting Decision Support... client applications that update the same database Through active databases, applications are more robust and conducive to evolution Operational Data Stores An Operational Data Store or ODS is a collection of integrated databases designed to support the monitoring of operations Unlike the databases of OLTP applications (that are function oriented), the Operational Data Store contains subject-oriented,... views of data in operational systems Data are transformed and integrated into a consistent, unified whole as they are obtained from legacy and other operational systems to provide business users with an integrated and current view of operations (see Figure 1- 5) Data in the Operational Data Store are constantly refreshed so that the resulting image reflects the latest state of operations Figure 1- 5 Legacy... processes Decisional Needs Data Warehouse The data warehouse concept developed as IT professionals increasingly realized that the structure of data required for transaction reporting was significantly different from the structure required to analyze data The data warehouse was originally envisioned as a separate architectural component that converted and integrated masses of raw data from legacy and other . In Summary 13 . Warehouse Metadata Metadata Are a Form of Abstration Why Are Metadata Important? Metadata Types Versioning Metadata as the Basis for Automating Warehousing Tasks. a Data Warehouse Initiative? How Important Is the Hardware Platform? What Technologies Are Involved? Do I Still Use Relational Databases for Data Warehousing? How Long Does a Data Warehousing. Extraction Tools Transformation Tools Data Quality Tools Data Loaders Database Management Systems Metadata Repository Data Access and Retrieval Tools Data Modeling Tools Warehouse Management