- DWAI launches DWAI portal for information growth.- DWAI launches DWAI online training with detailed Data Modeling training facility.

- Certification in collaboration with world wide associate member with assistance in jobs,placement, internships.

- Launching online testing certification module in collaboration with Australian branch of technology training and acknowledged member in Australia.

We have successfully build and documented BIG DW projects for our member groups and consulting companies who have approached us for helping them build the DW successfully.

The general notion which the industry clients have is the track record where many IT consulting companies have failed in IMPLEMENTATION of DW projects.

The need for Analysis is across all domains and all industries require people with skill sets who can analyze data, Data analysis is a true art which can be nurtured with excellent training programs and also providing an environment for aspirants to understand the GOLDEN RULES of the process to view and understand data, we have over the years of studies and implementation in DW brought down on the book the Successful GOLDEN RULES to achieve and accomplish a successful DW process.
We pick up randomly and on basis of sole discretion by the Board of Members to pick and provide INTERNSHIP to candidates who are extra ordinary and which may require this facility, we provide to the INTERN an opportunity to work with the experts on a project which may be going on at that particular time.
It would be the sole decision of the Project manager to give the Intern any job in the Project. We over the years have also provided Internships to upcoming candidates in within our ongoing projects
Internship program is a live training on the job process which we undertake at a random basis with our new members and at our own discretion.

With our new facility on Online DW trainings already proving benefits to so many we wish to achieve the WEB SEMINAR section soon
WebSeminar:

Venue:-New Delhi
Date :January 18, 2009

Venue:-Mumbai
Date :March25, 2009

Click for More Conferences and seminar by DWAI.

We found out that DW is not known as a career opportunity although the requirement of Analytics professional is in all Domains as compared to professionals in other technologies where the demand is only in technology companies.

So we are presenting a e-book section where students can view e-books on DW and other related technologies where there knowledge would upgrade, this section is FREE for all members and we also wish that members having information or e-books which can be benefiting to others, they upload the same to us, so that our cause to expand DW knowledge can be taken to new heights.

Books Available:

. PL/SQL User guide and refrence(10g)………Download
. Microsoft  Sql Server Black Book……………..Download
. Teach Your Self SQL in 21 days………………..Download

DWAI provides monthly newsletter to its members which is an option to get updated on new and high end technological changes in the DW environment, BI and Related Technologies.This section is created with the voluntary help and guidance given to us by the Elite group of members having decades of Experience in the DW environment and who have worked with majorly all industry domains and virtually all continents of the Globe.

Please subscribe for Newsletter:

E-mail Address

1. Why Do Data Warehouse Projects Run Into Scope Creep?

To quote Bill Inmon (guru and author of several great books on Data Warehousing) “Traditional projects start with requirements and end with data. Data Warehousing projects start with data and end with requirements.”
As soon as the project gets under way, users will find new applications, and with it will come new requests for data. Interestingly, these projects often are justified by moving Q&R work away from the ‘data people’. What we’ve seen is that the first thing that happens as soon as the project delivers is that more requests for special queries are submitted to these same ‘data people’. This may appear to undermine the initial business case but actually signals the onset of value creation from the DWH project.

2. Star Schema Versus Entity Relation Model?

There has been enormous debate in the community about the merits of different data models. At the risk of over simplifying: ER models tend to have better performance (processing time) for the end user, when they are accompanied by dedicated user marts (the Inmon model) and are often perceived as “easier” to understand by end users. Drawbacks are that ER models require more disk space, and, because of the intrinsic redundancy in the data, have consistency problems from a maintenance perspective.
Having said this, the practice seems to be that often some combination of the two is unavoidable in the practical setting, despite preferences (ER or Star) of the chief architects. Overall, Star models seem to have gained the most ground.

3. The Importance of a Data Warehouse Business Case:

Much has been written about the business case for a Data Warehouse. What goes in to a good business case? IT savings are ubiquitous in DWH business cases. The important point is to not limit this to ‘pure’ savings, but to connect to primary business processes as much as possible. As an example, faster turnaround cycles for list selections are fine (when quantified in hourly rates), but it is even better if the revenue from more customer acquisitions that follow from these selections can be tied in. Not only will the relation to revenue growth rather than savings make for a more balanced business case, more important is the intrinsic business buy-in that results from a direct connection to the company bottom line.
These days, changes in legislation (in particular Sarbanes-Oxley) play a major role in justifying business cases. This may be either through a higher company valuation for its transparent information gathering, or, less sleepless night for the CEO, which is of course priceless…

4. Why Do Data Warehouse Projects ‘Never’ Go Wrong?

Actually, Data Warehouse projects do sometimes fail. But, they fail so rarely, that it is actually very hard to believe… Especially after having talked to so many disgruntled end-users. And there are many ways a Data Warehouse project can go wrong. Delivering on time, data administration issues, and unavoidable data quality issues in feeding systems.
Corporate politics (see Tip 7) are probably the best explanation for this phenomenon of near 100% success rates on DWH projects. In my experience, the reason why a failure or ‘semi-failure’ can go unnoticed is either because senior management is not aware, or, let’s say “unmotivated” to talk about misspending of company funds. As a result, not enough is learned. Maybe we as consultants have a stake in this as well, as this assures the industry plenty of ongoing business…

5. What is Different About Warehousing Web Data?

Kimball & Merz (2000): “Although this clickstream data in many cases is raw and unvarnished, it has the potential of providing unprecedented detail about every gesture made by every human being using the Web medium”. The subatomic nature of clickstream data poses unique challenges. There are fewer built in feedback mechanisms to ensure data quality, compared to other data streams. The relation between user mouse clicks and server log records is not as tight as in “traditional” transaction processing due to technical issues like proxy servers and caching. Because of these differences, IT people need to adapt to the web process flow, rather than having the process adapt to IT needs as is common for most other DWH interfaces.

6. Which Data Should Be loaded In The Data Warehouse?

The data that enter the DWH ultimately determine its place in the organization. A “let’s load all data, to be safe”-attitude is a sure fire way to derail your DWH project. Choices as to what should and should not be included need to be made early on, to keep the project manageable.
After proven success of the delivered, deployed, and profitably exploited DWH, there always will be funding somewhere to include previously ignored interfaces. Given the anticipated lifecycle of the DWH, it makes perfect sense to consciously exclude certain sources.
The choice as to what data to include needs to be driven by business considerations, and in particular reference to the company bottom line. If it can’t be shown how data will be put to use profitably, they stay out! See also tip .

7. Data Warehousing & Company Politics

Data Warehouses have an impact on the company bottom line. Hence, they are likely candidates for turf battles, and are also at risk of becoming “small change” in budget allocation negotiations. None of these considerations benefit corporate long term goals. Managing a DWH project is hard enough as it is, and budget issues shouldn’t make it any harder than it already is.
Because DWH investments are in the present and revenues lie in the future, it is even more important to secure funding through a sound business case and buy-in from the appropriate (high) management level. See also Tip #3.
Access to data means power, and talking about power is one of the greatest management taboos, still around. Sensitive as they are, even budgets are more readily discussed…

8. Data Warehouse Projects Traps:

Some commonly recurring ‘roadblocks’ on the path to timely delivery of a Data Warehouse project:
• ETL processes have eaten up so much time (and still need “babysitters”), that little if any time is left to develop applications needed to exploit the DWH
• Some data are needed, but turn out not to be unavailable, or not in a timely fashion
• Maintenance required for tuning, indexing, and backup and recovery is severely underestimated
• Different ways of calculating the same phenomenon lead to different results, and nobody is able to conclusively explain the difference(s)
• The data that is loaded (and recombined) turn out to contain previously unknown inconsistencies in the source systems, the ‘classic’ data quality issues that trip DWH projects
• Metadata were lacking, and developers spend inordinate amounts of time finding out what a field really ‘means’

9. DWH Hardware and Software Go Hand in Hand:

In Data Warehousing, it is not about hardware, and not about software: it is about the perfect integration of these two. Those who begin their project from either end, will pay dearly for this mistake. Reasons are:
• in terms of price/performance, new, pre-integrated hardware-software combinations are taking the lead
• from a project management perspective, you never want to be caught between vendors when a proposed solution doesn’t work as expected
• database tuning and indexing is very important and a hugely complex job, necessarily left to specialists (in-house trained)

10. Performance is Key:

Although I don’t often find technology factors to be this important, in Data Warehouse acceptance, no other factor will be as important as performance. As size increases over time, this factor becomes even more important.

There are three reasons for this:
1. performance has a huge impact on the development speed (initial load is always very time consuming), and hence the overall maturity of the DWH at delivery time
2. performance can make or break end-user acceptance, in particular the predictability of performance
3. performance has a tremendous impact on end user productivity, the ultimate driver of the business pay-off

1. Money (and thus, governance)
2. Design,
3. Heritage, and
4. Data.

Money:

A Dutch proverb says, “He who pays for the music will determine the song that is played.” Money for IT projects is raised by the business based on the added value as it is perceived by the future users of the results of the projects. At least, that should be the way it works. In case of a data warehouse, the future user group consists of the management and their supporting staff. The logic of an enterprise data warehouse, the single version of the truth and the optimization of the data flows between sources and the BI application, is in itself clear and in fact indisputable. But then another phenomenon comes into play, and for this one of my former managers had a saying: “Managers and children have one thing in common, once they want something they want it right away.” An enterprise data warehouse is just one of those things you can’t have right away. On top of that, managers have an information need, which – for the most part – is fluid, subject to fast and unexpected changes and of an unpredictable nature. At the same time, this information need has to be satisfied now. And so the temptation is large to built or maintain – in spite of the bigger picture – another homegrown shortcut next to the enterprise initiative.
So, although the individual manager will support the idea of an enterprise data warehouse, his willingness to fund it from his project (“the first pedestrian paying for the whole bridge”) will not be really great. The same is true for his willingness to wait for the fulfillment of his information need until the enterprise data warehouse is up to it.
The lesson here is, funding of the enterprise data warehouse must be decoupled from the individual BI projects. It is a kind of basic infrastructure (“the plumbing and sewage”) which, only after a considerable threshold investment, can play a role in the factual information delivery towards its customers. Apart from that, the actual completion of the data warehouse, which data coming from which systems will be included when, must be determined by the priority and willingness to participate of the various BI projects. You have to avoid the data warehouse becoming a solution in search of a problem.

Design:

A second reason why it is so difficult to succeed with an enterprise data warehouse project can be found in the internal design of it. One of the most important requirements is the capability to follow the changes in the business with the pace dictated by the business. Managers want their information, and no matter which internal or external changes will occur, they want an undisrupted continuation of their information flow, including answers to questions coming from those changes. They want answers to questions that will arise only when the answers to the previous questions are on their desk.
In case the data warehouse organization is not capable of following the required pace, the users will once more drop out to set up their own information provision next to the data warehouse. The data warehouse organization is in fact a logistic service provider that ensures that data originating in a number of source systems can be used – within the given time to market – as meaningful information in a number of target systems. Therefore, a logistics process needs to be designed and implemented, which consists of collections points, means of transport, decoupling points where (temporary) storage and processing takes place and distribution points. While designing the processes, systems and applications for this logistic process, one should – in order to obtain the necessary flexibility and reduce change resistance – take into account the following four design principles:

Design Principle:

1. Generic instead of specific,
2. Reusable instead of one time,
3. Decoupling instead of integration, and
4. Federation instead of centralization.

Generic instead of Specific
In the end, a data warehouse is nothing more than just another database that consists of a number of physical (relational) tables. Such a physical database model is based on a logical database model in which the various relations between the different information elements are described. The logical database model, in its turn, is (or at least should be) based upon the information or business model. This information model represents the way users are looking at their world.
The most far-reaching changes in any enterprise data warehouse have to do with changes in this information model. Following this are changes in the logical and physical data model of the data warehouse. Adding dimensions or changing the structure of dimensions or new facts requires a lot of maintenance, not only in the data model itself, but also in the load processes between source system and data warehouse on the one side, and between the data warehouse and data marts on the other side.
Solutions are sought by using standard (generic) industry sector models (e.g., from IBM or Teradata) or by using a data warehouse for which a standard business content forming the underlying ERP system is loaded into the data warehouse (e.g., SAP or Peoplesoft). However, in somewhat larger organizations, standard will never be standard.
Industry sector models contain too much on one side and must be expanded or adapted on the other side, if only because each self-respecting company feels itself to be so unique that at least the necessary customization of the operational and managerial processes has to be done. The alternative, to organize yourself in line with the standard solution, is something that virtually never takes place, but this – at least in case the costs of reorganization are lower compared to the costs of changing the standard system – would bring the most return from the investment in the standard model.
Importing standard business content works (but almost always only if the underlying ERP is implemented in a “vanilla” fashion) for that ERP system. Unfortunately, that ERP system is never the only source system, and so, changes are necessary for those other source systems. And all of these changes (both with the industry model and the standard content) need to be applied to a normal logical and physical data model. So, in the end, what you get is – depending on the degree of the initial coverage – more or less a head start.
Fortunately, there are other possibilities. There is Datavault from Core Integration partners as a data modeling method to model your data warehouse in a generic fashion. There is also Kalido, a product originally developed back in the ’90s within Shell, which is based on a fully generic model, in which changes in the information model are (in principle) overcome by only changing or adding new rows in the generic logical and physical model underneath the “hood.” Some other products (e.g., from two small Dutch startups, Dynalytical and BI-Ready) generate new database schemes driven by the changes in the business model.

Reusable instead of One Time

As much as possible, use standardized and reusable components of processes. This sounds completely logical and should go without saying, but in the chaos and pressure of timely delivery, it is quite easy to lose sight of this aspect. This results in a situation where another specific set of software is developed (and has to be maintained).
An underlying reason for this is that too often, too little attention is given to reusability while designing and developing the initial process and the first increments. This is often caused by the time squeeze in delivering the first increment.

Decoupling instead of Integration

At first sight, this seems not to be beneficial for the resistance to change. After all, the idea of an integrated environment is that by doing so, fewer changes are needed. However, it is better to decouple the different information logistic functions and to implement different layers. In that case, changes will be limited – just as with the world of physical logistics – to one or just a few functions. And next to decoupling, the more the design of individual functions is as generic or reusable as possible, the easier it is to isolate the impact of changes and minimize the throughput time of applying these changes.

Federation instead of Centralization

In designing an enterprise data warehouse, it is quite common to base the design on central implementation, in which all data is brought together and managed from one central point in the organization. This central point is also responsible for refreshing the data marts. In most situations, it would be preferable to apply a federative model. In a federated model, the enterprise data warehouse is split up into a number of well aligned and connected partial implementations. Each of the individual business units (region, country or operating company) has its own implementation under their own managerial responsibility. Each unit has the full freedom to model and manage its own part of the company’s business model. The common part is managed at a higher level and will be pushed down to the lower levels. Data from a lower level which is of interest to the higher level can be pushed up to that level. In this way, it is much easier to create support in the autonomous parts of the organization, and it is only necessary to reach agreement on the commonly used part of the model.

Heritage:

The initiative to start with the development of an enterprise data warehouse is always taken in a moment in which – depending on the size of the company – dozens or maybe even hundreds of small and large systems exist, which provide management information. Local systems, departmental systems, regional systems, corporate systems, systems on the PC of the individual user (60 percent of all management information is still Excel) and so on, and so on. And in this situation, it is true that “the spirit is willing, but the flesh is weak.” Yes, everybody agrees that the existing situation is not good for anyone, but when push comes to shove, few people are really prepared to give up their autonomy.
Central funding of the basic infrastructure and a federated concept as described is the best way to deal with the problem. The advantages of a shared and common infrastructure must be bigger compared to the (perceived) disadvantages. In short, it is not a matter of imposing a central data warehouse and a common approach, but a matter of convincing, selling, creating buy in and a lot of time and patience.

Data

And in the end, everything is arranged and settled as described. Money and governance are where they belong; the design of the data warehouse and the logistic process complies with the rules; the applied technology is “state of the art;” the feelings of the organization are with respect to politics and existing systems are dealt with in the proper way and yet… After the celebration of the initial successes, the progress comes (almost) to a halt. The better the integration from data from different systems is settled, the more integrated reporting and analysis facilities will become available. That is the moment when it becomes apparent that quality and consistency of data are not at the level they should be.
This lack of quality has two aspects. First, defects in the source systems. A lot of source systems contain a lot of faulty data. There are many reasons for this. In old systems, certain input fields are used for completely different data than originally intended (and still showing in the documentation). Often, users have to input data that is not relevant to their jobs. A lot of checks on inputs are not correct, complete or can be easily bypassed. All too often, these faults surface only at the time when more advanced analytical facilities are made available. The size and seriousness of problems with source data in its first instance is often neglected or underestimated.
As users are just normal people, the chance exists that users will blame the new data warehouse and will turn away from the system. Analyzing the defects, finding the root-cause, correcting and preventing future defects requires time, patience and also relatively big (extra) investments, not only in the data warehouse but also in the source systems.
A second data quality problem is the lack of consistency between the master data within – and certainly between – different source systems and organizational units. A survey conducted by The Data Warehouse Institute (TDWI) in October 2006 shows that 83 percent of the responding companies has (had) problems caused by defects in the master data and that 54 percent has gained benefits from having correct master data. Similar percentages (81 and 54) are mentioned for problems or successes with reporting and other BI facilities.
Master data systems – just like a “normal” system – can and will have defects in the data itself. But often, the problem is more serious and lies deeper. In large organizations, the ownership of notions such as customer, product, revenue, calendar and other core entities is just as splintered across the organization as business governance and ownership of supporting systems are.
Individual systems and reports for the direct users of such a silo application probably present the correct figures. But at the moment when a higher-up in the organization questions things such as, “What is the profitability of the sales of the 20 best sold products to the top 20 customers during the last quarter,” and the answer has to come from different systems, the confusion can be considerable. Again, the blame will fall on the data warehouse.
However, solving this problem is compared to solving normal defects in data much more complicated. Master data, in its first instance, is an organizational issue. Who is owner of which element of master data, who are involved in defining master data, which source system is leading and which other systems need to follow and need to be synchronized and how do we ensure that the data warehouse holds “the golden copy?” The enterprise data warehouse might be the place where the problem becomes visible, but solving it is a responsibility of the business organization facilitated by proper IT systems.
The success of an enterprise data warehouse can be measured with the disappearance of all kinds of “owned” systems and the quality and use of the information in the data warehouse. The road to implementation is long and with a lot of pitfalls, but success is not impossible. The ambition of this article is not to provide a recipe for success but to shed some light on a number of the ingredients.
The author wishes the readers a lot of patience, endurance and success in their quests to happy users of the BI facilities within their organization.