Why Most Business Intelligence Projects Fail: Page 2
Where we part company is in just how that communication occurs. For 15 years at least, everyone has been talking about improving communications between IT and corporate; but what they really mean is communications between the top of the IT pyramid and the top of the corporate pyramid. Agile aims to do something quite different: a personal bear-hug between the actual user of the software and the developer so that both contribute to a constantly evolving improvement that ensures a product that is just as relevant to the user at “final ship” as it is when the user first sets out his or her needs. In effect, agile eliminates the failures classified as “not just what I wanted.”
The really counterintuitive part of agile, though, is that it also reduces or eliminates the other two types of failed project: late, and never finished. This part of agile is counterintuitive because anyone looking at a “spiral” agile process immediately sees inefficiencies and lack of quality control. The agile process seems to say, “go this way, no, the user changed his or her mind, go this way,” or “ready, fire, aim.” At the same time, there seems to be an uncomfortable choice between bug fixing efforts wasted on code never to be used, and scanting on testing until the end of the process, when everything is finally nailed down.
Everyone has his or her own version of why agile still lowers lateness and inability to pass final test, and many of them are versions of the remark of the impresario in “Shakespeare in Love”: it's a miracle. However, to my mind, the reason that agile actually produces better quality than quality control, and faster finishes than waterfall design-then-code-then-test programming, is that it focuses not on developing as fast as possible, but on iterating from prototype to prototype as fast as possible. In this way, the final product is built “middle in” from tested code steadily over the course of the project, user benefits from part of the product come well before the traditional “final ship,” and retrofitting features towards the end of development is kept to a minimum.
The implications for business intelligence projects are straightforward. The way to go is not to improve communication between CIO and CEO, useful though that can be in other situations. The path to pursue in a BI project is to establish ad-hoc, intense ties between developers or data miners and specific users in lines of business, in middle management, or in corporate, and to use those communications links to drive rapid prototyping that goes from “wouldn't it be nice if” to “boy, have I got some great further ideas for this” in nothing flat. It passes belief that, after such a truly agile process, someone would turn around and say, “no, actually I didn't want this,” or “gee, I could have gotten the same thing much faster the old way, even though I didn't even realize what I wanted back at the beginning.”
Bottom Line: Why Business Intelligence Projects Really Fail
By flipping this analysis over, we can see more clearly why present projects may fail. In effect, the company's effort to generate a new way of analyzing the data suffers from the same problem as a one-way mirror: it does not examine the business' own needs and the way they are evolving, but focuses its scrutiny on flaws in the BI feature development process. Perhaps it is the fault of those darn developers; perhaps the CIO just isn't hearing what the business wants; perhaps there needs to be more careful planning by the designers up front. Don't blame you, don't blame me, blame that fellow under the tree.
Agile development is actually a lead-in to increased business agility — a highly effective lead-in, to be sure, since increased new-product development agility has the biggest positive effect on the company's bottom line. That means that reducing BI project failure by using agile processes within IT not only speeds up introduction of effective new analyses that allow the business to respond to a rapidly changing environment, but also introduces a culture of agility and agile processes to the rest of the business — the business users that it serves. As the business user finds IT business intelligence responsive to his or her needs, that executive begins to focus more on insights about the ways that the environment is changing, and to perceive that these are valuable aids to one's career and to the company. Change you, change me, don't just change that fellow under the tree. A two-way mirror means less project failure, not just in BI but all over the company.
