What Agile Business Intelligence Really Means
One of the written works I most admire, Bruce Catton's history of the Civil War (“Never Call Retreat”), says of the prelude to Missionary Ridge that Grant and his commanders “had inspected the ground beforehand, and, as sometimes ironically befalls the diligent, had unanimously fallen into error” — by concluding that a flank attack could succeed.
Several months back, Jill Dyche, a long-time business intelligence consultant with a clear indication in her blog of diligence, savvy and experience in data warehousing, launched an attack on present-day enterprises attempting to do agile business intelligence (BI) “on the cheap,” claiming that they were not performing the drudge work of developing data models and improving data quality and therefore were less likely to achieve agile business intelligence. In the process of critiquing these users, Dyche also seemed to imply that the methods she laid out were sufficient to achieve significant improvements in business agility.
I believe that Ms. Dyche is suffering the misfortune of the diligent: assuming that her experience and efforts in BI and familiarity with agile development should lead to improved business agility from BI. It is no knock on her — quite the contrary — to say that I disagree.
Far more important, however, is my sense that we have not yet done the hard work of asking just how so-called “agile” BI can lead to increased business agility, and to the benefits that should attend better business agility. As a result, we are in danger of devaluing agility itself by showering the term “agility” on BI approaches that will, in the end, inevitably disappoint us.
Why BI and analytics in themselves are not agile
We understand the potential of better BI: a deeper understanding of business processes, and of customers. We understand the potential of better analytics: better anticipation of the consequences of business strategies and marketing campaigns. We understand the potential of faster BI and faster analytics: more rapid reaction to emergencies, and competitive advantage over slower competitors. What we do not understand is that faster is not the same thing as more agile.
The key reason that faster is not always better is captured in the saying: going faster gets you to the wrong place quicker. More fundamentally, improving the speed of BI deployment and upgrade ignores what marketing theory calls marketing myopia — the fact that the business is, or should be, in a different market. Collecting more and better information about existing customers, and improving the importance of reacting to them in the next release, as well as speeding the improvements those customers ask for, yields apparently excellent results in the short term — and embeds the wrong market ever more firmly into the business processes, so that switching out of that market becomes ever harder.
A glaring example of this in BI — one that it is amazing that more businesses do not note — is the entire housing bubble and derivatives-driven crash that has nearly led to a second Great Depression. The financial models that allowed leading-edge firms to identify seeming flaws in the market, to hedge, and to trade, literally in milliseconds, provided superb profits all along the line right up to the point where the bubble burst. The models themselves, and the processes of the investment banks that used this kind of BI, became ever more skewed towards managing mortgage-backed securities — with the result that despite the continuing danger of default, some continue to try again, using slight variations on the original scheme. The speed of deployment and upgrade of this BI software was by any standard leading-edge, as was the depth and quality of the actual information collected. But the inability to collect information that would indicate that something was wrong, and the inability to quickly adjust the business to a different market, should have indicated the truth: that the bank was becoming less agile, and more risky (in the sense of downward risk), and sooner or later it would pay.
What does real Agile BI look like?
One of the key insights from recent studies of agile software development is that increasing the focus on the agility of the process, and decreasing the focus on its costs, downward risks, and ROI actually — and permanently — can decrease costs, increase revenues, increase “upward risks” (which is good!), and decrease downward risks. So real agile BI should not only focus on improving the ability of the organization to change direction proactively as well as in response to changes in its environment, but also create a BI process that itself focuses on improving the ability to change direction within the process, in response not only to evolving user needs but also to changes in the environment.
These findings are seconded by a recent Sloan Management Review (Winter 2011, p. 21) survey of companies, most of whom were using analytics. The “top performers” in typical business metrics such as profit among these were different from the “bottom performers” primarily because they were using analytics to guide their actions in strategy development and product research and development, and because they were using insights from analytics not only periodically, to guide future strategies, but also on a daily basis, to steer day-to-day operations.
Let's be more concrete. Here's my model of an organization's process of ingesting and analyzing information, and its typical goals:
- Data entry — accuracy
- Data consolidation — consistency
- Data aggregation — scope
- Information targeting — fit
- Information delivery — timeliness
- Information analysis — analyzability
Now, clearly, the aim of this process should be to get the right data to the right person as fast as possible, with the right context and tools for deep analysis and maximum effectiveness of resulting actions. My survey two years ago suggested that each stage of the process was contributing to a result that fell far below this standard:
- 20% of data has errors in it (accuracy)
- 50% of data is inconsistent (consistency)
- It typically takes 7 days to get data to the end user (timeliness)
- It isn't possible to do a cross-database query on 70% of company data (scope)
- 65% of the time, executives don't receive the data they need (fit)
- 60% of the time, users can't do immediate online analysis of data they receive (analyzability)
- 75% of new key information sources that surface on the Web are not passed on to users within the year (agility)
This last result goes directly to the problem with the lack of agility of today's business intelligence. Almost all of the data that would allow the business to understand how the business strategy and products should change going forward is not delivered in a timely fashion, if at all.
So how would we change the BI/analytics version of this process, and the attendant BI/analytics tools, to improve not only the agility of the process, but also the agility of the business as a whole? On the next page, I examine each stage of the process, and the process as a whole, to suggest some ways that they could become more agile by focusing on agility — based on what we have found already are effective ways of improving agility.
Wayne Kernochan of Infostructure Associates has been an IT industry analyst focused on infrastructure software for more than 20 years.
Wayne Kernochan has been an IT industry analyst and auther for over 15 years. He has been focusing on the most important information-related technologies as well as ways to measure their effectiveness over that period. He also has extensive research on the SMB, Big Data, BI, databases, development tools and data virtualization solutions. Wayne is a regular speaker at webinars and is a writer for many publications.
