Is Hadoop Maturity in Sight?
There is no escaping the outpouring of hype about Hadoop. Some see it as heading toward maturity in terms of its enterprise readiness, while others think it is the enabler of a new class of so-called data lakes. But more than a few aren’t convinced.
Gartner analyst Nick Heudecker said enterprises remain tentative about heavy investment in Hadoop. This is due to concerns about overall business value as well as a shortage of skills in this specialized area of technology, he said.
Gartner surveys reveal more than half of enterprises have no plans to do anything at all with Hadoop within the next two years. Fewer than one in five plan Hadoop investments over the next two years.
So what’s going on with this darling of the Big Data analytics community?
There are plenty of well-promoted successes available about Hadoop's ability to distribute the processing of large data sets across clusters and scale out to thousands of servers. That said, Gartner believes that while Hadoop may be great for particular use cases, it could be overkill for the average business that does not need to derive instant insight from mountains of data, Twitter feeds, transactions and systems. That’s one of the reasons why the analyst firm called the outlook for Hadoop growth “anemic” for at least the next two years.
$100 Million Hadoop
Others are far more optimistic about Hadoop. Cloudera has been heralded as only the second open source company in history to beat the $100 million mark in annual revenues, a feat accomplished on the back of Hadoop. Mike Olson, the company's chief strategy officer and one of the original founders of Hadoop, is not surprisingly bullish on the subject.
Big Data is transforming industry, and Hadoop is faced with an enormous opportunity, he said. “The ability to ingest, store, process and analyze any kind of data is transformative. By being able to scale to hundreds of terabytes, you no longer can only keep a quarter’s worth of transactional data online, but a decade’s worth to analyze.”
In particular, he believes Hadoop's capabilities are opening the door to interactive and real time data analytics for such functions as better fraud monitoring, recognition of money laundering and improved security. Far from floundering, he sees Hadoop as making steady ground in the enterprise.
As organizations can now build apps that utilize Hadoop far more easily, Cloudera is seeing more usage of mobile data analytics, said Olson. He gave the example of Tableau as one of many companies providing tools that sit on top of Hadoop. Tableau connects to Hadoop distributions (Cloudera, Pivotal, HortonWorks, MapR and IBM InfoSphere BigInsights) to provide data analytics and visualization.
Hadoop has yet to attain the maturation needed to provide true end-to-end enterprise analytics, Olson conceded. But he cited the emergence of ERP running on relational databases as a historical example of technology evolution. After the advent of those early databases, it took at least a decade for a complete toolset to emerge to take advantage of these new capabilities, he pointed out.
Hadoop's Role in Data Infrastructure
Far from conquering all, Olson noted that Hadoop is not leading to the elimination of large swaths of traditional infrastructure. On the contrary, people are running data warehouses and relational databases to solve the problems they were designed to address.
What Cloudera is picking up is business related to ingesting and storing data from many different sources in order to do analytics and processing more cheaply at a much larger scale. These systems are gradually being integrated with legacy platforms into enterprise data hubs, he said.
High-end analytic workloads, for instance, are likely to remain on OLAP within a traditional data warehouse structure. And don’t expect many people to migrate transactional systems to Hadoop. Traditional databases will continue to deal with that traffic.
Drew Robb is a writer who has been writing about IT, engineering, and other topics. Originating from Scotland, he currently resides in Florida. Highly skilled in rapid prototyping innovative and reliable systems. He has been an editor and professional writer full-time for more than 20 years. He works as a freelancer at Enterprise Apps Today, CIO Insight and other IT publications. He is also an editor-in chief of an international engineering journal. He enjoys solving data problems and learning abstractions that will allow for better infrastructure.
