‘Big Data’ Offers Big Challenges – And Big Rewards
“Big data” is exploding as more and more information is collected and stored daily, and both businesses and governments need to prepare if they're not going to be completely overwhelmed, according to a new report.
Just in the U.S., there is a shortage of as many as 140,000 to 190,000 people with the deep analytical skills needed, as well as a need for an additional 1.5 million managers and analysts required to analyze big data and make decisions based on their findings, the McKinsey Global Institute (MGI) found in the report entitled, “Big data: The next frontier for innovation, competition, and productivity.”
Big data is a term for immense amounts of data that are so large as to be difficult to nearly impossible to store, analyze and manage using current technologies — ranging from millions of real-time data feeds to streaming video to medical imaging.
“The sheer volume of data is a global phenomenon — but what does it mean? Many citizens around the world regard this collection of information with deep suspicion, seeing the data flood as nothing more than an intrusion of their privacy. But there is strong evidence that big data can play a significant economic role to the benefit not only of private commerce but also of national economies and their citizens,” the report said.
How large a payoff is possible?
“For instance, if U.S. health care could use big data creatively and effectively to drive efficiency and quality, we estimate that the potential value from data in the sector could be more than $300 billion in value every year, two-thirds of which would be in the form of reducing national health care expenditures by about 8 percent,” the report said.
On the technology side, vendors have already begun responding to the need to store and access enormous amounts of data. This week, for example, IBM rolled out upgrades to one of its key tape library systems that will enable it to store and retrieve nearly three exabytes — thousands of petabytes — of data.
“In the private sector, we estimate … that a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent [while] in the developed economies of Europe, we estimate that government administration could save more than $149 billion in operational efficiency improvements alone by using big data,” the report said.
How much data?
“MGI estimates that enterprises globally stored more than 7 exabytes of new data on disk drives in 2010, while consumers stored more than 6 exabytes of new data on devices such as PCs and notebooks [as well as mobile devices],” it said.
All of which points to a ballooning shortage of personnel trained to be able to manage and analyze all of that burgeoning data.
“The United States — and other economies facing similar shortages — cannot fill this gap simply by changing graduate requirements and waiting for people to graduate with more skills or by importing talent … It will be necessary to retrain a significant amount of the talent in place; fortunately, this level of training does not require years of dedicated study,” the report added.
Stuart J. Johnston is a contributing editor at InternetNews.com, the news service of Internet.com, the network for technology professionals. Follow him on Twitter @stuartj1000.
For more on Big Data analytics, see Analytics and In-Memory Databases Are Changing Data Centers
