How HTAP Database Technology Can Help You
Page Contents
What does hybrid transaction/analytical processing (HTAP), a database technology that uses in-memory capabilities to perform OLAP (online analytical processing) and OLTP (online transaction processing) at the same time, have to do with the port authority of Hamburg, Germany?
One of the busiest ports in the world, it is expecting massive growth but expanding its physical footprint isn't an option amid a city of 1.7 million people. As many as 8,000 trucks pass through the port each day, and nobody wants those trucks just idling while they wait in a queue.
To handle growth, the port is undertaking a massive Internet of Things project that aims to coordinate traffic — marine and road — so that trucks arrive at the exact moment a crane can load a container onto the flatbed. The project involves 300 traffic sensors that track data such as how many parking spaces are available and whether open bridges have traffic stopped, then apply analytics to achieve these gains.
‘Go-to Architecture' for Operational Analytics
IoT is one of the latest use cases for hybrid transaction/analytical processing, a moniker that Gartner coined for databases that can perform transactions and analytics simultaneously.
“It's pretty much the go-to architecture for operational applications; anything to do with the customer journey, anything omni-channel,” said Timo Elliott, innovation evangelist for SAP.
“If you go to a website and want to recognize that this is the same person who's standing in your store checking the website to see if the price is different, you can bring up that person's purchase history and provide that to the sales assistant along with what product to recommend. The HTAP architecture is designed to do exactly that. Along with transactions, that information is immediately available,” he said.
[Read more about operational analytics: Your Next Big Data Project? Operational Analytics]
HTAP Growth
“The real story is that HTAP is going mainstream,” said Gartner analyst Massimo Pezzini.
While HTAP technology isn't new, adoption is accelerating, which Elliott attributes to falling costs and business imperatives. Adoption has long been far too expensive for anybody except financial services or tech companies.
“It's only in the past few years that the cost of these in-memory chips compared to hard drives has been a tipping point. There's still a cost differential, but it's completely outweighed by the business benefits — being able to access that information up to million times faster,” he said. “People want to do real-time everything, expect to do real-time everything.”
Gartner doesn't have firm numbers on HTAP's share of the database market, which is still fairly small, but Pezzini said he found a recent survey telling.
When it asked companies the primary use case for buying a new database, 20 percent cited HTAP capabilities. That was surprising; he'd expected it to be 4 percent. But 40 percent put it among their top three factors.
HTAP Capabilities
HTAP typically is used to address process optimization, portfolio optimization, real-time risk management and security, according to Barry Morris, co-founder of NuoDB.
What inventory do I need to update? Which ad is the best one to serve next? Where are the “free” Uber cars at this moment? Which is the best computer for me to run my next application? These are the kinds of questions HTAP can address.
Traditional analytics systems often provide business insights too late, Pezzini pointed out. Financial institutions want to address potential fraud as it's happening, not days or weeks later.
Among the new capabilities that HTAP offers businesses:
- The ability to adjust prices on the fly or update an online product catalog in real time based on the potential customer's search
- As a sales person, creating simulations to determine which version of a product to sell to a particular customer
- As a sales manager, monitoring your team in real time as the end of the quarter approaches
- Plan shipments of goods based on a reaction to an event, such as a strike or snowstorm
How HTAP Handles Different Workloads
Combining the two types of workloads, OLAP and OLTP, is “by no means a trivial engineering effort,” 451 Research analyst Matthew Aslett and Carnegie Mellon assistant professor Andrew Pavlo conclude in a research paper.
The two types of workloads have different processing needs. OLTP databases need rapid response times, handle many concurrent sessions and target specific records for reads/updates. OLAP workloads are the exact opposite: they use long- running queries, few concurrent sessions and large scans of whole tables.
Modern databases incorporate a number of innovations — new concurrency schemes, storage flexibility, and elastic capacity — that make HTAP possible, explained NuoDB's Morris.
For example, many historical databases use lock-based concurrency to prevent updates to any data being used in an analytical workload — thereby blocking transactional workloads against that data. By using version-based concurrency models, modern databases can enable both processes to run without sacrificing consistency.
Similarly, the advent of databases that can adjust capacity on demand and have flexible storage systems enables organizations to add machines optimized for that workload, then redeploy — or in the case of cloud, delete — the specialized machines once the job is over.
HTAP Benefits
Gartner initially outlined four key HTAP benefits:
- Data doesn't have to move from operational databases to data warehouses
- Transactional data is readily available for analytics when created
- Drill-down from analytic aggregates always points to fresh HTAP application data
- You eliminate or at least reduce the need for multiple copies of the same data
Elliott also points to faster querying, strong data compression and flexibility to make changes among HTAP's benefits.
HTAP Challenges
But there are HTAP challenges, too, said Pezzini, including:
- A skills problem. Finding developers and systems engineers with a deep understanding of this technology can be tough.
- Adapting legacy applications. “If you have an existing application, engineering the application to HTAP is not simple. You have to do some significant work. It you want to take advantage of HTAP capabilities to the fullest, you may need to rewrite the application,” he said.
- A business mismatch: It's difficult for IT people to explain all of the technology's capabilities. “The technology can do more than most business people can figure out.”
HTAP Vendors and Future Adoption
All the major database providers — SAP, Oracle, Microsoft, IBM, and Teradata — now offer this capability along with upstarts including VoltDB, NuoDB, Clustrix, MemSQL, Splice Machine, CockroachDB and others.
Pointing to the SAP ERP suite S/4HANA as an example, Pezzini foresees wider adoption as the major vendors embed HTAP into their core enterprise software products.
“The smaller players brought technology to market sooner than Oracle, Microsoft and IBM, but these big guys have an impact in the market,” he said.
Legacy vendors have been producing systems designed 15 or 20 years ago; they need to differentiate and innovate, he said. They're doing so in ways such as moving applications to the cloud and adding support for mobile.
“I don't believe many organizations are going to build their own HTAP applications, but many of them will use HTAP in packaged applications and SaaS solutions,” Pezzini said.
Susan Hall has been a journalist for more than 20 years at news outlets including the Seattle Post-Intelligencer, Dallas Times Herald and MSNBC.com. She writes for The New Stack and FierceHealthIT, among other publications.
