In order to overcome performance issues with traditional distributions of Hadoop, SunGard is developing Raptor in order to provide "near realtime analytics" required by financial services applications. The software giant is looking to open source the development, and also to use it in future products.
"Financial services does not have traditional big data problem to solve, per se - not like Google - but it has a need for analytics on large amounts of data, with different formats, including semi-structured and unstructured forms," says Neil Palmer, who heads the Advanced Technology practice at SunGard Consulting Services.
In its research of offerings to address this need, SunGard considered Hadoop, since it scales well across commodity hardware, and is a well supported open source offering. But a big drawback of Hadoop is its batch oriented approach to data processing, and its general performance.
To address these issues, the company has been working for several months on Raptor, which layers on top of Hive, an open source datawarehouse offering that runs in conjunction with Hadoop, and HBase, a non-relational database that also leverages the Hadoop Filesystem (HDFS). Palmer says that query performance of Raptor versus traditional Hadoop is a 5x to 10x improvement.
Raptor's architecture has three key components: the Digester, which segments large datasets into smaller ones for quicker searching; the DataNode, which handles block level indexing on HDFS; and the Optimiser Executor, which decides how best to conduct and direct queries. An in-memory caching component stores intermediate data within Raptor for increased performance.
Palmer cites financial services applications including fraud detection, risk analysis and compliance as likely to benefit from Raptor. The company has also explored predictive trading, but Palmer believes other solutions, such as BackType Technology's Storm and Apache Kafka, might be better suited to addressing the performance needs of automated trading.
In the near future for Raptor is support for running it in cloud environments, security policies, and a management console. Contributing Raptor to the Apache organisation as an open source project is also being pursued, and conversations are underway inside SunGard to leverage the technology for future products.
Comments