Company: ThinkData Works
Industry: Technology
ThinkData Works makes data access easy. The Toronto-based data technology company’s flagship product, Namara, an end-to-end data management platform, provides the tools for every stage of the data lifecycle.
As ThinkData started working on an enterprise version of Namara, it began building the application using a microservice architecture to combat scaling and performance concerns, and needed a new high-availability setup. The solution needed to be able to answer complex queries around data access extremely fast in real time.
After deciding that a graph database was the best solution, the team adopted Redis Enterprise and RedisGraph to manage service-resource access.
ThinkData has seen immense performance improvements since adopting RedisGraph, including a 62% decrease in total request time for functions such as text-query catalog searching. Plus, using a managed service like Redis Enterprise with RedisGraph enabled in the database freed the ThinkData team to focus on R&D.
When ThinkData Works was founded in 2014, its goal was to make open data more accessible. The founding team sought to create a solution that would make discovery of new and existing open datasets easier, and also help data scientists work with those datasets.
The solution became Namara, an all-in-one data management solution that makes working with any data easy, helping enterprises access, manage, and integrate their data. The end-user-facing application emphasizes easy signup and access, browsability of ThinkData’s data catalog, and powerful categorization and filtering capabilities.
Namara, originally a monolithic Ruby on Rails application powered by NoSQL document stores and text search engines, has historically been worked on by a small team of developers who make linear improvements and add new features. In the past year, ThinkData has launched a new enterprise offering of Namara. The transition to an enterprise platform brought performance and scaling concerns, so the team began rebuilding the application using a microservices architecture.
ThinkData has been using open source Redis as a key-value store since the company was founded, but with the primary purpose of the platform now focused on user authorization and secure resource access—driving browsing, categorization, and filtering behaviors—ThinkData began looking for a new, high-availability setup.
After reviewing case studies analogous to the data model, licensing, and growth concerns of Namara, the ThinkData team realized that RedisGraph would be an ideal approach for its own expansion. The team needed a solution that would work with its licensing strategy, where clients may be running dedicated deployments on a variety of cloud providers.
ThinkData was most concerned about storage consistency and recovery options when reviewing its first proof-of-concept implementations. Graph databases are ideal for answering complex queries, as they are designed to query connected data very quickly. After reviewing multiple graph databases, the team ultimately decided to use Postgres as its primary backend storage and RedisGraph for service-resource management. “RedisGraph offered a great mix of maturity, feature completeness, interoperability with the languages that we use, price, and performance,” says Brendan Stennett, CTO of ThinkData Works. The team was especially keen on RedisGraph’s integrations with other languages like Go and Cipher.
By leveraging the performance characteristics of the graph engine in RedisGraph for ThinkData’s highly connected data, the team has seen immense performance improvements on its platform. Since each service in the ecosystem is responsible for its own primary storage, shared graph access allows service-to-service read operations to take place without introducing additional network overhead.
When measuring open catalog query performance between the legacy and enterprise platforms—retrieving all datasets a user is able to view—RedisGraph can retrieve information in less than half the amount of time as the legacy platform takes. The total request time for the enterprise platform is roughly 380ms, a 43% decrease from the legacy platform’s request time of 678ms.
Similarly for text-query catalogue searching, RedisGraph is able to retrieve the results in 93ms, part of a total enterprise response time of 320ms. For comparison, the legacy database takes 851ms to resolve the same request. RedisGraph delivers a 62% performance boost!
“With our architecture plan, many graph queries go through the graph database for any given request, so we are particularly happy with the throughput that we received from RedisGraph compared to a couple of competitors,” Stennett says.
The team also appreciates the benefits of using a managed service like Redis Enterprise with RedisGraph enabled in the database. Because few on-call issues arise, ThinkData can field a considerably smaller-than-usual DevOps team—freeing up additional resources to focus on R&D, Stennett says.
He also emphasizes the ease and trust his team developed working with the Redis support team. “The Redis support team’s ability to respond quickly to some issues that came up in early development really allowed us to trust working with Redis, in the sense that we’ll get that support feedback when needed,” he says.
An added bonus of picking RedisGraph was getting access to RedisInsight, which helped get ThinkData’s team of developers excited to work on RedisGraph. “Oftentimes developers have never worked on a graph database, or don’t even know what a graph database is,” Stennett says. RedisInsight’s ability to “help them visualize these early concepts while developing really, really goes a long way in getting them productive.”
The ThinkData team plans on moving its existing data model to the graph database, while continuing to improve its recovery tooling to minimize downtime and performance impact. The team is also looking to take advantage of new RedisGraph features and improvements as they become available. ThinkData uses the RediSearch functionality that RedisGraph provides, but it is also looking into using other data structures offered in Redis Enterprise.
