Direct Query Comes to Atoti: Ending the Dilemma of Shadow IT vs. Slow IT

On June 18 2021, during our “Future of Analytics” client event, we unveiled the next major addition to the Atoti platform: “Direct Query”.

What is Direct Query?

Direct Query allows Atoti to query and analyze data from external databases, particularly Cloud databases such as Snowflake and Big Query, in addition to its own in-memory database. In practice it means that Atoti customers will be able to apply, seamlessly, the best-in-class data model of Atoti to third-party databases.

What is particularly compelling, in this case, is that the process is both very easy to implement for developers and completely transparent from the user’s point of view. Developers can use Atoti’s Python library, for instance, to very simply create models, which are then applied to the data in Snowflake – or another database -, while end users never have to worry about it.

‘Shadow IT’, the enduring issue that Direct Query for Atoti solves

Today, banks and other financial institutions have to use a mix of three, four or more off-the-shelf BI platforms and databases to perform their risk analysis, each with their own sweet spot of performance depending on the number of queries, the size of the dataset, the precision needed, etc. The first problem with this is that it requires maintaining and deploying  all those technologies and then training staff on how to use them.

The bigger issue however is that in virtually every case, it is still not enough. Remaining gaps have to be plugged through endless exchanges of spreadsheets and through the costly development of custom applications. 

It is this burdensome, expensive combination of “Shadow” IT and “Slow” IT that Atoti, with Direct Query, resolves with a single platform that delivers the best performance with the most optimized hardware footprint across all areas, leaving no gaps.

How is it different from other BI solutions?

Atoti with Direct Query differs from other BI platforms in two major ways: 

• Fintech business logic, with non-linear aggregations such as Value at Risk, netting rules for credit risk, dynamic bucketing, fund decomposition, predicting PnL using a sensitivity ladder… simply cannot be modeled in Tableau or PowerBI. This is a major reason why Shadow IT crops up at banks. Atoti, on the other hand, is designed for finance and supports such measures out of the box.
• Atoti is built from the ground up as a highly-optimized, very fast in-memory database and calculation engine. It was made to deliver the best analytics precision on several terabytes of datasets. The challenge, which we solve with Direct Query, is therefore to optimize hardware costs by making it work natively with on-disk databases. Other BI platforms, which are built primarily for on-disk operations, simply cannot scale up to deliver precise analytics past a certain data volume.

How Direct Query dramatically reduces the operating cost of risk analytics 

With Direct Query, Atoti can apply its data model both in-memory and in-database, seamlessly and it takes minutes to program rather than days.

Let’s take the example of a common use case: intraday market risk analysis with 100 days of historical data, adding up to about 4 terabytes. 

Intraday data is most frequently queried and where analysts need the most precision, therefore we can load it fully in-memory, on a dedicated Atoti server.

For historical data on the other hand, analysts usually only need basic figures to work. In this case we load an aggregated cache in memory, but leave the rest on disk, in Snowflake.

In that way, you can achieve optimal performance and cost-savings through the combination of a Snowflake cluster and an in-memory Atoti cluster. You achieve all the benefits of a complete Atoti deployment, but you can tailor the hardware to precisely match your needs for all parts of your dataset. 

From the user’s point of view, the whole experience is seamless. In-memory queries are resolved in milliseconds while on-disk queries may take a second or two to complete, but they are just as precise and flexible.

An example of a smart deployment of Atoti with Direct Query: the data most frequently queried, like the data of the current day and an aggregated cache of the previous 100 days, is loaded in memory while unaggregated historical data remains on the database. It’s very easy to program and completely transparent to use..

For the Market Risk use case described above, with around 4 TB of data, the hardware footprint for Atoti represents less than half a percent of the Cloud cluster. This is the most competitive, most comprehensive analytics offer on the market for finance.