Subsquid - Public data indexing Infrastructure for Polkadot and Kusama (Q3 2022)

Summary

Proponent: Subsquid Lab Official - FAyqRVLZmXVUU9pNeG8w8SxS5xRgN8FRGeWLL6u9S32TMHc

Short description: Ongoing costs for the running of high performance, scalable, and reliable data indexing infrastructure for most of the Polkadot and Kusama ecosystem (and growing).

Raw data: Archive Infrastructure Metrics - July, August, September

Requested KSM: 1,407.29 KSM

Previous proposals:

https://kusama.polkassembly.io/treasury/171

https://polkadot.polkassembly.io/treasury/134

Motivation

Archives are an important piece of Polkadot and Kusama data infrastructure provided by Subsquid. They provide access to on-chain data in GraphQL format for all major parachains on Kusama and Polkadot, with more chains getting support as the ecosystem grows. They are being used as a data source for Squids and for efficient data exploration and ad-hoc queries. An up-to-date list of Archives is kept in a public Archive Registry

All archives can be accessed and queried via an web intuitive GraphiQL interface and are often used as developer friendly replacement of a block explorer.

For example, here’s a query for the last ten transfers on Kusama blockchain run against the Kusama Archive endpoint

Over third Quarter of 2022 we've provided a free public data indexing service for ~35 Substrate blockchains (some archives have been dismissed, new ones have been added), so that everyone interested in the projects can publicly access indexed data on Substrate Events, Extrinsics, Storage Items and EVM logs. Here are Kusama's projects (parachains and mainnets aiming for future parachains slots) we currently support:

• polkadot
• rococo
• crust
• astar
• moonbeam
• bifrost
• acala
• equilibrium
• statemint
• interlay
• subsocial
• efinity
• phala
• hydradx
• litentry
• peaq
• kusama
• moonbase
• khala
• crust
• moonriver
• karura
• shiden
• statemine
• shibuya
• kintsugi
• calamari
• litmus
• hydradx
• gear_testnet
• karura_twin
• bajun
• invarch_tinkernet
• gmordie
• mangata_x

Achievements

At the end of Q2 Subsquid released a new generation of Archives, together with a major version of the SDK, both code named "Fire Squid".

The improvements brought by this release include:

• Substantial performance increase, with the ability to deploy a new Archive for a chain as big as Kusama, and the indexing to be ready in roughly a day
• Fire Squid Archives support batch requests, which lead to a hundredfold increase in performance of indexing project connecting to them (squids)
• Fire Squid Archives expose two separate gateways, configured and tuned for different uses (one for data exploration by humans, the other one specifically for programmatic and batch access)
• Fire Squid Archives "unwrap" proxy, sudo and batch calls, tracing the origin, but making them much easier to access them

Subsquid has also engaged in continuously improving the product, iteratively working on adding features, such as EVM Transactions indexing, and subscriptions to queries in the Graphql endpoints of indexing projects created with the Subsquid SDK (commonly called squids).

Statistics and data

Previously, we used to keep data almost indefinitely, now we only keep the last month, so because of changes in internal policies on data retention and in our infrastructure, only partial information is available for the period. The month of September can be used as a sample, though.

Between 2022-09-03 (the oldest available data) and 2022-09-30 Subsquid’s Archive infrastructure has collectively:

• Served a total of 129,468,038.2 (~129 million) queries in the month of September alone
• Our 7 day average peaks above 5 million queries and adjusts around 3.3 million queries per day at the end of the period

Data from the month of October confirms the linear trend, allowing us to make assumptions for the months of July and August, given the steady growth of queries over time shown.

Despite being only an estimation and result of linear regression, a chart encompassing the totality of Q3 can be produced, estimating the total queries served over the three months period, which roughly amounts to 430 millions:

Costing

The proponent and beneficiary of this proposal is Subsquid Lab Official (wallet address, contact email).

Over the course of Q3, Subsquid had access to Cloud credits, so there has been a sizable reduction of sustained costs, and this proposal only considers the subtotal, after Cloud credits and discounts have been applied.

This proposal includes the combined costs for running Archives for Polkadot, Kusama and their Parachains, which amounts to a total of 1,407.29 KSM - raw data is here.

The submission will alternate between treasuries, this has been advised by the council, to avoid generating overhead, involving two separate votings, for smaller amounts.

The table above is using the average of rate exchanges from the week of January 10, 2023, when the proposal is being submitted.