Introducing Streams: Blockchain Data, Without The Headaches
Streams is reshaping blockchain data ETL, linking 17+ chains and key datasets to destinations like webhooks and S3 storage, accessible in Public Beta.
In an era where data rivals currency in value, navigating the complexities of web3 data has emerged as a major innovation roadblock. Developers and enterprises alike have grappled with the daunting task of extracting blockchain insights, often hitting barriers of complexity, operational challenges, and the opaque economics of on-chain data mining.
Leveraging our extensive experience in overcoming infrastructure challenges and responding to feedback from a wide range of innovators, we have decided to bridge the gap between data tools and web3 with a portfolio of ETL products.
Our first solution, Streams, is designed to make blockchain data seamlessly available for conventional tools used for data, embodying our vision to streamline, simplify, and secure web3 data integration.
As we introduce Streams, we're not just presenting a product but laying a foundation for our new ETL pipeline vision for web3 data. Streams embody QuickNode's dedication to innovation, efficiency, and making blockchain data universally accessible and actionable.
Announcing Streams
Streams are a foundational layer of our ETL vision for blockchain data. It connects data sources consisting of 17+ chains and networks, 5+ essential datasets with multiple destinations, including webhooks, S3-compatible storage, PostgreSQL and Snowflake available during Public Beta.
Complexity is not just a barrier to entry; it's a continual source of inefficiency. Users often struggle with the methods, parameters, and ever-changing schemas, leading to frustration and wasted effort.
To dismantle these barriers, we introduce an intuitive, user-friendly interface that simplifies integrating and managing blockchain data. In less than 6 clicks you will get streamed essential web3 datasets in seconds.
Equipping users with preset datasets eliminates the guesswork and technical overhead associated with data retrieval, making the first step into blockchain data straightforward.
Exactly-Once, Guaranteed Delivery
Streams revolutionize web3 data infrastructure by acting as an exactly once guaranteed delivery service. This means you can seamlessly integrate Streams with your data lake in a few clicks, ensuring reliable data extraction and loading without the hassle and significant engineering time to manage retries, corrupt data, or missing blocks.
Streams eliminates common obstacles and developer pain points with endpoint infrastructure, such as:
- Rescaling your ingestion layer depending on chain dynamics — Streams will provision and scale compute, memory, and network resources without ongoing administration.
- Corrupt or missing data impacting your analysis — As a guaranteed delivery service, we mean it: every block, receipt, or trace will be delivered exactly-once in the order of the dataset finality.
Forget the cumbersome process of provisioning workers to pull data from blockchain endpoints and having to monitor their progress.
“Streams solves multiple data ingestion problems for us and we are very happy with it. It has the perfect fit for our data infrastructure and we’re currently working on integrating it into Nuant. It will require some effort, but we’re willing to do it because of the reorg handling, direct integrations with S3, Kafka and SQL. Webhooks allow us to ingest real-time data very easily. Can’t wait to see prestate traces, parquet files, and Snappy compression”.
Mark Boychuk, Head of Development at BlockAlliance
Real-Time Data Consistency
Navigating the complexities of real-time blockchain data—such as varying finality times across different datasets (like blocks, receipts, traces) or navigating through blockchain reorganizations—often results in data inconsistency. Streams guarantee the delivery of consistent, real-time data, empowering you to create live dashboards without reliability concerns.
Antonio Seveso, CTO at fluidkey
Efficient Historical Data Handling
For historical data analysis or backfills, Streams allows configuring large data batches, setting precise data ranges, and fine-tuning your destination. This capability enables you to select the most optimal setup and throughput for your chosen destination, significantly simplifying your data pipelines.
Comprehensive and Transparent User Experience
Logs offer transparency into the operational aspects of ingestion and retries, while metrics provide insights into performance and reliability.
Financial management is as crucial as data management. Our platform offers transparent billing and usage tracking, empowering users with the information they need to manage their costs effectively. This transparency ensures that users can allocate their resources wisely, avoiding unexpected expenses and optimizing their investment in web3 data infrastructure.
“Prototyping is very quick with Streams, this is amazing — with just 8 clicks we’re getting the data right away. It’s great to be able to choose how we want the data, just not worry about the scalability and the vast amount of data coming in. So simple and gives us resources to work on the actual product. It’s perfect that we have a simple way to backfill the data we need without having to worry about retries. We have our own nodes and we notice missing blocks too late.”
Founder, Stealth Startup
Pricing and Feature Availability
Throughout its Beta stage, Streams will follow a transparent and scalable pricing model designed to meet the diverse needs of our clients, from startups to large-scale data engineering teams.
At the core, our pricing is based on the volume of data you stream, measured in gigabytes, ensuring costs directly correlate with your usage.
Starting from the Build Plan, which is perfect for prototyping and basic data access, we scale up to the Enterprise Plan, which is designed for high-volume, real-time direct data integration.
Pricing Examples
To illustrate our pricing in action, consider the following scenarios of use for Streams.
As such these estimates offer a starting point for understanding your potential costs:
Streams currently supports: 👇
Chains
- Arbitrum
- Arbitrum Nova
- Avalanche C-Chain
- Base
- Bera
- Blast
- BNB Smart Chain
- Celo
- Ethereum
- Fantom
- Gnosis
- Immutable X
- Optimism
- Polygon
- Polygon zkEVM
- Scroll
- ZKSync
Datasets
- Blocks
- Transactions
- Receipts
- Logs
- Blocks with Receipts
Destinations
- Webhooks
- S3-compatible storage
- PostgreSQL
- Snowflake
See the full list of features and chains on our docs website.
A Complete Platform for Blockchain Data and ETL Expertise
Functions, also launching soon, can be combined with Streams to radically solve any blockchain data need; use common programming languages—including Javascript, Python, Go, or Ruby—to efficiently transform payloads from Streams, enrich them with contract data by calling QuickNode endpoints, and much more.
Building upon an already robust blockchain data product portfolio, including Core RPC and QuickAlerts, the addition of Streams and Functions ensures QuickNode is the trusted solution provider for all your ETL needs.
How To Access Streams
Streams will change the way you use blockchain data.
Ready to see for yourself?
Streams are now available on your QuickNode dashboard for accounts with the Build plan and higher. We value your input and invite you to contribute to our evolving roadmap. Get started with Streams today and discover the impact on your project.
For guidance and more information, our team is just a click away.
