The Importance of Web3 Provider Redundancy

Web3 providers, also known as Node providers, are essentially blockchain data keepers. These projects run networks of hundreds of blockchain nodes, and are tasked with providing applications with the latest and historical blockchain data.

The Web3 provider is an essential part of a blockchain-powered application. It is what wallets and interfaces/gateways like TrustWallet, MetaMask, and MyCrypto use to "talk" (read/write) to blockchain networks.

Every time you see or ask for a token balance in MetaMask, MetaMask has to first query the blockchain for this data. It does this through the Web3 provider (usually through a library like web3js, ethersjs, ethereum.rb, web3py, etc...).

The Web3 provider then queries the blockchain node(s), and returns the value/data to the app for it to display to the end-user.

Imagine thousands of users asking for their tokens' balance at the same time – that's a lot of requests! This is why companies like QuickNode exist, to be able to quickly and efficiently provide this data, so apps can function even at scale.

This function extends beyond just token balances on Bitcoin, Ethereum, or Solana – smart-contract states on Ethereum are updated almost every 13 seconds (average ETH MainNet block time). Apps relying on smart-contract events must receive updates in real-time. For this apps can subscribe to or poll for events using a Web3 provider.

Now, nodes are interesting pieces of peer-to-peer (p2p) software. Nodes are responsible for keeping a synced, up-to-date copy of the ledger, participating in consensus, validating blocks, serving light clients and maintaining peers. There is a variety of node flavors (eg. eth1, eth2), and each has its benefits & challenges.

Running a single node is no easy task, let alone a globally-balanced cluster at-scale.

- It takes several months to develop robust infrastructure
- Failures can be catastrophic, requiring weeks to recover
- Nodes on average have issues once every 5 days:
    - CPU spikes, memory leaks, disk issues
    - Inconsistent peering, corrupted databases
    - Transaction broadcasting issues
    - Frequent bugs + regressions
    - 1 in 6 "stable releases" have breaking changes

Without a Web3 provider, there is no data for the application to work with; no data from the blockchain to manipulate or display.

Web3 providers abstract the node infrastructure layer, so developers, teams and businesses can focus on their core app. Platforms like QuickNode also have tools & features which accelerate development and help users build the best version of their app.

Now that we understand how essential a Web3 provider is, let's get into the importance of redundancy and failover.

When you build your app with a single Web3 provider, you have a central point of failure. We want to decentralize data and applications, but most apps are running with 1 provider, and mostly a single provider. On November 11, we learned what happens when that single provider has an outage.

Do have to give it to Infura. Without their services and efforts, Ethereum ecosystem, app development, developer activity would never have reached the levels it is at now. It is estimated that they run 5-10% of all Ethereum nodes, and their network does more traffic in 1 day than the rest will do in a month.

That said, a node versioning oversight resulted in an outage lasting approximately 7 hours.

The outage has also caused crypto exchanges, including Binance and Bithumb, to disable ETH and ERC-20 tokens withdrawals. The popular Ethereum wallet MetaMask is also having difficulties since Infura is its default service provider.

This took a good chunk of Web3 applications down with it. What can we learn from this?

There are other Web3 provider options out there! One of them is QuickNode.com and it is a drop-in replacement for Infura. Users also get an endpoint which they can plug into their code, cURL, or compose with a tool like Etherflow – to get the data they or their app needs.

Why is Web3 provider redundancy important? To make dApps better, build-in Web3 provider redundancy from the beginning. This way if your primary provider has an outage, whether it's internal (ie. something broke), or an issue with hosting (eg. cloud provider outage), or global network-related (eg. fiber cut, network outage), it can be mitigated by having a backup/failover provider.

When the main provider stops responding, the backup provider takes over. Your app has minimal to zero downtime. You might also consider using >1 provider for cost savings. This sounds like common practice, but it's not so common!

Now that we understand the importance of Web3 provider redundancy, how do we put it into practice?

Well, our friend Igor, who runs unisignals.ai and filter.sitg.app, and now a QuickNode customer, experienced first-hand the effects an outage has on his app. He quickly implemented multiple providers so his app is a lot more robust and resilient to Web3 provider issues.

Here's how he did it:

At unisignals.ai, we search for valuable insight from Ethereum transactions. We scan and parse pending and confirmed transactions, then enhance with on-chain and off-chain data.

I'm sharing with you several things we learned during development.‌‌

• You can do a lot with free RPC nodes. See the list here https://ethereumnodes.com/
• If you use Infura, it makes sense to host your app at us-east-1 AWS region. Their API endpoints are in the same AWS data center, so you'll have super good latency.

• Don't rely on one provider. Have redundancy:
  • distribute the load between providers, especially if you use free providers
  • if some endpoint returns many errors or slow, disable it and check it later
  • time-limit RPC requests. If you handle incoming transactions within one CPU-process (node.js as an example), hanging Promises might crash your app – so you want to resolve them quickly.

See example implementation for retrievable, time-limited RPC operation distributed between different providers here:

https://github.com/kigorw/eth-utils/blob/main/eth.ts#L67

and usage: https://github.com/kigorw/eth-utils/blob/main/eth.ts#L149‌‌

• Have a cache of constant values that you often need (symbol or supply of ERC20 token, for example); so you will save on RPC querying.
• It also makes sense to subscribe to several providers at the same time if you do real-time processing:
• Transaction receipts https://github.com/kigorw/eth-utils/blob/main/eth.ts#L180
• Pending transactions https://github.com/kigorw/eth-utils/blob/main/eth.ts#L205
• Keep in memory received transactions and skip duplicates.‌‌

See usage example of all said above: https://github.com/kigorw/eth-utils/blob/main/index.ts‌‌

• For real-time processing, it also makes sense to keep track of processed blocks. In case of a server restart, you may catch up on transactions from missed blocks.
• If you send time-sensitive transactions (for example, swap on a DEX at a certain block), have a separate list of RPC APIs that will not be rate-limited and send the same transaction (same nonce) to several providers simultaneously. You may also try to defend yourself against front-running by using private transactions of https://taichi.network/

Excellent tips from Igor!

This is just one way to implement Web3 provider redundancy into your blockchain application.

What is great is that you can implement a robust, reliable backup provider using QuickNode for free (with premium plans starting at just $49). That is a fantastic insurance policy!

And if your app already has traction and requests volume, consider a Build or Scale plan from QuickNode!

Why QuickNode

QuickNode makes running a Blockchain node easy so developers can focus on all the wonderful dApps they’re building, instead of worrying about DevOps/NodeOps, uptime, scaling, security, etc...

QuickNode saves valuable time getting to-market while taking away the headache of node maintenance and synchronization. It’s a valuable tool in every dApp developer’s arsenal that was created by developers for developers. Visit QuickNode’s Guides section for more information.

Need help with your project or have questions? Contact us via this form, on Twitter @QuickNode, or ping us on Discord!

About QuickNode

QuickNode is building infrastructure to support the future of Web3. Since 2017, we’ve worked with hundreds of developers and companies, helping scale dApps and providing high-performance access to 16+ blockchains. Subscribe to our newsletter for more content like this and stay in the loop with what’s happening in Web3! 😃