Running a Relayer
Technical instructions that someone comfortable with command line can easily follow to run their own Across V2 relayer
All of the code in this repository can be found in this GitHub repository.
Requirements
The Across v2 Relay Bot is implemented in Node.js and is capable of running on a variety of platforms. See the following table for platform recommendations.
Resource | Recommended |
---|---|
CPU | 64-bit Dual Core @ 2+ GHz |
RAM | 4GB |
OS | UNIX-like (GNU/Linux, MacOS) |
Installation
Updating
A helper script is available to automate updates. This performs the following actions:
Flushes any existing installed dependencies.
Pulls down the latest relayer-v2 commit.
Installs all dependencies and builds the relayer.
Displays the latest commit in the relayer-v2 repository.
Important
This update helper is offered as a convenience. After update, the operator must manually verify that the update succeeded and that the commit shown matches the intended target.
Configuration
Environment Variables
This section describes the environment variable configuration that the bot requires in order to operate. This is the minimum configuration for running a relayer that operates on all supported tokens for all destination chains. You just need to change the configs at the top.
Operators can exclude tokens/destination chains by not having a balance for that token on that destination chain. For customizing tokens/destination chains while having balances, see the advanced section for all supported configs.
Advanced configurations
Notes on requirements to RPC Providers
The relayer is dependent on querying historical blocks on each chain. The RPC provider must therefore support making archive queries. If the RPC provider cannot service archive queries then the relayer will fail with reports of obscure RPC errors.
Using a Redis in-memory database to improve performance
The relayer queries a lot of events from each chain's RPC that Across supports. Therefore, we use an in-memory database to improve performance and cache repeated RPC requests. Installation instructions can be found here
. Once installed, run redis-server
in one terminal window and then open another one to continue running the relayer from.
The redis server is used to cache the responses of RPC-intensive repetitive requests, like eth_getLogs
, or internally-computed data like getBlockForTimestamp
. Caching of data is subject to the age of the input response from the RPC provider, such that a minimum block age is required before it will be retained. This provides some protection against caching invalid data, or valid data becoming invalid (or otherwise changing) due to chain forks/re-orgs.
Managing cross chain inventory
The relayer bot is designed to use the same account across each of the supported chains: Ethereum, Optimism, Polygon, Boba and Arbitrum. Therefore, the bot can be told to automatically rebalance its inventory across chains and target some allocation.
The best way to demonstrate how rebalancing can be customized is to walk through an example setting of the RELAYER_INVENTORY_CONFIG
environment variable:
Rebalances from L1 to L2
First let's look at the "tokenConfig". This informs the relayer how it should distribute its token balances across the different networks. Each of the keys of the "tokenConfig" object ("0xC02", "0x6B1", "0xA0b", "0x226") represent the Mainnet address of the ERC20 token that we want to automatically control inventory for. So, in order, the tokens in the example config are WETH, DAI, USDC, and WBTC.
Diving into WETH's token configuration, notice that it has an object containing "targetPct", "thresholdPct", "unwrapWethThreshold", and "unwrapWethTarget" mapped to each network ID. The "targetPct" and "thresholdPct" instructs the relayer account on Mainnet when to send funds to the network with the associated network ID.
The relayer is always aware of its aggregate funds across all chains, so for example if the account has 10 WETH on Mainnet, 5 on Optimism, 4 on Arbitrum, 3 on Polygon, and 2 on BOBA, then it has a total of 24 WETH. The allocations are: 5/24 on Optimism, 4/24 on Arbitrum, etc.
When the relayer's allocation for a specific chain drops below the "thresholdPct", then the relayer will send funds from its Mainnet account to the chain with the shortfall so that its post-transfer allocation is increased to the "targetPct". In the example config above, if the allocation percent on Optimism were to drop below 5%, then the relayer would send funds from Mainnet to Optimism to bring its allocation to 8%.
All funds are sent through the canonical L1-->L2 bridges and the logic implementing such rebalances can be found here.
Note that the "targetPct" and "thresholdPct" are unused for the mainnet token config (i.e. for the row with ID "1") but are included to avoid a compile-time error.
Overriding repayment chain
By default, all fills sent by the relayer are set to be repaid on Mainnet, so if the relayer sent a fill on Polygon for USDC, then it will receive the relayed USDC amount plus a relayer fee on Mainnet. This means that over time, the relayer's allocation percent on Polygon will decrease.
Therefore, the inventory rebalance logic also provides a function that overrides the repayment chain ID to try to maintain its target allocation. This happens automatically before submitting any fill by this function.
Unwrapping WETH
The "unwrapWethThreshold" and "unwrapWethTarget" tell the relayer when to unwrap WETH into ETH to keep enough ETH on hand for paying gas costs. These configs are unused on Polygon which does not pay gas in ETH. Moreover, these configs should only be set in the token configuration for WETH (i.e. "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2").
If the relayer's ETH balance drops below the "unwrapWethThreshold", then it will unwrap enough WETH to increase its ETH balance to the "unwrapWethTarget" value.
This logic can be found here.
Wrap ETH
The "wrapEtherThreshold" is used only for Optimism and Boba, the two OVM networks which requires that ETH, not WETH, is sent over the bridge from L2 to L1. Because of this fact, the relayer can end up holding a lot of ETH on L2 and not enough WETH.
If the relayer account's ETH balance on Optimism or Boba drops below this value, then it will wrap any excess ETH above the value into WETH. Its important that this value is set higher than the "unwrapWethTarget" for WETH for Optimism and Boba. In the example above, the "unwrapWethTarget/Threshold" is 1.5/0.75 and the "wrapEtherThreshold" is 2, meaning that the relayer on Optimism and Boba will unwrap WETH if its ETH balance is less than 0.75 and increase its balance to 1.5, and will wrap ETH if its balance is above 2.
Logic for wrapping ETH can be found here.
Security Considerations
This description describes a basic set of security considerations that relay bot operators should be aware of. See Recommendations for suggestions on how to improve the security of relay bot instances.
Disclaimer
This is not a complete security guide. Relay bot operators solely assume the risk of loss of fu
Host Attack Surface Area
The Across v2 relay bot communicates with various public RPC endpoints, and thus requires outbound network access. Unknown/untrusted network actors may be able to communicate with the relay bot host. It's important to reduce the attack surface area that the host environment exposes, in terms of:
Network services listening on public interfaces.
Interfaces opened for remote administration, and their permitted authentication mechanisms.
Third-party installations that may autonomously communicate (i.e. phone home) over the network.
Handling Secret Keying Material
The Across relay bot requires an in-memory copy of an Ethereum private key in order to sign relay transactions. This in-memory copy is typically loaded from the execution environment, via the following environment variables:
MNEMONIC (when run with --wallet mnemonic)
PRIVATE_KEY (when run with --wallet privateKey)
When not specified directly via environment variables, these configuration items can be saved in the filesystem in a .env file, in the relayer-v2 working directory. Relay bot operators should be aware of at least the following:
Depending on the .env mode flags, users or programs with filesystem access may be able to read secrets from the .env file.
When storing secrets on disk, anyone with raw disk access may be able to override filesystem permissions and recover file contents. This includes:
People with system administrative privileges.
People with physical disk/hardware access.
Vendors of cloud-based execution environments (i.e. VM hosts).
During operation, the relayer-v2 bot retains secret keying information in-memory. Anyone with the ability to dump application or system memory may be able to retrieve secret keying material.
Outsourcing Secret Keying Material Storage
The Across relay bot supports a bespoke Google Cloud key management interface (gckms), whereby the bot retrieves keying material from a secured, trusted key management. Keying material retrieved over the gckms interface is stored locally in-memory, but is not saved to disk.
The gckms interface is tailored for use by Risk Labs and is not currently intended for use by third-party bot operators. Support for generic third-party key management systems may be added to the relay bot in future.
Recommendations
Deploy relay bot instances in isolated environments (i.e. dedicated VM/container environment, or dedicated hardware).
Adhere to basic system administration and hardening practices. NIST 800-123 Guide to General Server Security may be useful.
Never install software from untrusted sources, and verify software packages before installation/execution.
Limit the ability for malicious network actors to gain system access by restricting inbound network traffic to trusted hosts and/or services. Drop all other traffic.
Avoid the use of password-based authentication schemes for remote login services. Use key-based authentication (i.e. SSH keys) instead.
Ensure that filesystem ownership and permission flags are set appropriately at all times. These attributes should be periodically reviewed for correctness.
Ensure that parties with raw disk access are trusted.
Note: Where untrusted or unknown parties may have raw disk access, filesystem encryption schemes may be useful in reducing the opportunity for theft of secret keying material.
Ensure that parties with the ability to dump system and/or application memory are trusted.
Note: This applies primarily to vendors providing virtualised execution environments - i.e. cloud/VM hosts.
Maintain at least one secure offline copy (backup) of the secret keying material. Backups should be periodically reviewed for correctness.
Running the Relayer for the first time
Once you've installed and built the relayer code and set your desired environment variables, you're all set to run the relayer code. The entry point to run the code is the command (choose one of the following):
This will run the relayer in "simulation mode" meaning that it will simulate the transactions that would fill deposits, but will not submit them. This will give you a chance to review transactions before funds are sent.
On the first run, the bot should approve various SpokePool contracts to withdraw ERC20's from it. This is required to fulfill relays. These approval transactions are not "simulated" currently and will still be sent even when running in simulation mode. Approvals will only be sent for tokens with nonzero balances in the relayer account.
If the bot successfully completes a run, you will see this log before it exits:
When you feel ready to run the relayer and send your first relay, set SEND_RELAYS=true
to exit simulation mode!
Which account will be used to send transactions?
The first account associated with the MNEMONIC
set in the environment will be used. Be sure to add ETH and any token balances to its account so that it can send relays.
Which tokens can be relayed?
WETH, USDC, DAI, WBTC, UMA. This list is likely to require updates in the future. Work is being done to automate the latest token list.
How can I learn more about the code behind the bot's logic?
The relayer's entry point file is Relayer/index.ts.
The relayer bot first identifies all unfilled deposits across each of the chains. Deposit events are fetched here by the SpokePoolClient for each chain. Using these unfilled deposits as input, the relayer will attempt to fill them based if it has the token balance on the destination chain to do so.
Auxiliary Topics:
Across V2 smart contracts
The smart contracts can be found at this repository. They have been audited by OpenZeppelin. A high level video overview of the contract architecture can be found here.
Across V2 UMIP
UMIP-157 explains how "valid" relays are identified, which are relays that correctly filled a deposit and paid the correct fees and are due to be returned their relayed amount plus a relay fee.
The relayer bot code is supposed to be one implementation of UMIP-157.
Additional relayer FAQs:
How do Relayers get refunded?
Relayers are paid back after a root bundle containing the relayer’s fulfillment is published to mainnet and passes a challenge period. Relayers can choose which chains they are paid back on. When running the example relayer code, they are by default repaid on the chain that they fulfill deposits on. The relayer can override this setting.
What is a root bundle?
A root bundle for a block range is a set of three Merkle roots that contains all of the information necessary to refund relayers who fulfilled a deposit during the block range. A root bundle is valid only if it contains all of the expected information for a block range. This UMIP explains at length exactly how to construct a valid root bundle.
How often do new root bundles get published and executed?
The current liveness period is 7200 seconds or 2 hours. You can always query it on the HubPool by calling the read-only method liveness()
, and only one root bundle can be proposed at a time. So, the fastest cadence for root bundles being proposed is every 2 hours. Each root bundle contains approximately all of the relayer refunds up to the current time (as of the proposal) and as old as right after the preceding root bundle proposal time. A 'dataworker' is what we refer to as the agent who gathers all Fill and Deposit events from all of the chains that Across V2 supports bridging to and from in order to construct these Merkle root bundles. A dataworker could choose to propose a new root bundle right after the current one passes liveness, or it can choose to wait. Realistically, we expect that a dataworker will only submit a new root bundle after it contains a certain volume of refunds, for capital efficiency reasons. So to summarize:
Relayer refunds are contained in root bundles that are optimistically published to the HubPool
Once the root bundle proposal passes liveness, refunds can be sent to relayers. At this point, relayers are made whole and receive an additional relayer fee
The fastest cadence of root bundle proposals is once every two hours. In the worst case, root bundles can take much longer if volume is low across the system
A dataworker can propose a bundle at any time. However, if a current bundle is in liveness - it is required that either that bundle be disputed or the bundle passes liveness and is executed before a new bundle is proposed.
Last updated