Validator Node Operations

Reward Minting and Traffic Funding

As explained in Tokenomics and Rewards, your validator node will need traffic to submit the transactions to execute withdrawals or accept multi-step deposits. As also explained in that section, the network provides rewards that can be used to fund traffic.

Note also that every validator node has an associated validator operator party that represents that validator node’s administrator (docs). The validator node automatically mints rewards for that party. It can further be configured to automatically purchase traffic using that party’s CC balance, which includes the minted rewards.

We thus recommend the following setup as a starting point to mint rewards and automatically fund traffic:

  1. Use the validator operator party as your featured exchangeParty. Follow Setup the featured exchange party to get it featured.

  2. Setup the treasury party to create a treasuryParty with a transfer preapproval managed by your exchangeParty.

  3. Setup automatic traffic purchases in the validator app.

  4. Optional: setup auto-sweep from the exchangParty to your treasuryParty to limit the funds managed directly by the validator node.

As a starting point for the automatic traffic purchase configuration, set targetThroughput to 2kB/s and minTopupInterval to 1 minute, which should be sufficient to execute about one withdrawal or deposit acceptance every 10 seconds. Please test this with your expected traffic pattern and adjust as needed. See this FAQ to measure the traffic spent on an individual transaction.

Setup Exchange Parties

Setup the treasury party

Setup the treasuryParty as follows with a transfer preapproval managed by your exchangeParty:

  1. Create the treasuryParty using the wallet SDK to Create an External Party (Wallet) with a key managed in a system of your choice

  2. Copy the party id of your exchangeParty from the Splice Wallet UI as explained above, or retrieve it by calling /v0/validator-user on the Validator API.

  3. Call /v2/commands/submit-and-wait on the Ledger API to create a #splice-wallet:Splice.Wallet.TransferPreapproval:TransferPreapprovalProposal (code) directly with the provider set to your exchangeParty.

    Note that setting up this transfer preapproval requires the exchangeParty to pay a small fee of about 0.25 $ worth of CC. The funds for this fee usually come from the validator liveness rewards that a validator node starts minting about 30 minutes after it is created. On DevNet or LocalNet, you don’t have to wait that long: just “Tap” the required funds from the built-in faucet.

Testing the party setup

You can test the party setup on LocalNet or DevNet as follows:

  1. Setup your exchangeParty and treasuryParty as explained above.

  2. Setup an additional testParty representing a customer.

  3. Transfer some CC from the testParty to the treasuryParty to simulate a deposit.

  4. Observe the successful deposit by listing holdings of the treasuryParty.

  5. Observe about 30’ later in the Splice Wallet UI of your validator operator user that the exchangeParty minted app rewards for this deposit. It takes 30’, as activity recording and rewards minting happen in different phases of a minting round.

Setup Ledger API Users

Clients need to authenticate as a Ledger API user to access the Ledger API of your Exchange Validator Node. You can manage Ledger API users and their rights using the /v2/users/... endpoints of the Ledger API.

You will need to authenticate as an existing user that has participant_admin rights to create additional users and grant rights. One option is to authenticate as the ledger-api-user that you configured when setting up authentication for your validator node. Another option is to log-in to your Splice Wallet UI for the validator operatory party and use the JWT token used by the UI.

We recommend that you setup one user per service that needs to access the Ledger API. This way you can easily manage permissions and access rights for each service independently. The rights required by the integration components are as follows:

Required Ledger API User Rights

Component

Required Rights

Purpose

Tx History Ingestion

canReadAs(treasuryParty)

Read transactions and contracts for the treasuryParty.

Withdrawal Automation

canActAs(treasuryParty)

Prepare and execute transactions on behalf of the treasuryParty.

Multi-Step Deposit Automation

canActAs(treasuryParty)

Prepare and execute transactions on behalf of the treasuryParty.

Automated exchange parties setup for Integration Testing

participant_admin and canActAs(treasuryParty)

Create parties and use the treasuryParty to create its TransferPreapprovalProposal. Hint: grant canActAs(treasuryParty) to the user doing the setup after allocating the treasuryParty.

.dar File Management

.dar files define the Daml workflows used by the token admins for their tokens. They must be uploaded to your Exchange Validator Node to be able to process withdrawals and deposits for those tokens.

The .dar files for Canton Coin are managed by the Validator Node itself. The .dar files for other tokens need to be uploaded by you using the /v2/packages endpoint of the Ledger API. See this how-to guide for more information.

Important

Only upload .dar files from token admins that you trust. The uploaded .dar files define the choices available on active contracts. Uploading a malicious .dar file could result in granting an attacker an unintended delegation on your contracts, which could lead to loss of funds.

Monitoring

See the Splice documentation for guidance on how to monitor your validator node. Note in particular that it includes Grafana dashboards for monitoring the traffic usage, balances of local parties (e.g., the exchangeParty), and many other metrics.

Rolling out Major Splice Upgrades

For major protocol changes, the global sychronizer undergoes a Major Upgrade Procedure. The schedule for these upgrades is published by the Super Validators and also announced in the #validator-operations slack channel.

As part of this procedure, the old synchronizer is paused, all validator operators create an export of the state of their validator, and deploy a new validator connected to the new synchronizer and import their state again. For a more detailed overview, refer to the Splice docs.

The procedure requires some experience to get it right, so it is highly recommended to run nodes on DevNet and TestNet so you can practice the procedure before you encounter it on MainNet.

From an integration perspective, there are a few things to keep in mind:

  1. A major upgrade only preserves the active contracts but not the update history. In particular, you will not be able to get transactions from before the major upgrade on the update service on the Ledger API of the newly deployed validator node.

  2. Offsets on the upgraded validator node start from 0 again.

  3. The update history will include special import transactions for the contracts imported from the old synchronizer.

We recommend to handle the upgrade as follows:

  1. Wait for the synchronizer to be paused and your node to have written the migration dump as described in the Splice docs.

  2. Open the migration dump and extract the acs_timestamp from it, e.g., using jq .acs_timestamp < /domain-upgrade-dump/domain_migration_dump.json. This is the timestamp at which the synchronizer was paused.

  3. Wait for your Tx History Ingestion to have caught up to record time acs_timestamp or higher. Note that you must consume offset checkpoints to guarantee that your Tx History Ingestion advances past acs_timestamp.

  4. Upgrade your validator and connect it to the new synchronizer following the Splice docs.

  5. Resume your Tx History Ingestion from offset 0.

  6. Ignore transactions with record time 0001-01-01T00:00:00.000000Z. These are the special import transactions for contracts imported from the old synchronizer so you have already processed them on the old synchronizer. Note that you can reuse the infrastructure used to ignore duplicate transactions after a backup restore to ignore the transactions here.

  7. After the initial import transactions, continue ingestion as usual.