Troubleshooting FAQ

Operational Runbook

Troubleshooting FAQ

Unable to authenticate with GitHub

Verify:

GitHub App ID
Installation ID
Private key Secret
The application has been installed into the correct repository

DARs are not synchronizing

Verify:

Repository owner and name
Branch and directory (if configured)
Repository permissions
GitHub App installation

DARs are being repeatedly unvetted

Verify:

Splice version on primary validator aligns with splice version on covalidating node

External parties are not being onboarded

Verify:

Correct Kubernetes secret for connectivity to Ledger API
Ledger API endpoint
Admin API endpoint
Participant IDs in configuration file are correct
3 participants have been referenced in the onboarding request
Onboarding request had expected threshold (2 of 3)

Unable to connect to the participant

Check:

Ledger API endpoint
Admin API endpoint
OIDC issuer configuration
Client credentials
Network connectivity

Security Considerations

Store all credentials in Kubernetes Secrets.
Do not commit GitHub private keys to version control.
Grant only the minimum required GitHub permissions.
Rotate GitHub App keys periodically.
Limit Kubernetes Secret access using RBAC.

Operational Runbook

Marking a party as read-only (Observation mode)

If a specific party is flagged for suspicious or malicious activities, you can follow the steps below to downgrade that party to read-only Observation permissions on your validator node.

Downgrading the party to Observation mode is the preferred and recommended approach to restrict write operations, since it is non-destructive and reversible.

Downgrade party to Observation only

This changes the party-to-participant mapping from Confirmation to Observation, meaning that this validator no longer participates in confirmations for that party and exposes only read-only access to the party’s data through the participant API. To enact this change, only your validator signs the transaction, and no signature is needed from the party. You can perform this activity using the example Canton console script below:

import com.digitalasset.canton.topology.ForceFlags
import com.digitalasset.canton.topology.ForceFlag.DisablePartyWithActiveContracts
import com.digitalasset.canton.topology.ForceFlag.AllowInsufficientParticipantPermissionForSignatoryParty
import com.digitalasset.canton.topology.ForceFlag.AllowConfirmingThresholdCanBeMet

val partyLiteral = "<your-party-literal-here>"
val party = PartyId.tryFromProtoPrimitive(partyLiteral)
val myValidator = participant.id
val synchronizerId = participant.synchronizers.id_of("global")

@ participant.topology.party_to_participant_mappings.propose_delta(
    party = party,
    adds = Seq(participant.id -> ParticipantPermission.Observation),
    store = synchronizerId,
    forceFlags = ForceFlags(
      DisablePartyWithActiveContracts,
      AllowInsufficientParticipantPermissionForSignatoryParty),
  )

Covalidation Service

Troubleshooting FAQ and Operational Runbook