Note

This page is a work in progress. It may contain incomplete or incorrect information.

Synchronizer Troubleshooting

Sequencer subscriptions fail for newly onboarded sequencers

Newly onboarded Sequencers only serve events more recent than the “onboarding snapshot” taken during the onboarding. In addition, some events may belong to transactions initiated before a Sequencer was onboarded, but the Sequencer is not in a position to sign such events and replaces them with “tombstones”. If a participant (or mediator) connects to a newly onboarded Sequencer too soon and the subscription encounters a tombstone, the Sequencer subscription aborts with a FAILED_PRECONDITION error specifying InvalidCounter or SEQUENCER_TOMBSTONE_ENCOUNTERED. If this occurs, the participant or mediator should connect to another Sequencer with a longer history of sequenced events before switching to the newly onboarded Sequencer.

Generating authentication token for a member

Most services exposed by the sequencer require an authentication token to be provided with the request. Participant and mediator nodes manage this token internally and transparently, but it may be useful to obtain such token to perform troubleshooting tasks. Authentication tokens can be generated on the sequencer admin API for such purpose. As a prerequisite, the enable-testing-commands config flag must be enabled:

canton.features.enable-testing-commands = yes

Important

This feature allows to generate authentication tokens for any member of the synchronizer and use them to authenticate on the API on behalf of this member on the sequencer. Use with care and do NOT share this token.

To generate the token, we’ll need the ID of the node for which the token will be valid. For example:

@ val memberId = participant1.id.toProtoPrimitive
    memberId : String = "PAR::participant1::12201ff69b1d24edbf0ee2028a304ea702ee8536790dab1a31e7136e6d90ff6d473c"

@ val token = sequencer1.authentication.generate_authentication_token(participant1, expiresIn = Some(1.minute))
    token : MemberAuthenticationToken = MemberAuthenticationToken(
      member = PAR::participant1::12201ff69b1d...,
      token = <ByteString@644e3681 size=20 contents="Cg\271-\033\236\000\027r\374\027\251A\035\326C\272\253f\344">,
      expiresAt = 2026-02-18T22:40:36.666670Z
    )

@ val tokenAsBase64 = token.tokenAsBase64
    tokenAsBase64 : String = "Q2e5LRueABdy/BepQR3WQ7qrZuQ="

The expiration argument is optional, and will default to the maximum validity duration configured on the sequencer.

With the token in hand one can make a call to the gRPC sequencer API directly. Before that we’ll also need the synchronizer ID to which the sequencer belongs to:

@ val synchronizerId = sequencer1.physical_synchronizer_id.toProtoPrimitive
    synchronizerId : String = "global::122032922613929d67857e621fb13e3da49ec13883e24908404520319eee6d31fb4d::34-0"

Write the data down to temporary files to use from a shell CLI:

@ val tmpDir = better.files.File("/tmp/troubleshoot/auth_token").createDirectories()
    tmpDir : better.files.File = /tmp/troubleshoot/auth_token

@ (tmpDir / "memberId").createFileIfNotExists().write(memberId)
    res6: qual$2 = /tmp/troubleshoot/auth_token/memberId

@ (tmpDir / "synchronizerId").createFileIfNotExists().write(synchronizerId)
    res7: qual$2 = /tmp/troubleshoot/auth_token/synchronizerId

@ (tmpDir / "token").createFileIfNotExists().write(tokenAsBase64)
    res8: qual$2 = /tmp/troubleshoot/auth_token/token

@ (tmpDir / "sequencer_address").createFileIfNotExists().write(sequencer1.config.publicApi.address)
    res9: qual$2 = /tmp/troubleshoot/auth_token/sequencer_address

@ (tmpDir / "sequencer_port").createFileIfNotExists().write(sequencer1.config.publicApi.port.unwrap.toString)
    res10: qual$2 = /tmp/troubleshoot/auth_token/sequencer_port

Then for example, calling the DownloadTopologyStateForInit endpoint with grpcurl (replace the token, member ID and synchronizer ID with the appropriate values):

TMPDIR=$(echo "/tmp/troubleshoot/auth_token")

grpcurl -plaintext -H "authToken-bin: $(cat $TMPDIR/token)" -H "memberId: $(cat $TMPDIR/memberId)" -H "synchronizerId: $(cat $TMPDIR/synchronizerId)" -d '{ "member": "'$(cat $TMPDIR/memberId)'"}' $(cat $TMPDIR/sequencer_address):$(cat $TMPDIR/sequencer_port) com.digitalasset.canton.sequencer.api.v30.SequencerService/DownloadTopologyStateForInit

A few things to note:

• The token needs to be set as a authToken-bin header in Base64 encoding

• The ID of the member needs to be set as a memberId header

• The ID of the synchronizer needs to be set as a synchronizerId header

Finally, to invalidate the token, call the com.digitalasset.canton.sequencer.api.v30.SequencerAuthenticationService/Logout RPC or use the following command on the sequencer.

@ sequencer1.authentication.logout(token.token)

Note that this will disconnect the member from the sequencer and may cause temporary failures while the sequencer connection / authentication is re-established.