- Overview
- Tutorials
- Getting started
- Get started with Canton and the JSON Ledger API
- Get Started with Canton, the JSON Ledger API, and TypeScript
- Get started with Canton Network App Dev Quickstart
- Get started with smart contract development
- Basic contracts
- Test templates using Daml scripts
- Build the Daml Archive (.dar) file
- Data types
- Transform contracts using choices
- Add constraints to a contract
- Parties and authority
- Compose choices
- Handle exceptions
- Work with dependencies
- Functional programming 101
- The Daml standard library
- Test Daml contracts
- Next steps
- Application development
- Getting started
- Development how-tos
- Component how-tos
- Explanations
- References
- Application development
- Smart contract development
- Daml language cheat sheet
- Daml language reference
- Daml standard library
- DA.Action.State.Class
- DA.Action.State
- DA.Action
- DA.Assert
- DA.Bifunctor
- DA.Crypto.Text
- DA.Date
- DA.Either
- DA.Exception
- DA.Fail
- DA.Foldable
- DA.Functor
- DA.Internal.Interface.AnyView.Types
- DA.Internal.Interface.AnyView
- DA.List.BuiltinOrder
- DA.List.Total
- DA.List
- DA.Logic
- DA.Map
- DA.Math
- DA.Monoid
- DA.NonEmpty.Types
- DA.NonEmpty
- DA.Numeric
- DA.Optional
- DA.Record
- DA.Semigroup
- DA.Set
- DA.Stack
- DA.Text
- DA.TextMap
- DA.Time
- DA.Traversable
- DA.Tuple
- DA.Validation
- GHC.Show.Text
- GHC.Tuple.Check
- Prelude
- Smart contract upgrading reference
- Glossary of concepts
Canton Network Quickstart observability & troubleshooting overview¶
Note
This page is a work in progress. It may contain incomplete or incorrect information.
Note
The screenshots in this guide are taken from multiple sessions and are inconsistent with each other. This will be rectified once some of the updates are committed.
Contents
It’s assumed that you have read the quickstart getting started guide and explore the demo. If you haven’t, we strongly encourage those documents to establish a baseline understanding of observability.
Overview of observability¶
The Canton Network quickstart deployment configuration includes a full observability suite. Tools preconfigured for monitoring and troubleshooting distributed Canton applications—both in development and production. The observability suite provides three key types of monitoring data:
consolidated structured logs for application and system events
distributed traces that visualize end-to-end transaction flows; and
metrics for monitoring key performance indicators.
The suite allows data types to be correlated with each other to provide insights for root cause analysis. In addition, the Canton Ledger also provides a variety of correlation and tracing ids that permit tracking transaction provenance across multiple organizations and environments.
The LocalNet configuration¶
The Quickstart runtime configuration is defined in .env.local
,
which allows the option to bring up a local deployment of the Observability Stack.
This file can be created using $ make setup
,
which wraps the command $ ./gradlew configureProfiles --no-daemon --console=plain --quiet
,
or can be edited manually to set environment variables LOCALNET_ENABLED
and OBSERVABILITY_ENABLED
to true
or false
as desired.
The LocalNet
runtime configuration is handled by docker-compose
configured in compose.yaml
using environment variables from .env
in the
quickstart/
project root directory.
The usual Docker commands and tooling applies.
Immediately useful commands you probably already know:
$ docker ps
lists the running containers.$ docker logs [-f] <container>
fetches the logs of a container, and follow the logs with the-f
option.If the system is not working well to the extent you do not trust the observability stack (discussed later),
docker logs backend-service
is a good place to start looking for errors that might provide an insight into what has gone wrong.
$ docker restart <container>
for those instances where a container seems to have become stuck.
Observability overview¶
The Quickstart application provides a foundational production Daml application. It includes a full observability configuration which is helpful to troubleshoot or debug an application. As a working demo, Quickstart is opinionated regarding its technology stack. However, the platform itself is agnostic. Individual components can be replaced as required.
The current troubleshooting and debugging services include:
Local ledger inspection via Daml shell (https://docs.daml.com/tools/daml-shell/index.html)
Datasource collection and management via OpenTelemetry
This uses the OTEL Collector (https://opentelemetry.io/docs/collector)
Metrics are aggregated using Prometheus (https://prometheus.io/)
Logs are consolidated using Loki (https://grafana.com/oss/loki/)
Traces are aggregated using Tempo (https://grafana.com/oss/tempo/)
Aggregated observations (metrics, logs, and traces) are viewable via Grafana (https://grafana.com/oss/grafana/) which acts to allow hyperlinked exploration of the Observability fields.
Daml Shell¶
Daml Shell is a terminal application that provides interactive local ledger inspection on top of PQS.
Quickstart is configured to launch Daml Shell in a Docker container and is configured to connect to the included
application provider’s PQS instance.
This is easiest to access via the top-level project scripts accessed via make
from quickstart/
.
To see this in action, build and start the quickstart app then:
Run $ make create-app-install-request
to use curl
to submit the
create AppInstallRequest ...
command to the ledger [1] to initiate user
onboarding [2]. Then you can use the following Daml Shell commands:
> active
to see a summary of the contracts you created; and,
> active quickstart-licensing:Licensing.AppInstall:AppInstallRequest
to
see the contract details for any Asset contracts on the ledger; finally,
> contract [contract-id from the previous command]
[3] to see the full
detail of the AppInstallRequest
contract on the ledger.
> help [command]
provides context help for daml shell commands. [4]
Grafana¶
Grafana is accessible via its web interface, which is port-mapped to http://localhost:3030/,
and can be opened in the current browser from the command line using make open-observe
.
Your debugging should focus on using Grafana’s trace and log facilities, as well as ledger inspection via Daml Shell. If you make sure that your exported logs and traces are sufficient to support debugging during development, they are more likely to support diagnostics in production, as well.
There is additional access configured into the Quickstart that can assist with debugging on LocalNet
.
Use the same diagnostic tools for development as you will for production.
If you add a log line that allows you to identify and fix a bug in development,
then keeping it around at trace
or debug
log levels increases your operational readiness.
Using tools that won’t be available in production to debug in development reduces operational readiness.
Direct Postgres access¶
All persistent state in the example application is stored in one or more postgres databases.
You can use the postgres configuration in .env
to connect directly to these instances.
$ docker exec -it <postgres container> psql -v --username <.env username> --dbname <.env dbname> --password
For example: if you connect to the postgres-splice-app-provider
container (default username cnadmin
, dbname scribe
, and password
supersafe
; then you can use the SQL interface to PQS to examine the
app-provider’s participant’s local ledger. The SQL API to PQS is
documented in the daml documentation.
Interactive debugger¶
If you review the compose.yaml
file and examine the configuration for backend-service you will see the lines:
backend-service:
environment:
...
JAVA_TOOL_OPTIONS: "-javaagent:/otel-agent.jar
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
ports:
- "${BACKEND_PORT}:8080"
- "5055:5005"
This enables remote debugging of the Java component backend in the user application (backend-service). You can use this to connect an IDE Debugger to the service at runtime if required. We recommend Grafana as your first resort, along with the consolidated logs in Loki. This keeps the system debuggable in production.
Observability and tracing¶
Faulty distributed systems can be notoriously hard to diagnose. From the start of a project, Quickstart provides the sort of observability and diagnostics facilities that are otherwise often only developed toward the end. Simplifying diagnostics for new Canton Network Applications from the outset of each project is one of the motivations behind the development of Quickstart.
The links in the overview include the official user and reference documentation for the various tools included in Quickstart. While there is no substitute for the official documentation, it is hoped the following tour of the capabilities configured into Quickstart can provide a starting point for your own experimentation.
Correlation identifiers¶
Inspecting Canton begins by correlating identifiers, much like inspecting any other distributed system. Canton can accept and/or generate a number of identifiers suitable for correlating across both time, various nodes, and the evolving state of the ledger.
A few of the key identifiers to be aware of are:
|
|
|
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The goal of the observability configuration is to make it easier to navigate through the provenance of any state or event in the wider system. Any or all of these identifiers can be used to correlate a combination of logs, metrics, and state. Three of these in particular are intended to be set to corresponding business identifiers derived from your specific business domain: application-id, workflow-id, and command-id.
Navigation is enabled by the use of structured logs from as many components as possible [9]. It is recommended that your custom components likewise emit structured logs for more accurate consumption by OpenTelemetery.
Direct Ledger inspection using correlation identifiers¶
Starting from $ make stop clean-all && make build start
, we proceed with initiating the example application app-user onboarding:
$ make create-app-install-request | cat -n
docker compose -f docker/app-user-shell/compose.yaml --env-file .env run --rm create-app-install-request || true
get_token ledger-api-user AppProvider
get_user_party AppProvider participant-app-provider
http://participant-app-provider:7575/v2/users/AppProvider
get_token ledger-api-user Org1
get_user_party Org1 participant-app-user
http://participant-app-user:7575/v2/users/Org1
get_token administrator Org1
http://validator-app-user:5003/api/validator/v0/scan-proxy/dso-party-id
http://participant-app-user:7575/v2/commands/submit-and-wait
--data-raw {
"commands" : [
{
"CreateCommand" : {
"template_id":
"#quickstart-licensing:Licensing.AppInstall:AppInstallRequest",
"create_arguments": {
"dso":
"DSO::1220015e721c8ec5c1a5868b418442f064530e367c2587a9b43bd66f58c7bfddfec4",
"provider":
"AppProvider::12202fe7b2bf950dca3858b880d9ee0dd58249af8821ff2330ea1b80420852e816ff",
"user":
"Org1::122072b20a515d939910f9412f915cff8c1a7a427ddde76c6d0b7646d0022d4d4551",
"meta": {"values": []}
}
}
}
],
"workflow_id" : "create-app-install-request",
"application_id": "ledger-api-user",
"command_id": "create-app-install-request",
"deduplication_period": { "Empty": {} },
"act_as":
["Org1::122072b20a515d939910f9412f915cff8c1a7a427ddde76c6d0b7646d0022d4d4551"],
"read_as":
["Org1::122072b20a515d939910f9412f915cff8c1a7a427ddde76c6d0b7646d0022d4d4551"],
"submission_id": "create-app-install-request",
"disclosed_contracts": [],
"domain_id": "",
"package_id_selection_preference": []
}
{"update_id":
"1220e48d6d59af99a1b61eca414fe25766c342bb4e7d8d485e049a11a7f2267ed5c0",
"completion_offset":73}
This is the output of a script submitting a create command to the app-user’s participant node, it already contains number of the correlation ids mentioned above:
14 |
TemplateId |
#quickstar t-licensing:Licensing.AppInstall:AppInstallRequest |
16 - 18 |
Party Ids |
DSO::1220015e721c8ec5c1a5868b…ddfec4 AppProvider::12202fe7b2bf950d…e816ff Org1::122072b20a515d939910f94…4d4551 |
25 |
Workflow Id |
create-app-install-request |
26 |
Application Id |
ledger-api-user |
27 |
Command Id |
create-app-install-request |
31 |
Submission Id |
create-app-install-request |
36 |
Transaction Id |
1220e48d6d59af99a1b61eca414fe…7ed5c0 |
We can immediately use the transaction id in Daml Shell to view the associated ledger transaction:
$ make shell
docker compose -f docker/daml-shell/compose.yaml --env-file .env run --rm daml-shell || true
Connecting to jdbc:postgresql://postgres-splice-app-provider:5432/scribe...
Connected to jdbc:postgresql://postgres-splice-app-provider:5432/scribe
postgres-splice-app-provider:5432/scribe> transaction 1220e48d6d59af99a1b61eca414fe25766c342bb4e7d8d485e049a11a7f2267ed5c0
transactionId: 1220e48d6d59af99a1b61eca414fe25766c342bb4e7d8d485e049a11a7f2267ed5c0, offset: 48, workflowId: create-app-install-request - Feb 17, 2025, 5:26:09 AM
+ #1220e48d6d59af99a1b61eca414fe25766c342bb4e7d8d485e049a11a7f2267ed5c0:0
quickstart-licensing:Licensing.AppInstall:AppInstallRequest (005c17f89b7fd1d5fde9c548740c32924edeeddacc6320256892636b4e3b7d66aaca1)
{"dso": "DSO::1220015e721c8ec5c1a5868b418442f064530e367c2587a9b43bd66f58c7bfddfec4", "meta": {"values": []}, "user": "Org1::122072b20a515d939910f9412f915cff8c1a7a427ddde76c6d0b7646d0022d4d4551", "provider": "AppProvider::12202fe7b2bf950dca3858b880d9ee0dd58249af8821ff2330ea1b80420852e816ff"}
postgres-splice-app-provider:5432/scribe 3f → 48>
From here we can get more identifiers:
Ledger Offset |
48 |
Ledger Event Id |
#122026e55e3f82e27542…:0 |
Contract Id |
00cb53139ff0eb7ec57b… |
The Workflow Id, Template Id, and Party Ids are also visible here. The ledger offset can be very useful if you are going to query PQS or the Ledger API directly for more information. The Contract Id can be used to immediately display the contract in Daml Shell:
postgres-splice-app-provider:5432/scribe 3f → 48> contract 005c17f89b7fd1d5fde9c548740c32924edeeddacc6320256892636b4e3b7d66aaca101220777c5420863adb012c4f38847049346014c44eba7cd54bf58950dd6a18679053
╓───────────────────────────────────────────────────────────────────────────╖
| identifier: quickstart-licensing:Licensing.AppInstall:AppInstallRequest |
| Type: Template |
| Created at: 48 (not yet active) |
| Archived at: <active> |
| Contract ID: 005c17f89b7fd1d5fde9c548740c32924edeeddacc6320256892636b... |
| Event ID: #1220e48d6d59af99a1b61eca414fe25766c342bb4e7d8d485e049a11a7... |
| Contract Key: |
| Payload: dso:1220015e721c8ec5c1a5868b418442f064530e367c2587a9b43bd66f5... |
| meta: |
| values: [] |
| user: Org1:122072b20a515d939910f9412f915cff8c1a7a427ddde76c6d0b7646d00... |
| provider: AppProvider:12202fe7b2bf950dca3858b880d9ee0dd58249af8821ff23... |
╙───────────────────────────────────────────────────────────────────────────╜
postgres-splice-app-provider:5432/scribe 3f → 48>
If the problem is a bug in your smart contract, then exploring the transaction and related provenance within Daml Shell and using the Daml IDE to synthesize and rerun the relevant transactions will normally identify the issue. However, if only due to the comparative lines of code, the root cause of most issues will be off ledger. Consequently, significant value in these identifiers derives from correlating these identifiers with the consolidated logs and other information collected through Open Telemetry.