Configure

PQS ascertains its configuration from the following sources, in order of priority:

• HOCON configuration files (--config argument)

• command-line arguments

• Java system properties (-D arguments to java command)

• environment variables

Note

Consult the command ./scribe.jar pipeline --help-verbose for further information on individual configuration items, and the conventions used to specify them in the above forms.

PQS sets its configuration at startup. It does not perform dynamic configuration updates, so making a configuration change (such as adding a new party, new template, or new interface to filters) requires a restart.

Warning

PQS will not go back in time and recover history, but only move forward by consuming new transactions it has not previously seen. It is important that scope only be expanded when it is known to have no prior history, at the point in time that PQS was stopped. Otherwise, a re-seed operation will be required to reinitialize from an empty datastore.

The following command connects to a non-auth ledger and replicates the latest state of the ledger (excluding prior-history) from the perspective of the supplied Daml party. It uses the ledger source and supplied database:

$ ./scribe.jar pipeline ledger postgres-document \
     --pipeline-filter-parties "Alice::* | Bob::*" \
     --pipeline-ledger-start Latest \
     --source-ledger-host canton-host \
     --source-ledger-port 6865 \
     --target-postgres-host postgres-host \
     --target-postgres-port 5432 \
     --target-postgres-database postgres

Consult Configuration options how to configure each command.

Transactions data source

To understand how PQS stores data, you need to understand the ledger data model https://docs.daml.com/concepts/ledger-model/ledger-structure.html. In simple terms, the Daml ledger is composed of a sequence of transactions, which contain events. Events can be:

• Create: creation of contracts / interface views

• Exercise: of a choice of contracts / interface views

• Archive: end of the lifetime of contracts / interface views

Warning

When defining the scope of ledger data being stored, it is important to understand the implications of the data source and the filters applied. The data source and filters determine the data that is available to the SQL functions (see SQL API), and this cannot be changed. Since a change in scope will result in a change to the breadth of data being stored, a re-seed is required to widen or narrow the scope of the data. The only exception to this is where you widen the scope into an area (e.g. new templates, new parties) that you know has no historical data, in which case a re-seed is not required. Or, operators may also the reset function to roll-back the datastore to a prior state where this was true.

PQS can run in two modes as specified by the --pipeline-datasource configuration. The following table shows the differences between the two modes, in terms of data availability via the respective SQL functions:

Data / Mode	TransactionStream	TransactionTreeStream
creates() of contracts	✓	✓
exercises() on contracts	✗	✓
archives() of contracts	✓	✓
creates() of interfaces	✓	✗
exercises() on interfaces	✗	✓
archives() of interfaces	✓	✗
Transient (create-archive in a transaction)	✗	✓
Stakeholders	✓	✓
Note
Default	✓	✗
Data size	Smaller	Larger

Contract filtering

--pipeline-filter-contracts specifies a filter expression to determine which the Daml templates, interface views and choices to include. A filter expression is a simple wildcard inclusion (*) with basic boolean logic (& ! | ( )), where whitespace is ignored. For example:

• *: everything (default)

• pkg:*: everything in this package

• pkg@=1: everything in this package that has major version 1.x.x

• pkg@>=1.2.3: everything in this package starting with version 1.2.3 inclusively

• foo@1.2.3|bar@3.2.1: everything in pinpointed packages foo at version 1.2.3 and package bar at version 3.2.1

• pkg:a.b.c.Bar: just this one fully qualified name for template Bar

• a.b.c.*: all members of the a.b.c namespace

• * & !pkg:a.b.c.Bar: everything except this one fully qualified name

• (a.b.c.Foo | a.b.c.Bar): these two fully qualified names

• (a.b.c.* & !(a.b.c.Foo | a.b.c.Bar) | g.e.f.Baz): everything in a.b.c except for Foo and Bar, and also include g.e.f.Baz

• a.b.c.Foo & a.b.c.Bar: error (the identifier can’t be both)

Note

There are further conditions placed upon the filtering of templates and interfaces to avoid potential ambiguity. It is required to include any filter for:

• All interface views of included templates

• All templates of included interface views

Party filtering

Similarly, the --pipeline-filter-parties option specifies a filter expression to determine which parties to supply data for. For example:

• *: everything (default)

• Alice::* | Bob::*: any party with an Alice or Bob hint

• Alice::122055fc4b190e3ff438587b699495a4b6388e911e2305f7e013af160f49a76080ab: just this one party

• * & !Alice::*: all parties except those with an Alice hint

• Alice* | Bob* | (Charlie* & !(Charlie3::*)): Alice and Bob parties, as well as Charlie except Charlie3

When the ledger requires authentication, this filter applies within the scope of parties for which PQS’ Ledger API user has access. Naturally, the --pipeline-filter-parties cannot be used to access data for parties for which the user is not authorized.

Explicit contract disclosure

--pipeline-filter-metadata specifies an inclusion filter expression to determine the Daml templates and interface views to capture metadata for. Same syntax as section above applies. Captured data will be available in the metadata column (bytea PostgreSQL type) in the query functions output. This column stores the contents of created_event_blob (see How do stakeholders disclose contracts to submitters?) from the respective event.