> ## Documentation Index
> Fetch the complete documentation index at: https://docs.digitalasset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> Monitoring and Alerting Best Practices

## Monitoring

The Covalidations Service contains metrics/traces out of the box to support monitoring. This behavior is enabled by adding the following block
to your configuration:

```yaml theme={null}
  monitoring:
    otel:
      endpointURL: <OTEL Monitoring URL>
```

The `endpointURl` field is an OpenTelemetry(OTEL) URL and port for sending metrics and traces. Different solutions are available to configure an
OTEL endpoint ie. Datadog-agents.

Logs in the Covalidation Service can be made more verbose by adding the following to your configuration:

```yaml theme={null}
  log:
    level: debug
```

It is good practice to monitor for things like :

* Kubernetes liveness/readiness probes for the covalidation-service is passing
* No ERROR log entries observed
* Log indicating last checkpoint processed is progressing

***

## Metrics

The covalidation service offers metrics out of the box for both successful and failing operations. Metrics embedded in the code include:

* covalidation.error: Metric counting all the errors encountered during the various syncs that occur during the covalidation process
* covalidation.github.dars\_uploaded: Metric measuring the total sum of dars uploaded to participant
* sync: Metric measuring the total number of sync attempts by the covalidation service
* offset: Metric capuring the current highest offset of the ledger API

Additionally, a `sync` trace is also included to capture a sync and raise any errors that may occur during the three covalidation processes:

* `sync`: Overarching trace that includes 3 subtraces for the 3 processes of covalidation
* `github`: Trace for the `Github` dar upload service
* `vetter`: Trace for the `Vetter` package vetting service
* `partysyncer`: Trace for the `Partysyncer` transaction sync service

## Any errors that occurs during one of the services are captured as a span event and appear under the respective trace to help with debugging service issues.

## Alerts

We recommend creating the following alerts for overall infrastructure and application health:

* Memory usage over time: Set alert on memory usage of the node the service is hosted on
* Kubernetes Deployment Replica Pods: Ensure pods are READY and healthy
* Covalidation Error Rate Over Time: Change in error rate in the previous X amount of minutes compared to the last Y minutes

***