# Distributed Tracing

Teleport leverages [OpenTelemetry](https://opentelemetry.io/) to generate traces and export them to any [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) capable exporter. In the event that your telemetry backend doesn't support receiving OTLP traces, you may be able to leverage the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to proxy traces from OTLP to a format that your telemetry backend accepts.

## Configure Teleport

In order to enable tracing for a `teleport` instance, add the following section to that instance's configuration file (`/etc/teleport.yaml`). For a detailed description of these configuration fields, see the [configuration reference](https://goteleport.com/docs/reference/deployment/monitoring/tracing-service-configuration.md) page.

```
tracing_service:
   enabled: true
   exporter_url: grpc://collector.example.com:4317
   sampling_rate_per_million: 1000000

```

### Sampling rate

It is important to choose the sampling rate wisely. Sampling at a rate of 100% could have a negative impact on the performance of your cluster. Teleport honors the sampling rate included in any incoming requests, which means that even when the `tracing_service` is enabled and the sampling rate is 0, if Teleport receives a request that has a span which is sampled, then Teleport will sample and export all spans that are generated in response to that request.

### Exporter URL

The `exporter_url` setting indicates where Teleport should send spans to. Supported schemes are `grpc://`, `http://`, `https://`, and `file://` (if no scheme is provided, then `grpc://` is used).

When using `file://`, the url must be a path to a directory that Teleport has write permissions for. Spans will be saved to files within the provided directory, each file containing one proto encoded span per line. Files are rotated after exceeding 100MB, in order to override the default limit add `?limit=<desired_file_size_in_bytes>` to the `exporter_url` (i.e. `file:///var/lib/teleport/traces?limit=100`).

By default the connection to the exporter is insecure, to support TLS add the following to the `tracing_service` configuration:

```
   # Optional path to CA certificates are used to validate the exporter.
  ca_certs:
    - /var/lib/teleport/exporter_ca.pem
  # Optional path tp TLS certificates are used to enable mTLS for the exporter
  https_keypairs:
    - key_file: /var/lib/teleport/exporter_key.pem
      cert_file: /var/lib/teleport/exporter_cert.pem

```

After updating `teleport.yaml`, start your `teleport` instance to apply the new configuration.

## tsh

To capture traces from `tsh` simply add the `--trace` flag to your command. All traces generated by `tsh --trace` will be proxied to the `exporter_url` defined for the Auth Service of the cluster the command is being run on.

```
$ tsh --trace ssh root@myserver
$ tsh --trace ls
```

Exporting traces from `tsh` to a different exporter than the one defined in the Auth Service config is also possible via the `--trace-exporter` flag. A URL must be provided that adheres to the same format as the `exporter_url` of the `tracing_service`.

```
$ tsh --trace --trace-exporter=grpc://collector.example.com:4317 ssh root@myserver
$ tsh --trace --trace-exporter=file:///var/lib/teleport/traces ls
```