# Profiling

Teleport leverages Go's diagnostic capabilities to collect and export profiling data. Profiles can help identify the cause of spikes in CPU, the source of memory leaks, or the reason for a deadlock.

## Using the Debug Service

The Teleport Debug Service enables administrators to collect diagnostic profiles without enabling pprof endpoints at startup. The service, enabled by default, ensures local-only access and must be consumed from inside the same instance.

`teleport debug profile` collects a list of pprof profiles. It outputs a compressed tarball (`.tar.gz`) to STDOUT. You decompress it using `tar` or direct the result to a file.

By default, it collects `goroutine`, `heap` and `profile` profiles.

Each profile collected will have a correspondent file inside the tarball. For example, collecting `goroutine,trace,heap` will result in `goroutine.pprof`, `trace.pprof`, and `heap.pprof` files.

```
Collect default profiles and save to a file.
$ teleport debug profile > pprof.tar.gz
$ tar xvf pprof.tar.gz

Collect default profiles and decompress it.
$ teleport debug profile | tar xzv -C ./

Collect "trace" and "mutex" profiles and save to a file.
$ teleport debug profile trace,mutex > pprof.tar.gz

Collect profiles setting the profiling time in seconds
$ teleport debug profile -s 20 trace > pprof.tar.gz
```

---

SPECIFY YOUR TELEPORT CONFIGURATION PATH

If your Teleport configuration is not placed on the default path (`/etc/teleport.yaml`), you must specify its location to the CLI command using the `-c/--config` flag.

---

If you're running Teleport on a Kubernetes cluster you can directly collect profiles to a local directory without an interactive session:

```
$ kubectl -n teleport exec my-pod -- teleport debug profile > pprof.tar.gz
```

After extracting the contents, you can use `go tool` commands to explore and visualize them:

```
Opens the terminal interactive explorer
$ go tool pprof heap.pprof

Opens the web visualizer
$ go tool pprof -http : heap.pprof

Visualize trace profiles
$ go tool trace trace.pprof
```

## Using diagnostics endpoints

The profiling endpoint is only enabled if the `--debug` flag is supplied.

Teleport's diagnostic HTTP endpoints are disabled by default. You can enable them via:

**Command line**

Start a `teleport` instance with the `--diag-addr` flag set to the local address where the diagnostic endpoint will listen:

```
$ sudo teleport start --debug --diag-addr=127.0.0.1:3000
```

**Config file**

Edit a `teleport` instance's configuration file (`/etc/teleport.yaml` by default) to include the following:

```
teleport:
    diag_addr: 127.0.0.1:3000

```

To enable debug logs:

```
log:
    severity: DEBUG

```

Ensure you can connect to the diagnostic endpoint

Verify that Teleport is now serving the diagnostics endpoint:

```
$ curl http://127.0.0.1:3000/healthz
```

### Collecting profiles

Go's standard profiling endpoints are served at `http://127.0.0.1:3000/debug/pprof/`. Retrieving a profile requires sending a request to the endpoint corresponding to the desired profile type. When debugging an issue it is helpful to collect a series of profiles over a period of time.

#### CPU

CPU profile shows execution statistics gathered over a user specified period:

```
Download the profile into a file:
$ curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile?seconds=30

Visualize the profile
$ go tool pprof -http : cpu.profile
```

#### Goroutine

Goroutine profiles show the stack traces for all running goroutines in the system:

```
Download the profile into a file:
$ curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutine

Visualize the profile
$ go tool pprof -http : goroutine.profile
```

#### Heap

Heap profiles show allocated objects in the system:

```
Download the profile into a file:
$ curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heap

Visualize the profile
$ go tool pprof  -http : heap.profile
```

#### Trace

Trace profiles capture scheduling, system calls, garbage collections, heap size, and other events that are collected by the Go runtime over a user specified period of time:

```
Download the profile into a file:
$ curl -o trace.out http://127.0.0.1:3000/debug/pprof/trace?seconds=5

Visualize the profile
$ go tool trace trace.out
```

## Further Reading

- More information about diagnostics in the Go ecosystem: <https://go.dev/doc/diagnostics>
- Go's profiling endpoints: <https://golang.org/pkg/net/http/pprof/>
- A deep dive on profiling Go programs: <https://go.dev/blog/pprof>