# Export Teleport Audit Events to Splunk

Teleport's Event Handler plugin receives audit events from the Teleport Auth Service and forwards them to your log management solution, letting you perform historical analysis, detect unusual behavior, and form a better understanding of how users interact with your Teleport cluster.

In this guide, we will show you how to configure the Teleport Event Handler plugin to send your Teleport audit events to Splunk.

## How it works

The Teleport Event Handler authenticates to the Teleport Auth Service to receive audit events over a gRPC stream, sends them to a local Fluentd instance as HTTP requests. The Fluentd instance forwards these requests to the Splunk HTTP Event Collector (HEC), which in turn sends them to Splunk Cloud Platform or Splunk Enterprise for visualization and alerting.

## Prerequisites

- Splunk Cloud Platform or Splunk Enterprise v9.0.1 or above.
- Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new fluent.conf file you can integrate into an existing Fluentd system, or use with a fresh setup.
- A server, virtual machine, Kubernetes cluster, or Docker environment to run the Teleport Event Handler plugin.
- On Splunk Enterprise, port `8088` should be open to traffic from the host running the Teleport Event Handler and Fluentd instance.

This guide requires you to have completed one of the Event Handler setup guides:

- [Set up the Event Handler with tctl](https://goteleport.com/docs/zero-trust-access/export-audit-events/event-handler-setup.md)
- [Set up the Event Handler with the Teleport Kubernetes Operator](https://goteleport.com/docs/zero-trust-access/export-audit-events/event-handler-setup-operator.md)

The instructions below demonstrate a local test of the Event Handler plugin on your workstation. You will need to adjust paths, ports, and domains for other environments.

## Step 1/4. Configure Splunk

In this section, you will configure Splunk to ingest and index Teleport audit events, and create a token for the HEC to authenticate to Splunk. All steps within this section take place within the Splunk web interface.

### Create an index for Teleport audit events

1. Visit the home page of the Splunk UI and navigate to **Settings > Indexes**. Click **New Index**. Name your index `teleport-audit-logs` and assign the **Index Data Type** field to `Events`.
2. Fill in the values of the remaining fields, **Max raw data size** and **Searchable retention (days)** based on the needs of your organization.
3. Click **Save**.

### \[Optional] Create a source type for Teleport audit events

By default, Splunk's `_json` source type expects the `time` field to be in a different format than that of Teleport audit events. This means that the event will show up in Splunk at the time it was ingested, not at the time it was generated. Adjust the format of the `time` field in Splunk so Teleport audit events appear as expected.

1. Navigate to **Settings -> Source Types**.
2. Find the `_json` source type and click **Clone**.
3. Name the new source type `_json-gotime`.
4. Under **Timestamp** click **Advanced**.
5. Input `%Y-%m-%dT%H:%M:%S.%3NZ` and click **Save**.

### Create a token for the HTTP Event Collector

1. Visit the home page of the Splunk UI.
2. Navigate to **Settings > Data inputs**.
3. In the **Local inputs** table, find the HTTP Event Collector row and click **Add new**.
4. Enter a name you can use to recognize the token later so you can manage it, e.g., `Teleport Audit Events`.
5. **Click Next**.
6. In the **Input Settings** view, next to the **Source type** field, click **Select**.
7. In the **Select Source Type** dropdown menu, click **Structured**, then choose the `_json-gotime` type you created earlier. If you skipped that optional step, choose `_json`. Splunk will index incoming logs as JSON, which is the format the Event Handler uses to send logs to Splunk.
8. In the **Index** section, select the `teleport-audit-logs` index you created earlier.
9. Click **Review,** then view the summary and click **Submit**.
10. Copy the **Token Value** field and assign TOKEN to it so you can use it later in this guide.

## Step 2/4. Connect Fluentd to the Splunk HTTP Event Collector

While Splunk used to maintain a dedicated output plugin at <https://github.com/splunk/fluent-plugin-splunk-hec>, this has been deprecated. We can still connect Fluentd to the HEC using Fluentd's built-in HTTP output plugin.

In Step 1, you generated a configuration file for Fluentd at `fluent.conf`. Edit your `fluent.conf` file as follows:

1. Edit the `<parse>` section with the following section:

   ```
       <parse>
         @type json
         json_parser oj

         # This time format is used by the plugin. This field is required.
         time_type string
         time_format %Y-%m-%dT%H:%M:%S
         keep_time_key true
       </parse>

   ```

   This will preserve the `time` field in the JSON when it is sent to Splunk. Some Fluentd output plugins require having a parsed time available.

2. Assign splunk-endpoint to your Splunk HTTP Event Collector endpoint.

   For Splunk Enterprise, this will be of the format:

   - `https://<host>:8088/services/collector/raw`

   For Splunk Cloud, this will be of the format:

   - AWS: `https://http-inputs-<host>.splunkcloud.com:443/services/collector/raw`
   - Google Cloud / Azure: `https://http-inputs.<host>.splunkcloud.com:443/services/collector/raw`
   - Free Trial: `https://<host>.splunkcloud.com:8088/services/collector/raw`

   For more information on Splunk HEC URIs, see [Splunk HTTP Event Collector ](https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/10.2/get-data-with-http-event-collector/set-up-and-use-http-event-collector-in-splunk-web#send-data-to-http-event-collector-on-splunk-cloud-platform-0).

3. Edit the `<match test.log>` section with the following section.

   ```
    <match test.log>
      @type http
      endpoint splunk-endpoint
      headers {"Authorization": "Splunk TOKEN"}
      # tls_verify_mode none
      <buffer>
        flush_interval 2s
      </buffer>
    </match>

   ```

   This configures Fluentd to authenticate to the HEC using the token you generated earlier. If using Splunk Cloud Free Trial, uncomment `tls_verify_mode none` to allow self-signed certificates.

4. Test your changes by running Fluentd:

   ```
   $ docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge
   ```

## Step 3/4. Run the Event Handler plugin

Now that you have configured your Fluentd instance to receive logs via HTTP and forward them to Splunk, you will modify the Event Handler configuration and run the Event Handler to test your configuration.

### Configure the Event Handler

Edit the configuration for the Event Handler, depending on your installation method.

**Executable**

Earlier, we generated a file called `teleport-event-handler.toml` to configure the Teleport Event Handler. This file includes setting similar to the following:

```
storage = "./storage"
timeout = "10s"
batch = 20
# concurrency is the number of concurrent sessions to process. By default, this is set to 5.
concurrency = 5
# The window size configures the duration of the time window for the event handler
# to request events from Teleport. By default, this is set to 24 hours.
# Reduce the window size if the events backend cannot manage the event volume
# for the default window size.
# The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
window-size = "24h"
# types is a comma-separated list of event types to search when forwarding audit
# events. For example, to limit forwarded events to user logins
# and new Access Requests, you can assign this field to
# "user.login,access_request.create".
types = ""
# skip-event-types is a comma-separated list of audit log event types to skip.
# For example, to forward all audit events except for new app deletion events,
# you can include the following assignment:
# skip-event-types = ["app.delete"]
skip-event-types = []
# skip-session-types is a comma-separated list of session recording event types to skip.
# For example, to forward all session events except for malformed SQL packet
# events, you can include the following assignment:
# skip-session-types = ["db.session.malformed_packet"]
skip-session-types = []

[forward.fluentd]
ca = /home/bob/event-handler/ca.crt
cert = /home/bob/event-handler/client.crt
key = /home/bob/event-handler/client.key
url = "https://fluentd.example.com:8888/test.log"
session-url = "https://fluentd.example.com:8888/session"

[teleport]
addr = teleport.example.com:443
identity = "identity"

```

**Helm Chart**

Earlier, we generated a file called `teleport-plugin-event-handler-values.yaml` to configure the Teleport Event Handler. This file includes setting similar to the following:

```
eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  # concurrency is the number of concurrent sessions to process. By default, this is set to 5.
  concurrency: 5
  # The window size configures the duration of the time window for the event handler
  # to request events from Teleport. By default, this is set to 24 hours.
  # Reduce the window size if the events backend cannot manage the event volume
  # for the default window size.
  # The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
  windowSize: "24h"
  # types is a list of event types to search when forwarding audit
  # events. For example, to limit forwarded events to user logins
  # and new Access Requests, you can assign this field to:
  # ["user.login", "access_request.create"]
  types: []
  # skipEventTypes lists types of audit events to skip. For example, to forward all
  # audit events except for new app deletion events, you can assign this to:
  # ["app.delete"]
  skipEventTypes: []
  # skipSessionTypes lists types of session recording events to skip. For example,
  # to forward all session events except for malformed SQL packet events,
  # you can assign this to:
  # ["db.session.malformed_packet"]
  skipSessionTypes: []

teleport:
  address: teleport.example.com:443
  identitySecretName: teleport-event-handler-identity
  identitySecretPath: identity

fluentd:
  url: "https://fluentd.fluentd.svc.cluster.local/events.log"
  sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

```

**Helm Chart with Kubernetes Operator**

Your helm configuration file `teleport-plugin-event-handler-values.yaml` should contain settings similar to the following:

```
eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  # concurrency is the number of concurrent sessions to process. By default, this is set to 5.
  concurrency: 5
  # The window size configures the duration of the time window for the event handler
  # to request events from Teleport. By default, this is set to 24 hours.
  # Reduce the window size if the events backend cannot manage the event volume
  # for the default window size.
  # The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
  windowSize: "24h"
  # types is a list of event types to search when forwarding audit
  # events. For example, to limit forwarded events to user logins
  # and new Access Requests, you can assign this field to:
  # ["user.login", "access_request.create"]
  types: []
  # skipEventTypes lists types of audit events to skip. For example, to forward all
  # audit events except for new app deletion events, you can assign this to:
  # ["app.delete"]
  skipEventTypes: []
  # skipSessionTypes lists types of session recording events to skip. For example,
  # to forward all session events except for malformed SQL packet events,
  # you can assign this to:
  # ["db.session.malformed_packet"]
  skipSessionTypes: []

crd:
  create: true
  namespace: operator-namespace

tbot:
  enabled: true
  clusterName: teleport.example.com
  teleportProxyAddress: teleport.example.com:443

fluentd:
  url: "https://fluentd.fluentd.svc.cluster.local/events.log"
  sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

```

Update the following fields.

**Executable**

**`[teleport]`**

`addr`: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443

`identity`: Fill this in with the path to the identity file you exported earlier.

If you are providing credentials to the Event Handler using a `tbot` binary that runs on a Linux server, make sure the value of `identity` in the Event Handler configuration is the same as the path of the identity file you configured `tbot` to generate, `/opt/machine-id/identity`.

**`[forward.fluentd]`**

`ca`: Include the path to the CA certificate: /home/bob/event-handler/ca.crt

`cert`: Include the path to the Fluentd client certificate. /home/bob/event-handler/client.crt

`key`: Include the path to the Fluentd client key. /home/bob/event-handler/client.key

`url`: Include the Fluentd URL where the audit event logs will be sent.

`session-url`: Include the Fluentd URL where the session logs will be sent.

**Helm Chart**

**`teleport`**

`address`: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443

`identitySecretName`: Fill in the `identitySecretName` field with the name of the Kubernetes secret you created earlier.

`identitySecretPath`: Fill in the `identitySecretPath` field with the path of the identity file within the Kubernetes secret. If you have followed the instructions above, this will be `identity`.

**`fluentd`**

`url`: Include the Fluentd URL where the audit event logs will be sent.

`sessionUrl`: Include the Fluentd URL where the session logs will be sent.

`certificate.secretName`: Include the name of the Kubernetes secret containing the Fluentd client credentials. If you have followed the instructions above, this will be `teleport-event-handler-client-tls`.

`certificate.caPath`: Include the path to the CA certificate inside the secret.

`certificate.certPath`: Include the path to the Fluentd client certificate inside the secret.

`certificate.keyPath`: Include the path to the Fluentd client key inside the secret.

**Helm Chart with Kubernetes Operator**

**`crd`**

`namespace`: Include the namespace that the Teleport Kubernetes Operator is running in: operator-namespace

`tokenSpecOverride`: Optionally include a specific join token specification for the bot user that `tbot` will authenticate as.

**`tbot`**

`clusterName`: Include the name of your Teleport cluster: teleport.example.com

`teleportProxyAddress`: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443

**`fluentd`**

`url`: Include the Fluentd URL where the audit event logs will be sent.

`sessionUrl`: Include the Fluentd URL where the session logs will be sent.

`certificate.secretName`: Include the name of the Kubernetes secret containing the Fluentd client credentials. If you have followed the instructions above, this will be `teleport-event-handler-client-tls`.

`certificate.caPath`: Include the path to the CA certificate inside the secret.

`certificate.certPath`: Include the path to the Fluentd client certificate inside the secret.

`certificate.keyPath`: Include the path to the Fluentd client key inside the secret.

### Start the Event Handler

Start the Teleport Event Handler by following the instructions below.

**Linux server**

Copy the `teleport-event-handler.toml` file to `/etc` on your Linux server. Update the settings within the `toml` file to match your environment. Make sure to use absolute paths on settings such as `identity` and `storage`. Files and directories in use should only be accessible to the system user executing the `teleport-event-handler` service such as `/var/lib/teleport-event-handler`.

Next, create a systemd service definition at the path `/usr/lib/systemd/system/teleport-event-handler.service` with the following content:

```
[Unit]
Description=Teleport Event Handler
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled=true
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-event-handler.pid

[Install]
WantedBy=multi-user.target

```

If you are not using Machine & Workload Identity to provide short-lived credentials to the Event Handler, you can remove the `--teleport-refresh-enabled true` flag.

Enable and start the plugin:

```
$ sudo systemctl enable teleport-event-handler
$ sudo systemctl start teleport-event-handler
```

Choose when to start exporting events

You can configure when you would like the Teleport Event Handler to begin exporting events when you run the `start` command. This example will start exporting from May 5th, 2021:

```
$ teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"
```

You can only determine the start time once, when first running the Teleport Event Handler. If you want to change the time frame later, remove the plugin state directory that you specified in the `storage` field of the handler's configuration file.

Once the Teleport Event Handler starts, you will see notifications about scanned and forwarded events:

```
$ sudo journalctl -u teleport-event-handler
DEBU   Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-21
18:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140
...
```

**Helm chart**

Run the following command on your workstation:

```
$ helm install teleport-plugin-event-handler teleport/teleport-plugin-event-handler \
  --values teleport-plugin-event-handler-values.yaml \
  --version 18.7.3
```

**Local Docker container**

Navigate to the directory where you ran the `configure` command earlier and execute the following command:

```
$ docker run --network host -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:18.7.3 start --config=teleport-event-handler.toml
```

This command joins the Event Handler container to the preset `host` network, which uses the Docker host networking mode and removes network isolation, so the Event Handler can communicate with the Fluentd container on `localhost`.

## Step 4/4. Visualize your audit events in Splunk

Since our setup forwards audit events to Splunk in the structured JSON format, Splunk automatically indexes them, so fields will be available immediately for use in visualizations. You can use these fields to create dashboards that track the way users are interacting with your Teleport cluster.

For example, from the Splunk UI home page, navigate to **Search & Reporting** > **Dashboards** > **Create New Dashboard**. Enter "Teleport Audit Log Types" for the title of your dashboard and click **Classic Dashboards**. Click **Create** then, in the **Edit Dashboard** view, click **Add Panel**.

In the **Add Panel** sidebar, click **New** > **Column Chart**. For the **Search String** field, enter the following:

```
index="teleport-audit-logs" | timechart count by event

```

Once you click **Add to Dashboard** you will see a count of Teleport event types over time, which gives you a general sense of how users are interacting with Teleport:

![Event Types over Time](/docs/assets/images/splunk-dashboard-cff4b3f3c3706ac8317146373dfbc83f.png)

## Troubleshooting connection issues

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

- The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the `--ttl` flag in the `tctl auth sign` command, which is 12 hours by default.
- In your Teleport Event Handler configuration file, you have provided the correct host *and* port for the Teleport Proxy Service.

## Next steps

Now that you are exporting your audit logs to Splunk, consult our [audit log reference](https://goteleport.com/docs/reference/deployment/monitoring/audit.md) so you can plan visualizations and alerts.

- To see all of the options you can set in the values file for the `teleport-plugin-event-handler` Helm chart, consult our [reference guide](https://goteleport.com/docs/reference/helm-reference/teleport-plugin-event-handler.md).