# Access Requests with Datadog Incident Management With Teleport's Datadog Incident Management integration, engineers can access the infrastructure they need to resolve incidents without longstanding admin permissions that can become a vector for attacks. Teleport's Datadog Incident Management integration allows you to treat Teleport Role Access Requests as Datadog incidents, notify the appropriate on-call team, and approve or deny the requests via Teleport. This guide will explain how to set up Teleport's Access Request plugin for Datadog. ## How it works The Teleport Datadog Incident Management integration connects to the Teleport gRPC API and listens for Role Access Requests. When the integration receives a status change for a Role Access Request, it either creates a new incident in Datadog or, if an incident exists, updates it. The integration also reads annotations on each Access Request and, depending on the annotations, may approve the Access Request automatically. ## Prerequisites - A running Teleport Enterprise (v17.0.1 or higher) cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/get-started/deploy-community.md). - The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/installation.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` * A Datadog account with the role "Datadog Admin Role". The admin role is required to create a Service Account and generate required credentials for the plugin. You can see your role by visiting your user page in Datadog and navigating to **Personal Settings -> Profile** and checking the value of the **Roles** field. * Either a Linux host or Kubernetes cluster where you will run the Datadog plugin. * To check that you can connect to your Teleport cluster, sign in with `tsh login`, then verify that you can run `tctl` commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email\@example.com to your Teleport username: ``` $ tsh login --proxy=teleport.example.com --user=email@example.com $ tctl status Cluster teleport.example.com Version 18.7.3 CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 ``` If you can connect to the cluster and run the `tctl status` command, you can use your current credentials to run subsequent `tctl` commands from your workstation. If you host your own Teleport cluster, you can also run `tctl` commands on the computer that hosts the Teleport Auth Service for full permissions. * Your Teleport user will require the `editor` role or a role that provides permissions to create Teleport roles and grant them to users. ## Step 1/6. Create Datadog team To demonstrate the Datadog Incident Management plugin, create a team in Datadog. In the Datadog web UI, navigate to the **Teams** tab and click on **New Team** to create a new Datadog team. Name it "Teleport Access". We will configure the Datadog Incident Management plugin to route certain access requests to the "Teleport Access" team. ## Step 2/6. Define RBAC resources Before you set up the Datadog Incident Management plugin, you will need to enable Role Access Requests in your Teleport cluster. ### Create a requester and reviewer role For the purpose of this guide, we will define an `editor-requester` role, which can request the built-in `editor` role, and an `editor-reviewer` role that can review requests for the `editor` role. In the Teleport Web UI navigate to **Zero Trust Access -> Roles**. Then select **Create New Role** and create the desired roles. ``` kind: role version: v7 metadata: name: editor-reviewer spec: allow: review_requests: roles: ['editor'] --- kind: role version: v7 metadata: name: editor-requester spec: allow: request: roles: ['editor'] thresholds: - approve: 1 deny: 1 ``` ### Create a requester and reviewer user First, assign yourself the `editor-reviewer` role. This will allow your user to review Access Requests for the `editor` role. To edit your user roles navigate to **Zero Trust Access -> Users**, then for your user select **Options -> Edit** and add the `editor-reviewer` role. Next, create a user called `myuser@example.com` who has the `editor-requester` role. Later in this guide, you will create an Access Request as this user to test the Datadog plugin. To this user, navigate to **Zero Trust Access -> Users**. Then select **Enroll Users** and create a user with the `editor-requester` role. You should end up with two users that look like this: ![Teleport Users](/docs/assets/images/teleport-users-1d774a8d4957208ef27b6a691cde3926.png) ## Step 3/6. Generate Datadog credentials The Access Request plugin for Datadog receives Access Request events from the Teleport Auth Service, formats them into Datadog incidents, and sends them to the Datadog API. For this to work, you must provide a valid Datadog API and Application key to the Teleport plugin. ### Generate a Datadog API key First, in the Datadog web UI navigate to **Organization Settings -> Access -> API Keys** to view your organization's API keys. To generate a new API key, click on **New Key** and name it "Teleport Access". Copy the API key to paste in a later step. ![Datadog dashboard for API Keys](/docs/assets/images/dashboard-api-keys-02611ae102bc050d31894735ae52ab35.png) ### Create a Datadog Service Account Next, navigate to **Organization Settings -> Identity & Accounts -> Service Accounts** to view your organization's service accounts. To create a new service account, click on **New Service Account** and name it "Teleport Access". You can provide whatever email you'd like to associate with the service account, then assign the role "Datadog Standard Role". This role provides the required permissions necessary to manage the incidents in Datadog. ![Datadog dashboard for Service Accounts](/docs/assets/images/dashboard-service-accounts-ea0db2566e3bfb70bb0725510bdd3430.png) Now click on the "Teleport Access" service account that was just created and click on the **Application Keys** tab and click on **New Key** and name it "Teleport Access" to create a new Application key. Copy the Application key to paste in a later step. ![Datadog dashboard for Application Keys](/docs/assets/images/dashboard-application-keys-b706806737c92b8381333196c58ef9f9.png) ## Step 4/6. Enroll the Datadog Incident Management plugin At this point, you're now ready to enroll the Datadog Incident Management plugin. In Teleport Enterprise Cloud, Teleport manages the Datadog Incident Management for you, and you can enroll the Datadog Incident Management from the Teleport Web UI. Visit the Teleport Web UI and on the left sidebar, click **Add New** followed by **Integration**: ![Enroll an Access Request plugin](/docs/assets/images/enroll-ee64e35054da594e264c55422bf39c7b.png) On the "Select Integration Type" menu, click the tile for your integration. You will see a page with instructions to set up the integration, as well as a form that you can use to configure the integration. Provide the API and Application keys generated above. Select the desired API endpoint. Then provide the Datadog team handle, that you created earlier, as the fallback recipient. This should be "teleport-access". The fallback recipient will be the default recipient for notifications. The recipient can be a Datadog user email, or a Datadog team handle. You can configure more custom notification routing rules afterwards using [Access Monitoring Rules](https://goteleport.com/docs/identity-governance/access-requests/notification-routing-rules.md). If the recipient is a Datadog team, the team name will be added to the Datadog incident teams attribute. The plugin creates incidents and labels them with the recipient team. By default, the team members won't be notified, you must configure Datadog notification rules to page the team members. To do so, navigate to **Incidents -> Settings -> Notifications -> Rules** and create a new rule matching alerts labeled with the recipient team. ![Datadog dashboard for Incident Notification Rules](/docs/assets/images/dashboard-notification-rules-e055f7121366a400aa8ea0c6c89fdba1.png) Suggested reviewers Users can suggest specific reviewers when they create an Access Request by providing Datadog user emails, e.g.,: ``` $ tsh request create --roles=editor --reviewers=alice@example.com,ivan@example.com ``` If an Access Request includes suggested reviewers, the Datadog Incident Management plugin will add these to the list of users to notify. ## Step 5/6. Test your Datadog Incident Management plugin ### Create an Access Request As the Teleport user `myuser@example.com`, create an Access Request for the `editor` role. Navigate to the **Access Requests** tab and click on **New Access Request**: ![New access request](/docs/assets/images/create-access-request-c7c5775d178e067c9b652cbfa678e328.png) In Datadog, you will see a new incident containing information about the Access Request: ![Datadog dashboard showing an Access Request](/docs/assets/images/new-access-request-incident-bba1f9db2cad77180dba0611151737c3.png) ### Resolve the request Once you receive an Access Request message, click the link to visit Teleport and approve or deny the request: ![Review access request](/docs/assets/images/review-access-request-561fb9e3670523913212c5bdecc3aeaf.png) --- AUDITING ACCESS REQUESTS When the Datadog plugin sends a notification, anyone who receives the notification can follow the enclosed link to an Access Request URL. While users must be authorized via their Teleport roles to review Access Requests, you should still check the Teleport audit log to ensure that the right users are reviewing the right requests. When auditing Access Request reviews, check for events with the type `Access Request Reviewed` in the Teleport Web UI. --- ## Step 6/6. Configure Automatic Approvals ### Setup a Datadog on-call schedule To enable automatic approvals for Access Requests, you will need to set up an on-call schedule for your Datadog team. Navigate to **On-Call -> Teams** and select **Add Team to On-Call** to set up an on-call schedule for the "Teleport Access" team. ### Update Teleport RBAC to support automatic approvals Configure the Teleport `editor-requester` role and add the `teleport.dev/schedules` annotation. The `teleport.dev/schedules` annotation is used to specify on-call schedules, services, and teams for auto-approval. For the Datadog plugin, this annotation expects a list of Datadog team handle names. Add the "teleport-access" team to the annotation. Access requests made by an on-call user of this team will be automatically approved. ``` kind: role version: v7 metadata: name: editor-requester spec: allow: request: annotations: teleport.dev/schedules: - teleport-access roles: ['editor'] thresholds: - approve: 1 deny: 1 ``` --- TIP You can also create and edit roles using the Web UI. Go to **Access -> Roles** and click **Create New Role** or pick an existing role to edit. --- ### Trigger an auto-approval To trigger an auto-approval, login to Teleport as the current on-call user in Datadog, and create an Access Request for the `editor` role. Automatic approvals requires that the Teleport username matches the Datadog on-call user email. ## Troubleshooting Access Request plugins need permissions to list and read any Teleport resource types included in a request. This is because, when the plugin receives a resource request, it queries the Teleport Auth Service API for data about the requested resources. If you receive an error message similar to the following, the Teleport roles for the Access Request plugin's identity do not have permissions to perform one or more operations against the Teleport API. In the example below, the Access Request plugin needs `list` and `read` permissions on the `user_group` resource: ``` ERRO Failed to process request error:[ ERROR REPORT: Original Error: *interceptors.RemoteError access denied to perform action "list" on "user_group", access denied to perform action "read" on "user_group" ``` Make sure the Teleport roles for the Access Request plugin's identity include permissions to list requested resources. To resolve the error above, for example, you could grant the following role to the Access Request plugin's identity: ``` kind: role version: v7 metadata: name: read-user-groups spec: allow: rules: - resources: [user_group] verbs: [list, read] ``` ## Next steps - Read our guide on [Routing Access Request notifications](https://goteleport.com/docs/identity-governance/access-requests/notification-routing-rules.md) to configure custom notification routing rules for your plugin.