# Teleport EKS Auto-Discovery EKS Auto-Discovery can automatically discover any EKS cluster and enroll it in Teleport if its tags match the configured labels. Teleport cluster auto-discovery involves two components: 1. The Teleport Discovery Service that watches for new clusters or changes to previously discovered clusters. It dynamically registers each discovered cluster as a `kube_cluster` resource in your Teleport cluster. It does not need connectivity to the clusters it discovers. 2. The Teleport Kubernetes Service that monitors the dynamic `kube_cluster` resources registered by the Discovery Service. It proxies communications between users and the cluster. --- TIP This guide presents the Discovery Service and Kubernetes Service running in the same process, however both can run independently and on different machines. For example, you can run an instance of the Kubernetes Service in the same private network as the clusters you want to register with your Teleport cluster, and an instance of the Discovery Service in any network you wish. --- ## How it works The Teleport Discovery Service scans configured cloud providers, including AWS, for Kubernetes clusters that match specified filtering labels, creating dynamic resources within Teleport for any new clusters identified. The Teleport Kubernetes Service monitors these dynamic resources, forwarding requests to the corresponding Kubernetes clusters. Both services require access to the AWS API to perform their functions. Additionally, the Kubernetes Service needs direct access to the target clusters and the necessary permissions to forward requests. ## Prerequisites - A running Teleport cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/get-started/deploy-community.md). - The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/installation.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` * An AWS account with permissions to create and attach IAM policies. * A host to run the Teleport Discovery and Kubernetes services. * One or more EKS clusters running. --- NOTE Starting with Teleport v15.3.8, the Discovery Service can self-bootstrap access to EKS clusters by automatically creating and managing Access Entries for each discovered cluster. This contrasts with earlier versions of EKS Auto-Discovery, where agents could not access a cluster without having pre-existing access configured. --- ## Step 1/3. Set up AWS IAM credentials Create and attach the following AWS IAM policy to the identity of the instance running the Teleport Discovery Service: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "EKSDiscovery", "Effect": "Allow", "Action": [ "eks:DescribeCluster", "eks:ListClusters" ], "Resource": "*" }, { "Sid": "EKSManageAccess", "Effect": "Allow", "Action": [ "eks:AssociateAccessPolicy", "eks:CreateAccessEntry", "eks:DeleteAccessEntry", "eks:DescribeAccessEntry", "eks:TagResource", "eks:UpdateAccessEntry" ], "Resource": "*" } ] } ``` | Statement | Purpose | | ----------------- | ----------------------------------------------------------------- | | `EKSDiscovery` | Discover EKS clusters and fetch additional details about them. | | `EKSManageAccess` | Automatically set up Teleport access for discovered EKS clusters. | You can use a list of ARNs and narrow the scope of the permissions to specific regions or EKS clusters instead of using a wildcard. The resource ARN has the following format: ``` arn:{Partition}:eks:{Region}:{Account}:cluster/{ClusterName} ``` The permissions in the `EKSManageAccess` statement are optional because the Discovery Service will discover EKS clusters even when it cannot ensure that the Teleport Kubernetes Service has access to the clusters it discovers. If you omit any of the `EKSManageAccess` permissions, then it is your responsibility to ensure that the Teleport Kubernetes Service can access each EKS cluster. ## Step 2/3. Configure EKS cluster authorization --- WARNING If you are running Teleport Discovery v15.3.8 or later and the IAM role used by the Discovery Service has the necessary permissions to create and update Access Entries, you can skip this section. The service can self-bootstrap the required permissions automatically. --- When the Kubernetes Service uses an IAM role that is different from the one that creates the clusters, you need to configure the mapping between the Teleport IAM Role and the Kubernetes RBAC permissions by editing the `aws-auth` `Configmap` on each of the discovered clusters. To forward requests to the Kubernetes cluster, the Teleport Kubernetes Service requires cluster-wide permissions to `Impersonate` RBAC users and groups, to create `SelfSubjectAccessReviews` and `SelfSubjectRulesReviews`, and, finally, read access to `Pods`. If your Kubernetes cluster does not have an RBAC group with the required permissions, you can create the `ClusterRole`, `ClusterRoleBinding`, and the mapping by following the Creating RBAC group guide. If your cluster already has an RBAC group that satisfies the required permissions, you can reuse it and map it into the IAM Role used by the Teleport Kubernetes Service. For simplicity, it is also possible to map the Teleport IAM role onto a built-in Kubernetes RBAC group like `system:masters`, but not recommended in production. **Creating RBAC group** Connect to your target cluster with your credentials and create the following resources using `kubectl`. ### ClusterRole Create the `ClusterRole` RBAC definition with the required permissions for Teleport Kubernetes Service to forward requests to the cluster. ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: teleport rules: - apiGroups: - "" resources: - users - groups - serviceaccounts verbs: - impersonate - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "authorization.k8s.io" resources: - selfsubjectaccessreviews - selfsubjectrulesreviews verbs: - create ``` ### ClusterRoleBinding Link the previously created `ClusterRole` into a `teleport` RBAC group. ``` kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: teleport roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: teleport subjects: - kind: Group name: teleport apiGroup: rbac.authorization.k8s.io ``` ### IAM mapping If your cluster includes the `aws-auth` config map, edit the `configmap/aws-auth` in the `kube-system` namespace and append the following to `mapRoles`. Replace `{teleport_aws_iam_role}` with the appropriate IAM role that Teleport Kubernetes Service will use. This step will link the Teleport IAM role into the Kubernetes RBAC group `teleport`, allowing Teleport Kubernetes Service to forward requests into the cluster: ``` apiVersion: v1 data: mapRoles: | - groups: - teleport rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role username: teleport ``` Otherwise, create an EKS access entry to link the arn:aws:iam::222222222222:role/teleport-role to the Kubernetes Group `teleport` we created in the previous step: ``` $ aws eks create-access-entry \ --cluster-name eks-cluster \ --region eu-west-1 \ --principal-arn arn:aws:iam::222222222222:role/teleport-role \ --kubernetes-groups teleport { ... } ``` At this point, the Teleport IAM role already has the minimum permissions to forward requests to the cluster. **Reuse an existing RBAC Group** To associate the Teleport IAM role with an existing Kubernetes RBAC group, edit the `configmap/aws-auth` in the `kube-system` namespace and append the following to `mapRoles`. ``` apiVersion: v1 data: mapRoles: | ... - groups: - {rbac_group} rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role username: teleport ``` Please replace `{teleport_aws_iam_role}` with the appropriate IAM role that Teleport Kubernetes Service is using and `{rbac_group}` with the existing Kubernetes RBAC Group that satisfies the required permissions. At this point, the Teleport IAM role already has the minimum permissions to forward requests to the cluster. **Use system:masters** --- WARNING Granting the `system:masters` group to the IAM role associated with the Teleport service means granting administrator-level permissions on the Kubernetes cluster. To follow least privilege principle we do not recommend using this method in production. --- If your cluster contains an `aws-auth` config map, you can use this to associate the Teleport IAM role with the `system:masters` RBAC group. Edit the `configmap/aws-auth` in the `kube-system` namespace and append the following to `mapRoles`: ``` apiVersion: v1 data: mapRoles: | ... - groups: - system:masters rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role username: teleport ``` Replace `{teleport_aws_iam_role}` with the appropriate IAM role that Teleport Kubernetes Service is using. Otherwise, create an EKS access entry and Access Policy to link the arn:aws:iam::222222222222:role/teleport-role to the cluster wide policy `arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy` (equivalent of `cluster-admin` `ClusterRole`): ``` $ aws eks create-access-entry \ --cluster-name eks-cluster \ --region eu-west-1 \ --principal-arn arn:aws:iam::222222222222:role/teleport-role { ... } $ aws eks associate-access-policy \ --cluster-name eks-cluster \ --region eu-west-1 \ --principal-arn arn:aws:iam::222222222222:role/teleport-role \ --policy-arn "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy" \ --access-scope type=cluster { ... } ``` At this point, the Teleport IAM role already has the minimum permissions to forward requests to the cluster. --- TIP If you provision your EKS clusters using tools such as `terraform`, `eksctl` or `Cloudformation`, you can use them to automatically configure the `aws-auth` `Configmap` or access entry and create the `ClusterRole` and `ClusterRoleBinding` resources during cluster provisioning. --- ## Step 3/3. Configure Teleport to discover EKS clusters ### Install Teleport Install Teleport on the host you are using to run the Kubernetes Service and Discovery Service: To install a Teleport Agent on your Linux server: The recommended installation method is the cluster install script. It will select the correct version, edition, and installation mode for your cluster. 1. Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https\://). 2. Run your cluster's install script: ``` $ curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash ``` ### Get a join token Teleport EKS Auto-Discovery requires a valid Teleport auth token for the Discovery and Kubernetes services to join the cluster. Generate one by running the following command against your Teleport Auth Service and save it in `/tmp/token` on the machine that will run Kubernetes Discovery: ``` $ tctl tokens add --type=discovery,kube --format=text abcd123-insecure-do-not-use-this ``` ### Configure the Teleport Kubernetes and Discovery Services On the host running the Kubernetes Service and Discovery Service, create a Teleport configuration file with the following content at `/etc/teleport.yaml`: --- WARNING Discovery Service exposes a configuration parameter - `discovery_service.discovery_group` - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services. When running multiple Discovery Services, you must ensure that each service is configured with the same `discovery_group` value if they are watching the same cloud resources or a different value if they are watching different cloud resources. It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration. - 2 Discovery Services configured with `discovery_group: "prod"` polling data from Production account. - 2 Discovery Services configured with `discovery_group: "staging"` polling data from Staging account. --- Enabling EKS Auto-Discovery requires that the `discovery_service.aws` section include at least one entry and that `discovery_service.aws.types` include `eks`. It also requires configuring the `kubernetes_service.resources.tags` to use the same labels configured at `discovery_service.aws.tags` or a subset of them to make the Kubernetes Service listen to the dynamic resources created by the Discovery Service. ``` version: v3 teleport: join_params: token_name: "/tmp/token" method: token proxy_server: "teleport.example.com:443" auth_service: enabled: false proxy_service: enabled: false ssh_service: enabled: false discovery_service: enabled: true discovery_group: "aws-prod" aws: - types: ["eks"] regions: ["*"] tags: "env": "prod" # Match EKS cluster tags where tag:env=prod kubernetes_service: enabled: true resources: - labels: "env": "prod" # Match Kubernetes Cluster labels specified earlier ``` ### Start the Kubernetes and Discovery Services Grant the Kubernetes and Discovery Services access to credentials that it can use to authenticate to AWS. - If you are running the Kubernetes and Discovery Services on an EC2 instance, you may use the EC2 Instance Metadata Service method - If you are running the Kubernetes and Discovery Services in Kubernetes, you can use IAM Roles for Service Accounts (IRSA) - Otherwise, you must use environment variables **Instance Metadata Service** Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials. The EC2 instance should be configured to use an EC2 instance profile. For more information, see: [Using Instance Profiles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html). **Kubernetes IRSA** Refer to [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role. **Environment Variables** Teleport's built-in AWS client reads credentials from the following environment variables: - `AWS_ACCESS_KEY_ID` - `AWS_SECRET_ACCESS_KEY` - `AWS_DEFAULT_REGION` When you start the Kubernetes and Discovery Services, the service reads environment variables from a file at the path `/etc/default/teleport`. Obtain these credentials from your organization. Ensure that `/etc/default/teleport` has the following content, replacing the values of each variable: ``` AWS_ACCESS_KEY_ID=00000000000000000000 AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000 AWS_DEFAULT_REGION= ``` Have multiple sources of AWS credentials? Teleport's AWS client loads credentials from different sources in the following order: - Environment Variables - Shared credentials file - Shared configuration file (Teleport always enables shared configuration) - EC2 Instance Metadata (credentials only) While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Kubernetes and Discovery Services with the `AWS_PROFILE` environment variable assigned to the name of your profile of choice. If you have a specific use case that the instructions above do not account for, consult the documentation for the [AWS SDK for Go](https://docs.aws.amazon.com/sdk-for-go/api/aws/session/) for a detailed description of credential loading behavior. Configure the Kubernetes and Discovery Services to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Kubernetes and Discovery Services. **Package Manager** On the host where you will run the Kubernetes and Discovery Services, enable and start Teleport: ``` $ sudo systemctl enable teleport $ sudo systemctl start teleport ``` **TAR Archive** On the host where you will run the Kubernetes and Discovery Services, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport: ``` $ sudo teleport install systemd -o /etc/systemd/system/teleport.service $ sudo systemctl enable teleport $ sudo systemctl start teleport ``` You can check the status of the Kubernetes and Discovery Services with `systemctl status teleport` and view its logs with `journalctl -fu teleport`. Once the Kubernetes and Discovery Services start, EKS clusters matching the tags and regions specified in the AWS section will be added to the Teleport cluster automatically. ## Troubleshooting ### Discovery Service troubleshooting First, check if any Kubernetes clusters have been discovered. To do this, you can use the `tctl get kube_cluster` command and check if the expected Kubernetes clusters have already been registered with your Teleport cluster. If some Kubernetes clusters do not appear in the list, check if the Discovery Service selector labels match the missing Kubernetes cluster tags or look into the Discovery Service logs for permission errors. Check that the Discovery Service is running with credentials for the correct AWS account. It can discover resources in another AWS account, but it must be configured to assume a role in the other AWS account if that's the case. Check if there is more than one Discovery Services running: ``` $ tctl inventory status --connected ``` If you are running multiple Discovery Services, you must ensure that each service is configured with the same `discovery_group` value if they are watching the same cloud Kubernetes clusters or a different value if they are watching different cloud Kubernetes clusters. If this is not configured correctly, a typical symptom is `kube_cluster` resources being intermittently deleted from your Teleport cluster's registry. ### Kubernetes Service troubleshooting If the `tctl get kube_cluster` command returns the discovered clusters, but the `tctl kube ls` command does not include them, check that you have set the `kubernetes_service.resources` section correctly. ``` kubernetes_service: enabled: true resources: - labels: "env": "prod" ``` If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport.