# Using Teleport with OpenSSH in agentless mode (manual installation) In this guide, we will show you how to configure the OpenSSH server `sshd` to join a Teleport cluster. Existing fleets of OpenSSH servers can be configured to accept SSH certificates dynamically issued by a Teleport CA. Using Teleport and OpenSSH has the advantage of getting you up and running, but in the long run, we would recommend replacing `sshd` with `teleport`. `teleport` SSH servers have support for multiple features that are incompatible with OpenSSH: - RBAC and resource filtering based on [dynamically updated labels](https://goteleport.com/docs/zero-trust-access/rbac-get-started/labels.md) - [Session recording without SSH connection termination](https://goteleport.com/docs/reference/architecture/session-recording.md#record-at-the-proxy-service) - [Session sharing](https://goteleport.com/docs/connect-your-client/teleport-clients/tsh.md) - [Advanced session recording](https://goteleport.com/docs/enroll-resources/server-access/guides/bpf-session-recording.md) ## How it works At a high level, Teleport supports OpenSSH servers by proxying SSH connections through the Proxy Service. The OpenSSH server will be set up to trust connections from the Proxy Service rather than trusting direct connections from users. This ensures that all connections to OpenSSH servers go through the Proxy Service, where session IO and audit events can be recorded and RBAC can be enforced. For deeper details as to how this works, you may be interested in the [Registered OpenSSH Nodes RFD](https://github.com/gravitational/teleport/blob/master/rfd/0098-registered-openssh-nodes.md) Below are some key details from the RFD: - The Proxy Service is uniquely able to request certificates signed by the Auth Service with the Teleport OpenSSH CA on behalf of users, and the OpenSSH server is set up to only trust certificates signed by this CA. Therefore, in order to connect to an OpenSSH server, a user must connect through the Proxy to request and use an OpenSSH certificate. - Before connecting the user to the OpenSSH server, the Proxy Service performs RBAC checks to see if the user should be allowed to access it. - In order to record the session and its audit events, the Proxy Service terminates (decrypts) the client SSH connection, and establishes its own connection to the OpenSSH server. It then acts as a pipe between the two connections, recording events and IO as it proceeds. - See [Recording Proxy Mode](https://goteleport.com/docs/reference/architecture/session-recording.md#record-at-the-proxy-service) for more details. --- NOTE This guide shows you how to register an OpenSSH node by creating a node resource and configuring OpenSSH to trust the Teleport CA. If you can copy the `teleport` binary onto your OpenSSH node and execute it however, you can follow the [standard registration guide](https://goteleport.com/docs/enroll-resources/server-access/openssh/openssh-agentless.md) instead, which has fewer steps. Teleport is able to perform many of the steps we show in this guide automatically. --- ## Prerequisites - OpenSSH version 6.9 or above on your local machine. View your OpenSSH version with the command: ``` $ ssh -V ``` * A running Teleport cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/get-started/deploy-community.md). * The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/installation.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` - A Linux host with the OpenSSH server `sshd` version 7.4 or above installed, but not Teleport. The SSH port on this host must be open to traffic from the Teleport Proxy Service host. - To check that you can connect to your Teleport cluster, sign in with `tsh login`, then verify that you can run `tctl` commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email\@example.com to your Teleport username: ``` $ tsh login --proxy=teleport.example.com --user=email@example.com $ tctl status Cluster teleport.example.com Version 18.7.3 CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 ``` If you can connect to the cluster and run the `tctl status` command, you can use your current credentials to run subsequent `tctl` commands from your workstation. If you host your own Teleport cluster, you can also run `tctl` commands on the computer that hosts the Teleport Auth Service for full permissions. ## Step 1/5. Add a node resource to your Teleport cluster When you request an SSH connection to a OpenSSH node, Teleport needs to be able to find the node's IP address so it can establish a connection to it. Declare a `node` resource so Teleport knows how to reach your OpenSSH server. On your local machine, create a file called `openssh-node-resource.yaml` with the following content: ``` kind: node version: v2 sub_kind: openssh metadata: name: a100fdd0-52db-4eca-a7ab-c3afa7a1564a labels: env: prod spec: addr: 1.2.3.4:22 hostname: openssh-node ``` `spec.addr` and `spec.hostname` are mandatory. Assign `spec.addr` to the address and port of your node and `spec.hostname` to the name of the node as you would like users to see it in Teleport. The `metadata.labels` field labels the SSH Service instance so you can apply RBAC rules to it. The `metadata.name` field isn't mandatory, but if supplied, it must be a universal unique identifier (UUID). To generate a new UUID suitable for a `node` name, use the `uuidgen` command on Linux or macOS, or use the `New-Guid` cmdlet in Powershell on Windows. Create the node resource: ``` $ tctl create openssh-node-resource.yaml ``` --- NOTE This step can be done with Infrastructure-as-Code (IaC) tools (tctl, Terraform, or Kubernetes Operator). This is described [in the OpenSSH server IaC guide](https://goteleport.com/docs/zero-trust-access/infrastructure-as-code/managing-resources/agentless-ssh-servers.md). --- ## Step 2/5. Configure `sshd` to trust the Teleport CA Later in this guide, we will generate an SSH client configuration that will use a certificate signed by the Teleport Auth Service to authenticate to your SSH server. For this to work, `sshd` must be told to allow users to log in with certificates generated by the Teleport Auth Service. Start by exporting the Teleport CA public key. On the host where you are running `sshd`, run the following commands, assigning proxy to the address of your Teleport Proxy Service: ``` $ export KEY=$(curl 'https://proxy/webapi/auth/export?type=openssh' | sed "s/cert-authority\ //") ``` Make the public key accessible to `sshd`: ``` $ sudo bash -c "echo \"$KEY\" > /etc/ssh/teleport_openssh_ca.pub" $ sudo bash -c "echo 'TrustedUserCAKeys /etc/ssh/teleport_openssh_ca.pub' >> /etc/ssh/sshd_config" ``` Restart `sshd`. For systemd-enabled hosts, run the following command: ``` $ sudo systemctl restart sshd ``` Now, `sshd` will trust users who present a Teleport-issued certificate. ## Step 3/5. Configure host authentication Next, ask Teleport to issue a valid host certificate for your `sshd` host. Later in this guide, we will configure your SSH client to trust the certificate, authenticating your `sshd` host for your SSH client. Like the user certificate we created earlier, the host certificate will be signed by the Teleport Auth Service. ### Ensure that your user has the correct privileges Your user must be authorized to read and write host certificates. On your local machine, create a file called `host-certifier.yaml` with the following content: ``` kind: role version: v5 metadata: name: host-certifier spec: allow: rules: - resources: - host_cert verbs: - list - create - read - update - delete ``` Create the role resource: ``` $ tctl create host-certifier.yaml role 'host-certifier' has been created ``` --- TIP You can also create and edit roles using the Web UI. Go to **Access -> Roles** and click **Create New Role** or pick an existing role to edit. --- Assign the `host-certifier` role to your Teleport user by running the appropriate commands for your authentication provider: **Local User** 1. Retrieve your local user's roles as a comma-separated list: ``` $ ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")') ``` 2. Edit your local user to add the new role: ``` $ tctl users update $(tsh status -f json | jq -r '.active.username') \ --set-roles "${ROLES?},host-certifier" ``` 3. Sign out of the Teleport cluster and sign in again to assume the new role. **GitHub** 1. Open your `github` authentication connector in a text editor: ``` $ tctl edit github/github ``` 2. Edit the `github` connector, adding `host-certifier` to the `teams_to_roles` section. The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization. Here is an example: ``` teams_to_roles: - organization: octocats team: admins roles: - access + - host-certifier ``` 3. Apply your changes by saving and closing the file in your editor. 4. Sign out of the Teleport cluster and sign in again to assume the new role. **SAML** 1. Retrieve your `saml` configuration resource: ``` $ tctl get --with-secrets saml/mysaml > saml.yaml ``` Note that the `--with-secrets` flag adds the value of `spec.signing_key_pair.private_key` to the `saml.yaml` file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource. 2. Edit `saml.yaml`, adding `host-certifier` to the `attributes_to_roles` section. The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization. Here is an example: ``` attributes_to_roles: - name: "groups" value: "my-group" roles: - access + - host-certifier ``` 3. Apply your changes: ``` $ tctl create -f saml.yaml ``` 4. Sign out of the Teleport cluster and sign in again to assume the new role. **OIDC** 1. Retrieve your `oidc` configuration resource: ``` $ tctl get oidc/myoidc --with-secrets > oidc.yaml ``` Note that the `--with-secrets` flag adds the value of `spec.signing_key_pair.private_key` to the `oidc.yaml` file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource. 2. Edit `oidc.yaml`, adding `host-certifier` to the `claims_to_roles` section. The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization. Here is an example: ``` claims_to_roles: - name: "groups" value: "my-group" roles: - access + - host-certifier ``` 3. Apply your changes: ``` $ tctl create -f oidc.yaml ``` 4. Sign out of the Teleport cluster and sign in again to assume the new role. You will now have the required permissions to export a host key for your `sshd` host. ### Issue a host certificate Find your node's UUID When you created a `node` resource and if you didn't set the `metadata.name` field earlier, the Teleport Auth Service generated a universal unique identifier (UUID) for that node. Teleport Proxy Services uses the UUID to differentiate nodes with the same hostname, so it must be added to the host certificate. To find your node's UUID, first determine if its hostname is unique: ``` $ tctl get node/openssh-node --format text ``` If only one node is displayed and you have `jq` installed, you can run the following command to get your node's UUID: ``` $ tctl get node/openssh-node --format=json | jq -r ".[0].metadata.name" ``` Otherwise, find your node's UUID in the `metadata.name` field of the YAML output of this command: ``` $ tctl get node/openssh-node ``` #### Create the host certificate When creating host certificates, it is important to specify all the domain names and addresses that refer to your node. If you try to connect to a node with a name or address that was not specified when creating its host certificate, Teleport will reject the SSH connection. On your local machine, assign the IP address, fully qualified domain name of your node, and the node's UUID to an environment variable. If you won't be connecting to your node with its hostname, you can safely omit it. ``` $ ADDR=1.2.3.4,openssh-node,a100fdd0-52db-4eca-a7ab-c3afa7a1564a ``` Run the following `tctl` command to generate a host certificate: ``` $ tctl auth sign \ --host=${ADDR?} \ --format=openssh \ --out=myhost The credentials have been written to myhost, myhost-cert.pub ``` The above command will result in a private key and certificate. Issuing certificates for multiple hosts To generate certificates for multiple hosts, assign the `host` flag to a comma-separated list of addresses. Certificates for wildcard domains are not supported by OpenSSH, so each domain must be fully qualified. Use `ssh-keygen` to verify the contents of the certificate: ``` $ ssh-keygen -L -f myhost-cert.pub ``` The `Principals` section should contain the address you assigned to `ADDR` earlier: ``` myhost-cert.pub: Type: ssh-rsa-cert-v01@openssh.com host certificate Public key: RSA-CERT SHA256:nHkp6SnrAW4AV00VUaqPgR6SgdyvV9MmjUrYnwZ779A Signing CA: RSA SHA256:euqx2Y8Pq+r0c94GKVNXAklBVTmAJtaQUn3/ehrfEJE (using rsa-sha2-512) Key ID: "" Serial: 0 Valid: after 2022-04-22T11:14:16 Principals: 1.2.3.4 openssh-node a100fdd0-52db-4eca-a7ab-c3afa7a1564a Critical Options: (none) Extensions: x-teleport-authority UNKNOWN OPTION (len 33) x-teleport-role UNKNOWN OPTION (len 8) ``` Copy the host key and certificate to your `sshd` host, placing them in the directory `/etc/ssh`. Make sure these files have the correct permissions: ``` $ sudo chmod 0600 /etc/ssh/myhost $ sudo chmod 0600 /etc/ssh/myhost-cert.pub ``` Then add the following lines to `/etc/ssh/sshd_config` on your `sshd` host: ``` HostKey /etc/ssh/myhost HostCertificate /etc/ssh/myhost-cert.pub ``` Restart `sshd`. ## Step 4/5. Generate an SSH client configuration The next step is to configure your OpenSSH client to connect to your `sshd` host using credentials managed by Teleport. This configuration will use your user's Teleport-issued certificate to authenticate to the `sshd` host. It will also authenticate the `sshd` host using the host certificate you generated earlier. First, make sure you have logged in to your Teleport cluster: **Teleport Community Edition** ``` $ tsh status > Profile URL: https://teleport.example.com:443 Logged in as: myuser Cluster: teleport.example.com Roles: access, auditor, editor, host-certifier Logins: ubuntu, root Kubernetes: enabled Valid until: 2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s] Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty ``` **Teleport Enterprise** ``` $ tsh status > Profile URL: https://teleport.example.com:443 Logged in as: myuser Cluster: teleport.example.com Roles: access, auditor, editor, reviewer, host-certifier Logins: ubuntu, root Kubernetes: enabled Valid until: 2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s] Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty ``` **Teleport Enterprise Cloud** ``` $ tsh status > Profile URL: https://mytenant.teleport.sh:443 Logged in as: myuser Cluster: mytenant.teleport.sh Roles: access, auditor, editor, reviewer, host-certifier Logins: ubuntu, root Kubernetes: enabled Valid until: 2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s] Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty ``` On your local machine, run the following `tsh` command. This will print a configuration block that tells your SSH client to use credentials managed by Teleport to connect to hosts in your cluster. ``` $ tsh config > ssh_config_teleport ``` This command creates an SSH configuration file at a nonstandard location in order to make it easier to clean up, but you can append the output of `tsh config` to the default SSH config file (`~/.ssh/config`) if you wish. How does the config work? Teleport implements an SSH server that includes several **subsystems**, or predefined commands that are run when the server handles a connection. The Proxy Service implements a `proxy` subsystem that forwards SSH traffic to remote hosts and trusted clusters. Here is a brief explanation of the configuration that `tsh config` generates: ``` # Common flags for all {{ .ClusterName }} hosts Host *.{{ .ClusterName }} {{ .ProxyHost }} UserKnownHostsFile "{{ .KnownHostsPath }}" IdentityFile "{{ .IdentityFilePath }}" CertificateFile "{{ .CertificateFilePath }}" ``` If the host you are `ssh`ing into belongs to your Teleport cluster (i.e., its address is a subdomain of your cluster's domain), use a Teleport-managed known hosts file, private key, and certificate that are stored in the `.tsh` directory. ``` # Flags for all {{ .ClusterName }} hosts except the proxy Host *.{{ .ClusterName }} !{{ .ProxyHost }} Port 3022 ProxyCommand "{{ .TSHPath }}" proxy ssh --cluster={{ .ClusterName }} --proxy={{ .ProxyHost }} %r@%h:%p ``` If the host that you are `ssh`ing into belongs to your Teleport cluster, the OpenSSH client will first execute a command, the `ProxyCommand`, that establishes an SSH connection to the Proxy Service. This command, `tsh proxy ssh`, requests the `proxy` subsystem in order to forward SSH traffic through the Proxy Service to your chosen host (including a host in a Trusted Cluster). The `tsh proxy ssh` command requests the `proxy` subsystem through a command similar to the following, which assumes you are logging in to a node called `mynode` as `root` with a cluster called `teleport.example.com`: ``` $ /usr/bin/ssh -l root -A -o UserKnownHostsFile=/root/.tsh/known_hosts -p 11105 teleport.example.com -s proxy:mynode:3022@teleport.example.com ``` Notice that the `known_hosts` file used by the command is managed by `tsh`. Since the `sshd` host's information is listed in this file, your SSH client can authenticate the host via the certificate we generated earlier. Using PowerShell on Windows? If using PowerShell on Windows, note that normal shell redirection may write the file with the incorrect encoding. To ensure it's written properly, try the following: ``` $ tsh.exe config | out-file .ssh\config -encoding utf8 -append ``` Details Dialing uppercase hostnames with OpenSSH Routing in Teleport clusters is case-sensitive by default, but OpenSSH always lowercases hostnames. If you are using an OpenSSH client and have hosts with uppercase letters in their hostnames, you may need to set `case_insensitive_routing: true` in either the `auth_service` block of your Teleport config, or in the `cluster_networking_config` resource. --- MULTIPLE CLUSTERS If you switch between multiple Teleport Proxy Servers, you'll need to re-run `tsh config` for each to generate the cluster-specific configuration. Similarly, if trusted clusters are added or removed, be sure to re-run `tsh config` and replace the previous configuration. --- ## Step 5/5. Connect to your `sshd` host Once you have appended the new text to your OpenSSH client configuration file, you can log in to your `sshd` host using the configuration we generated earlier. First, define environment variables for the address of your Teleport cluster, the username you will use to log in to your `sshd` host, and the port on your `sshd` host you are using for SSH traffic: **Teleport Community Edition/Enterprise** ``` See the available logins you can use to access your sshd host $ tsh status | grep Logins Logins: ubuntu, root $ USER=ubuntu $ CLUSTER=teleport.example.com $ PORT=22 ``` **Teleport Enterprise Cloud** ``` See the available logins you can use to access your sshd host $ tsh status | grep Logins Logins: ubuntu, root $ USER=ubuntu $ CLUSTER=mytenant.teleport.sh $ PORT=22 ``` Next, SSH in to your remote host: ``` $ ADDR_NODE=openssh-node $ ssh -p ${PORT?} -F ssh_config_teleport "${USER?}@${ADDR_NODE?}.${CLUSTER?}" ``` This name does not need to be resolvable via DNS as the connection will be routed through your Teleport Proxy Service. Why are we overriding the port here? By default, the OpenSSH client configuration generated by `tsh config` directs the Teleport Proxy Service to dial port 3022 of a node in your Teleport cluster. This works if the node's SSH Service is listening on port 3022, and means that you can connect to the Teleport SSH Service via your OpenSSH client. When you join a Teleport node to a cluster, the node creates a reverse tunnel to the cluster's Proxy Service. When you run an `ssh` command to access a host in your Teleport cluster using the configuration we generated, the Teleport Proxy Service will attempt to connect to the host via this reverse tunnel and, if that fails, try directly dialing the address. In our case, the `sshd` host is not running Teleport, so no reverse tunnel will exist. Instead, the Proxy Service will establish a direct connection on the host's SSH port. Using trusted clusters? You can log in to a host in a trusted leaf cluster by placing the name of the cluster between the name of the node and the name of your root cluster: ``` $ ssh -F ssh_config_teleport ${USER?}@node2.leafcluster.${CLUSTER} ``` --- NOTE Teleport uses OpenSSH certificates instead of keys. When you connect to a remote host, OpenSSH verifies that the address of the host is listed under the `Principals` section of the OpenSSH certificate. Usually, this is a fully qualified domain name, rather than an IP address. --- ### OpenSSH rate limiting Be aware of OpenSSH's built-in rate-limiting. On large numbers of Proxy Service connections, you may encounter errors like: ``` channel 0: open failed: connect failed: ssh: handshake failed: EOF ``` See the `MaxStartups` setting in `man sshd_config`. This setting means that by default, OpenSSH only allows 10 unauthenticated connections at a time and starts dropping connections 30% of the time when the number of connections goes over 10. When it hits 100 authentication connections, all new connections are dropped. To increase the concurrency level, increase the value to something like `MaxStartups 50:30:100`. This allows 50 concurrent connections and a max of 100.