# Automatically Discover Azure Virtual Machines This guide shows you how to set up automatic server discovery for Azure virtual machines. ## How it works The Teleport Discovery Service can connect to Azure and automatically discover and enroll virtual machines matching configured labels. It will then execute a script on these discovered instances that will install Teleport, start it and join the cluster. ## Prerequisites - A running Teleport cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/get-started/deploy-community.md). - The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/installation.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` * Azure subscription with virtual machines and permissions to create and attach managed identities. * Azure virtual machines to join the Teleport cluster, running Ubuntu/Debian/RHEL if making use of the default Teleport install script. (For other Linux distributions, you can install Teleport manually.) * To check that you can connect to your Teleport cluster, sign in with `tsh login`, then verify that you can run `tctl` commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email\@example.com to your Teleport username: ``` $ tsh login --proxy=teleport.example.com --user=email@example.com $ tctl status Cluster teleport.example.com Version 18.7.3 CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 ``` If you can connect to the cluster and run the `tctl status` command, you can use your current credentials to run subsequent `tctl` commands from your workstation. If you host your own Teleport cluster, you can also run `tctl` commands on the computer that hosts the Teleport Auth Service for full permissions. ## Step 1/5. Create an Azure invite token When discovering Azure virtual machines, Teleport makes use of Azure invite tokens for authenticating joining SSH Service instances. Create a file called `token.yaml`: ``` # token.yaml kind: token version: v2 metadata: # the token name is not a secret because instances must prove that they are # running in your Azure subscription to use this token name: azure-discovery-token # set a long expiry time, as the default for tokens is only 30 minutes expires: "3000-01-01T00:00:00Z" spec: # use the minimal set of roles required roles: [Node] # set the join method allowed for this token join_method: azure azure: allow: # specify the Azure subscription which Nodes may join from - subscription: "123456789" ``` Assign the `subscription` field to your Azure subscription ID. Add the token to the Teleport cluster with: ``` $ tctl create -f token.yaml ``` ## Step 2/5. Configure IAM permissions for Teleport The Teleport Discovery Service needs Azure IAM permissions to discover and register Azure virtual machines. ### Configure an Azure service principal There are a couple of ways for the Teleport Discovery Service to access Azure resources: - The Discovery Service can run on an Azure VM with attached managed identity. This is the recommended way of deploying the Discovery Service in production since it eliminates the need to manage Azure credentials. - The Discovery Service can be registered as a Microsoft Entra ID application and configured with its credentials. This is only recommended for development and testing purposes since it requires Azure credentials to be present in the Discovery Service's environment. **Using managed identity** Go to the [Managed Identities](https://portal.azure.com/#browse/microsoft.managedidentity%2fuserassignedidentities) page in your Azure portal and click *Create* to create a new user-assigned managed identity: ![Managed identities](/docs/assets/images/managed-identities@2x-db5917e6ecdac0bed024d29bb9aa963d.png) Pick a name and resource group for the new identity and create it: ![New identity](/docs/assets/images/new-identity@2x-55c4ee99a8f28282af2082a1fee7659c.png) Take note of the created identity's *Client ID*: ![Created identity](/docs/assets/images/created-identity@2x-04036ba4bf8d3b7edc3de084aaff33ed.png) Next, navigate to the Azure VM that will run your Discovery Service instance and add the identity you've just created to it: ![VM identity](/docs/assets/images/vm-identity@2x-c8cd8f72b017c1a8e74a7dd1b45c8fc1.png) Attach this identity to all Azure VMs that will be running the Discovery Service. **Using app registrations** --- NOTE Registering the Discovery Service as a Microsoft Entra ID application is suitable for test and development scenarios, or if your Discovery Service does not run on an Azure VM. For production scenarios prefer to use the managed identity approach. --- Go to the [App registrations](https://portal.azure.com/#blade/microsoft_aad_iam/activedirectorymenublade/registeredapps) page in Microsoft Entra ID and click on *New registration*: ![App registrations](/docs/assets/images/app-registrations@2x-a69dc84c4caccb1152df850ea6590a1e.png) Pick a name (e.g. *DiscoveryService*) and register a new application. Once the app has been created, take note of its *Application (client) ID* and click on *Add a certificate or secret*: ![Registered app](/docs/assets/images/registered-app@2x-9103c8e2e953e62007b399a68cacc0cb.png) Create a new client secret that the Discovery Service agent will use to authenticate with the Azure API: ![Registered app secrets](/docs/assets/images/registered-app-secrets@2x-3e754c49f06eeaf6c14af3b9d5067e67.png) The Teleport Discovery Service uses Azure SDK's default credential provider chain to look for credentials. Refer to [Azure SDK Authorization](https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization) to pick a method suitable for your use-case. For example, to use environment-based authentication with a client secret, the Discovery Service should have the following environment variables set: ``` export AZURE_TENANT_ID= export AZURE_CLIENT_ID= export AZURE_CLIENT_SECRET= ``` ### Create a custom role Teleport requires the following permissions to discover and enroll Azure VMs: - `Microsoft.Compute/virtualMachines/read` - `Microsoft.Compute/virtualMachines/runCommand/action` - `Microsoft.Compute/virtualMachines/runCommands/write` - `Microsoft.Compute/virtualMachines/runCommands/read` - `Microsoft.Compute/virtualMachines/runCommands/delete` Here is a sample role definition allowing Teleport to read and run commands on Azure virtual machines: ``` { "properties": { "roleName": "TeleportDiscovery", "description": "Allows Teleport to discover Azure virtual machines", "assignableScopes": [ "/subscriptions/11111111-2222-3333-4444-555555555555" ], "permissions": [ { "actions": [ "Microsoft.Compute/virtualMachines/read", "Microsoft.Compute/virtualMachines/runCommand/action", "Microsoft.Compute/virtualMachines/runCommands/write", "Microsoft.Compute/virtualMachines/runCommands/read", "Microsoft.Compute/virtualMachines/runCommands/delete" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] } } ``` The `assignableScopes` field above includes a subscription `/subscriptions/`, allowing the role to be assigned at any resource scope within that subscription or the subscription scope itself. If you want to further limit the `assignableScopes`, you can use a resource group `/subscriptions//resourceGroups/` or a management group `/providers/Microsoft.Management/managementGroups/` instead. Now go to the [Subscriptions](https://portal.azure.com/#view/microsoft_azure_billing/subscriptionsblade) page and select a subscription. Click on *Access control (IAM)* in the subscription and select *Add > Add custom role*: ![IAM custom role](/docs/assets/images/add-custom-role@2x-9b486170e269494f0a84ad7af362a346.png) In the custom role creation page, click the *JSON* tab and click *Edit*, then paste the JSON example and replace the subscription in `assignableScopes` with your own subscription id: ![Create JSON role](/docs/assets/images/vm-create-role-from-json@2x-ac6a49b61728b702f69eb3fe67d3811c.png) ### Create a role assignment for the Teleport Discovery Service principal To grant Teleport permissions, the custom role you created must be assigned to the Teleport service principal - either the managed identity or the app registration you created earlier. Navigate to the resource scope where you want to make the role assignment. Click *Access control (IAM)* and select *Add > Add role assignment*. Choose the custom role you created as the role and the Teleport service principal as a member. ![Assign role](/docs/assets/images/create-role-assignment@2x-ca2f5091cc62b65d7d73249813e5c2e3.png) --- AZURE ROLE ASSIGNMENTS The role assignment should be at a high enough scope to allow the Teleport Discovery Service to discover all matching virtual machines. See [Identify the needed scope](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-steps#step-3-identify-the-needed-scope) for more information about Azure scopes and creating role assignments. --- ## Step 3/5. Set up an identity for discovered nodes Every Azure VM to be discovered must have an identity assigned to it: either system assigned or user assigned managed identity. **Azure Portal** To set up a Managed Identity: 1. Navigate to [Virtual machines view](https://portal.azure.com/#view/hubsextension/browseresource/resourcetype/microsoft.compute%2fvirtualmachines) if you're hosting Teleport on an Azure VM, or navigate to [Virtual machine scale sets view](https://portal.azure.com/#view/hubsextension/browseresource/resourcetype/microsoft.compute%2fvirtualmachinescalesets) if you're hosting Teleport on an Azure VMSS. 2. Select the VM or VMSS hosting your Teleport Service. 3. In the right-side panel, click the **Security/Identity** tab. 4. Under the **Identity** section, select the **System assigned** tab. 5. Toggle the **Status** switch to **On**. 6. Click **Save**. If you're using VMSS and it is configured with manual upgrade mode, you must update the VM instances for the identity changes to take effect: - Click the **Instances** tab in the right panel. - Select the VM instances to update. - Click **Restart**. **Azure CLI** To attach a system-assigned identity to a regular VM, run: ``` $ az vm identity assign --resource-group --name ``` To attach a system-assigned identity to an Azure VMSS, run: ``` $ az vmss identity assign --resource-group --name ``` If you're using VMSS and it is configured with manual upgrade mode, you must update the VM instances for the identity changes to take effect. Run the following command to propagate the identity change: ``` $ az vmss update-instances --resource-group --name --instance-ids * ``` If the VMs to be discovered have no system-managed identity and more than one user-managed identity assigned to them, copy the client ID of one of your user-managed identities for Step 5. ## Step 4/5. Install the Teleport Discovery Service --- TIP If you plan on running the Discovery Service on a host that is already running another Teleport service (Auth or Proxy, for example), you can skip this step. --- Install Teleport on the virtual machine that will run the Discovery Service: To install a Teleport Agent on your Linux server: The recommended installation method is the cluster install script. It will select the correct version, edition, and installation mode for your cluster. 1. Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https\://). 2. Run your cluster's install script: ``` $ curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash ``` ## Step 5/5. Configure Teleport to discover Azure instances If you are running the Discovery Service on its own host, the service requires a valid invite token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service: ``` $ tctl tokens add --type=discovery ``` Save the generated token in `/tmp/token` on the virtual machine that will run the Discovery Service. --- WARNING Discovery Service exposes a configuration parameter - `discovery_service.discovery_group` - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services. When running multiple Discovery Services, you must ensure that each service is configured with the same `discovery_group` value if they are watching the same cloud resources or a different value if they are watching different cloud resources. It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration. - 2 Discovery Services configured with `discovery_group: "prod"` polling data from Production account. - 2 Discovery Services configured with `discovery_group: "staging"` polling data from Staging account. --- Assign teleport.example.com:443 to the host and port of the Teleport Proxy Service in your cluster, and azure-prod to a name that identifies a group of resources that you will enroll: ``` # teleport.yaml version: v3 teleport: join_params: token_name: "/tmp/token" method: token proxy_server: "teleport.example.com:443" auth_service: enabled: false proxy_service: enabled: false ssh_service: enabled: false discovery_service: enabled: true discovery_group: azure-prod ``` Create a matcher for the resources you want to enroll. --- TIP Dynamic configuration uses Discovery Configs which can be managed using Terraform. See the [Terraform `discovery_config` reference](https://goteleport.com/docs/reference/infrastructure-as-code/terraform-provider/resources/discovery_config.md) for more information. Static configuration while simpler at first, has less flexibility because enrollment changes require edits to `teleport.yaml` and the restart of the Discovery Service. --- **Dynamic configuration (recommended)** Create a Discovery Config resource, that has the same discovery group you configured earlier, to enable Azure VM discovery. Create a file named `discovery-azure-prod.yaml` with the following content: ``` kind: discovery_config version: v1 metadata: name: example-discovery-config spec: discovery_group: azure-prod azure: - types: ["vm"] subscriptions: [""] resource_groups: [""] regions: [""] tags: "env": "prod" # Match virtual machines where tag:env=prod install: azure: # Optional: If the VMs to discover have more than one managed # identity assigned to them, set the client ID here to the client # ID of the identity created in step 3. client_id: "" ``` Adjust the keys under `spec.azure` to match your Azure environment, specifically the resource groups, regions and tags you want to associate with the Discovery Service. Create the Discovery Config by running the following command: ``` $ tctl create -f discovery-azure-prod.yaml ``` Matching instances will be added to the Teleport cluster automatically. You can update the Discovery Config at any time, and the service will automatically re-apply the changes. **Static configuration** In order to enable Azure VM discovery the `discovery_service.azure` section of `teleport.yaml` must include at least one entry: ``` # teleport.yaml # ... discovery_service: enabled: true discovery_group: azure-prod azure: - types: ["vm"] subscriptions: [""] resource_groups: [""] regions: [""] tags: "env": "prod" # Match virtual machines where tag:env=prod install: azure: # Optional: If the VMs to discover have more than one managed # identity assigned to them, set the client ID here to the client # ID of the identity created in step 3. client_id: "" ``` Adjust the keys under `discovery_service.azure` to match your Azure environment, specifically the regions and tags you want to associate with the Discovery Service. ## Auto-discovery labels Teleport applies a set of default labels to resources on AWS, Azure, and Google Cloud that join a cluster via auto-discovery. See the auto-discovery labels [reference](https://goteleport.com/docs/enroll-resources/auto-discovery/reference/labels.md) ## Advanced configuration This section covers configuration options for discovering and enrolling servers. ### Install multiple Teleport agents on the same instance When using blue-green deployments or other multiple clusters setups, you might want to access your instances from different clusters. Teleport supports installing and running multiple agents on the same instance, using a suffixed installation which allows you to isolate each installation. **Dynamic configuration (recommended)** To configure the Discovery Service to use a suffixed installation, edit the Discovery Config and set the `spec.azure.install.suffix` key: ``` kind: discovery_config # ... spec: azure: - install: suffix: "blue-cluster" ``` **Static configuration** To configure the Discovery Service to use a suffixed installation, specify the `install.suffix` key in your Discovery Service configuration: ``` # teleport.yaml version: v3 # ... discovery_service: enabled: true azure: - install: suffix: "blue-cluster" ``` Requires [agent managed updates](https://goteleport.com/docs/upgrading/agent-managed-updates.md) to be enabled. ### Define the group for Managed Updates If you are using Teleport [Agent managed updates](https://goteleport.com/docs/upgrading/agent-managed-updates.md), you can configure the update group so that you can control which instances get updated together. **Dynamic configuration (recommended)** To set the update group, edit the Discovery Config and set the `spec.azure.install.update_group` key: ``` kind: discovery_config # ... spec: azure: - install: update_group: "update-group-1" ``` **Static configuration** To set the update group, specify the `install.update_group` key in your Discovery Service configuration: ``` # teleport.yaml version: v3 # ... discovery_service: enabled: true azure: - install: update_group: "update-group-1" ``` ### Configure HTTP Proxy during installation For instances which require a proxy to access the installation files, you can configure HTTP Proxy settings in the Discovery Service. **Dynamic configuration (recommended)** To set the HTTP proxy settings, edit the Discovery Config and set the `spec.azure.install.http_proxy_settings` key: ``` kind: discovery_config # ... spec: azure: - install: http_proxy_settings: https_proxy: http://172.31.5.130:3128 http_proxy: http://172.31.5.130:3128 no_proxy: my-local-domain ``` **Static configuration** You must set the `install.http_proxy_settings` key in your configuration: ``` # teleport.yaml version: v3 # ... discovery_service: enabled: true azure: - install: http_proxy_settings: https_proxy: http://172.31.5.130:3128 http_proxy: http://172.31.5.130:3128 no_proxy: my-local-domain ``` ### Use a custom installation script To customize an installer, your user must have a role that allows `list`, `create`, `read` and `update` verbs on the `installer` resource. Create a file called `installer-manager.yaml` with the following content: ``` kind: role version: v5 metadata: name: installer-manager spec: allow: rules: - resources: [installer] verbs: [list, create, read, update] ``` Create the role: ``` $ tctl create -f installer-manager.yaml role 'installer-manager' has been created ``` --- TIP You can also create and edit roles using the Web UI. Go to **Access -> Roles** and click **Create New Role** or pick an existing role to edit. --- The preset `editor` role has the required permissions by default. To customize the default installer script, execute the following command on your workstation: ``` $ tctl edit installer/default-installer ``` After making the desired changes to the default installer, save and close the file in your text editor. Multiple `installer` resources can exist and be specified in the `azure.install.script_name` section: **Dynamic configuration (recommended)** Edit the Discovery Config to specify a custom installer script: ``` kind: discovery_config # ... spec: azure: - types: ["vm"] tags: - "env": "prod" install: # optional section when default-installer is used. script_name: "default-installer" - types: ["vm"] tags: - "env": "devel" install: script_name: "devel-installer" ``` **Static configuration** Edit the `teleport.yaml` configuration to specify a custom installer script: ``` discovery_service: # ... azure: - types: ["vm"] tags: - "env": "prod" install: # optional section when default-installer is used. script_name: "default-installer" - types: ["vm"] tags: - "env": "devel" install: script_name: "devel-installer" ``` --- The `installer` resource has the following templating options: - `{{ .MajorVersion }}`: the major version of Teleport to use when installing from the repository. - `{{ .PublicProxyAddr }}`: the public address of the Teleport Proxy Service to connect to. - `{{ .RepoChannel }}`: Optional package repository (apt/yum) channel name. Has format `/` e.g. stable/v18. See [installation](https://goteleport.com/docs/installation/linux.md) for more details. - `{{ .AutomaticUpgrades }}`: indicates whether Automatic Updates are enabled or disabled. Its value is either `true` or `false`. See [Automatic Agent Updates](https://goteleport.com/docs/upgrading/agent-managed-updates.md) for more information. - `{{ .TeleportPackage }}`: the Teleport package to use. Its value is either `teleport-ent` or `teleport` depending on whether the cluster is enterprise or not. These can be used as follows: ``` kind: installer metadata: name: default-installer spec: script: | echo {{ .PublicProxyAddr }} echo Teleport-{{ .MajorVersion }} echo Repository Channel: {{ .RepoChannel }} version: v1 ``` Which, when retrieved for installation, will evaluate to a script with the following contents: ``` echo teleport.example.com echo Teleport-18.7.3 echo Repository Channel: stable/v18.7.3 ``` The default installer will take the following actions: - Add an official Teleport repository to supported Linux distributions. - Install Teleport via `apt` or `yum`. - Generate the Teleport config file and write it to `/etc/teleport.yaml`. - Enable and start the Teleport service. If `client_id` is set in the Discovery Service config, custom installers will also have the `{{ .AzureClientID }}` templating option. ## Troubleshooting ### No credential providers error If you see the error `DefaultAzureCredential: failed to acquire a token.` in Discovery Service logs then Teleport is not detecting the required credentials to connect to the Azure SDK. Check whether the credentials have been applied in the machine running the Teleport Discovery Service and restart the Teleport Discovery Service. Refer to [Azure SDK Authorization](https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization) for more information. ### Teleport reports no error but VM does not join Check your Discovery Service config and make sure that the VM you want to discover matches. In debug mode, Teleport will log the Subscription IDs and names of VMs it discovers. The Azure run command API does not report the output of commands, so Teleport has no way of knowing if a command succeeded or failed. Run command logs can be found on the targeted VM at `/var/log/azure/run-command-handler/handler.log`. ## Next steps - Read [Joining Nodes via Azure Managed Identity](https://goteleport.com/docs/enroll-resources/agents/azure.md) for more information on Azure tokens. - Full documentation on Azure discovery configuration can be found through the [config file reference documentation](https://goteleport.com/docs/reference/deployment/config.md).