The goal of this article is to define a solution for the management of Azure Private Endpoints and Private DNS Zones in a complex hub-and-spoke network topology.
The objective of this article is to define a solution for the management of Azure Private Endpoints and Private DNS Zones in a complex hub-and-spoke network topology. In this context, application team users have limited permissions in the spoke virtual networks and read-only permissions on the resources in the hub virtual network. The final objective is to let individual application teams to create and scale Azure resources within their Azure subscription or landing zone without the need to interact with the platform team (SysOps, NetOps, and SecOps) via tickets and any manual or automated approval process. The platform team defines and implements the guard rails on subscriptions in which the application team can operate. Application team users operate autonomously within those guard rails in their subscriptions. Centrally managed components for implementing guard rails and shared services include Management Groups, Policies, Blueprints, Template Specs, and shared services like the Hub virtual network, Azure Firewall, and DNS server. The goal is to maximize the autonomy and agility of individual application teams by minimizing centrally managed processes and thus reduce the operational burden on central IT.
The conclusion of our investigation is that the centralized model for managing Private DNS Zones should be used as a reference approach, while the decentralized model should only be adopted when application teams have very strict requirements in terms of workload isolation, sandboxing, and security of private endpoints. The lifecycle of the DNS records required by private endpoints can be entirely managed via Azure Policy and PrivateDnsZoneGroup as explained in the second part of the present article. For more information on DNS configuration of private endpoints, see Azure Private Endpoint DNS configuration and in particular Virtual network and on-premises workloads using a DNS forwarder.
Contoso Ltd has started a journey to assess and migrate workloads to Azure with the goal of de-commissioning their on-premises data center within the upcoming years. Contoso is building a custom datacenter control plane platform for Azure to provide foundational guard rails, services, and infrastructure on Azure for the entire company. Contoso is employing a top-down, business aligned, application centric approach to application migrations and transformations. This approach will help product owners and application teams control their own transformation pace, agility, and autonomy when moving to Azure. The datacenter control plane adheres to the following characteristics and is aligned with the principles of the Cloud Adoption Framework:
The Contoso cloud environment is composed of hundreds of subscriptions organized in hierarchy of Management Groups by compliance requirements. Contoso built a custom datacenter control plane for Azure that implements infrastructure automation which includes CI/CD pipelines to deploy and manage Management Group hierarchies, Azure Policies, Azure Blueprints or Template Specs as well as company-wide shared services (i.e. central Firewalls, hub networks, cross-premises connectivity across multiple regions) as follows:
Azure Private Link is a networking technology that enables applications running on-premises or in the cloud to access a list of Azure services (for example, Storage Account or and SQL Database) and Azure hosted customer-owned/partner services over a private endpoint defined in a virtual network. When you're connecting to a private link resource using its fully qualified domain name (FQDN) as part of the connection string, it's important to correctly configure your DNS settings to resolve the FQDN to the allocated private IP address. For more information on this, see Azure Private Endpoint DNS configuration. Traffic between the virtual network and the service travels the Microsoft backbone network.
Azure Private Link provides the following benefits:
Privately access services on the Azure platform: Connect your virtual network to services in Azure without a public IP address at the source or destination. Service providers can render their services in their own virtual network and consumers can access those services in their local virtual network. The Private Link platform will handle the connectivity between the consumer and services over the Azure backbone network.
On-premises and peered networks: Access services running in Azure from on-premises over ExpressRoute private peering, VPN tunnels, and peered virtual networks using private endpoints. There's no need to set up public peering or traverse the internet to reach the service. Private Link provides a secure way to migrate workloads to Azure.
Protection against data leakage: A private endpoint is mapped to an instance of a PaaS resource instead of the entire service. Consumers can only connect to the specific resource. Access to any other resource in the service is blocked. This mechanism provides protection against data leakage risks.
Global reach: Connect privately to services running in other regions. The consumer's virtual network could be in region A and it can connect to services behind Private Link in region B.
Extend to your own services: Enable the same experience and functionality to render your service privately to consumers in Azure. By placing your service behind a standard Azure Load Balancer, you can enable it for Private Link. The consumer can then connect directly to your service using a private endpoint in their own virtual network. You can manage the connection requests using an approval call flow. Azure Private Link works for consumers and services belonging to different Azure Active Directory tenants.
For more information on Private Link and Private Endpoints, see the following resources:
Azure Private Endpoint is a network interface that connects applications privately and securely to an Azure service powered by Azure Private Link. Private Endpoint uses a private IP address from a virtual network, effectively bringing the service into your virtual network. The service could be an Azure service such as Azure Storage, Azure Cosmos DB, SQL, etc. or your own Private Link Service. Here are some key details about private endpoints:
Private endpoint enables connectivity between the consumers from the same VNet, regionally peered VNets, globally peered VNets and on premises using VPN or Express Route and services powered by Private Link.
Network connections can only be initiated by clients connecting to the Private endpoint, Service providers do not have any routing configuration to initiate connections into service consumers. Connections can only be established in a single direction.
As shown in the picture below, when creating a private endpoint, a read-only network interface is also created for the lifecycle of the resource. By convention, the name of this network interface is equal to <private-endpoint-name>.nic.<unique-identifier>. The interface is assigned dynamically a private IP address from the subnet that maps to the private link resource. The value of the private IP address remains unchanged for the entire lifecycle of the private endpoint.
The private endpoint must be deployed in the same region as the virtual network.
The private link resource can be deployed in a different region than the virtual network and private endpoint.
Multiple private endpoints can be created using the same private link resource. For a single network using a common DNS server configuration, the recommended practice is to use a single private endpoint for a given private link resource to avoid duplicate entries or conflicts in DNS resolution.
Multiple private endpoints can be created on the same or different subnets within the same virtual network. There are limits to the number of private endpoints you can create in a subscription. For details, see Azure limits.
The PrivateDnsZoneGroup resource type establishes a relationship between the Private Endpoint and the Private DNS zone used for the name resolution of the fully qualified name of the resource referenced by the Private Endpoint. The PrivateDnsZoneGroup is a sub-resource or child-resource of a Private Endpoint. The PrivateDnsZoneGroup type a single property that contains the resource if of the referenced Private DNS Zone. The user that creates a PrivateDnsZoneGroup requires write permission on the private DNS Zone. Once created, this resource is used to manage the lifecycle of the DNS A record used to resolve the fully qualified name of the resource referenced by the Private Endpoint. When creating a Private Endpoint, the related A record will automatically be created in the target Private DNS Zone with the private IP address of the network interface associated to the Private Endpoint and the name of the Azure resource referenced by the Private Endpoint. When deleting a Private Endpoint, the related A record gets automatically deleted from the corresponding Private DNS Zone. The network resource provider (Microsoft.Network) identity is used to perform both operations. This means that the user provisioning a Private Endpoint doesn't require any write permissions on the Private DNS Zone or it's A records.
The following ARM templates under Azure Quickstart Templates GitHub account demonstrate how to properly deploy and configure private endpoints on Azure:
This picture represents the architecture of the Contoso environment. This topology is subject to change based on the architectural decision about adopting a centralized or decentralized model for Private DNS Zones and Private Endpoints.
The centralized topology is characterized by the following requirements and configurations:
As an alternative solution, you can create an automated procedure to detect the provisioning and deprovisioning of private endpoints and of the related network interfaces and contextually create or delete related A record in the corresponding privatelink.* Private DNS Zone, depending on the resource type. We created the following event-driven solution based on Event Grid to handle the events raised by an Azure subscription whenever a private endpoint, or better, the related NIC, gets created or deleted. The Azure Function runs with a system-assigned managed identity or service principal that has the necessary permissions (e.g. Reader built-in role over all the subscriptions) to read NIC information from any subscription, and also read, write and delete permissions on A records in the centralized privatelink.* Private DNS Zones. The solution works as expected, but Contoso perceives this solution as too complex. The solution is available here on GitHub.
The centralized topology is characterized by the following requirements and configurations:
Every subscription has a dedicated instance of the privatelink.* Private DNS Zones (e.g. privatelink.blob.core.windows.net for blob storage accounts) used for the name resolution of private endpoints, or better, for the name resolution of the FDNQ of the resource to the private IP address of the NIC associated to the private endpoint via an A record.
A virtual network link exists between each virtual network and the privatelink.* Private DNS Zones defined in the same subscription.
All the spoke virtual networks are configured to use a local, custom DNS server\forwarder defined in the same subscription.
DNS servers within a hub or spoke virtual network can forward DNS queries to the recursive resolvers in Azure. Access to the recursive resolvers in Azure is provided via the virtual IP 168.63.129.16.
DNS forwarding also enables DNS resolution between virtual networks and allows virtual machines in the on-premises virtual network that simulates the corporate datacenter to resolve Azure-provided host names.
To resolve a VM's host name, the DNS server VM must reside in the same virtual network and be configured to forward host name queries to Azure.
In each spoke\workload subscription, there is a subscription.local.contoso.com Private DNS Zones used for the name resolution of virtual machines.
Because the DNS suffix is different in each virtual network, conditional forwarding rules can be used by the DNS servers in the spoke virtual network to send DNS queries to the DNS resolver in the hub virtual network which in turn can use forwarding rules to forward the DNS query to DNS server in the right spoke virtual network or use subscription.local.contoso.com Private DNS Zones for the name resolution of virtual machines.
A virtual network link with auto-registration enabled exists between each spoke virtual network and the subscription.local.contoso.com Private DNS Zone in the same subscription.
A virtual network link with auto-registration disabled exists between the hub virtual network and each subscription.local.contoso.com Private DNS Zone and its subscription.
Private endpoints used by spoke virtual networks are created in the corresponding workload subscription and use the local DNS server + privatelink.* Private DNS Zones for the name resolution to the private IP of the NICs associated to the private endpoints.
Private endpoints used by the hub virtual network or by the on-premises network are created in a CommonSubnet in the shared services subscription and use the local DNS server + privatelink.* Private DNS Zones for the name resolution to the private IP of the NICs associated to the private endpoints.
Only the platform team (SysOps, NetOps, and SecOps) have write permissions on any resource in the shared services\hub subscription.
The users of the application teams are assigned the Contoso Owner or Contoso Contributor custom role which included the following on the Private DNS Zones located in their workload subscriptions:
Hence, app users can read, create, update, and delete A records for the private endpoints in full autonomy without the intervention of the central platform team.
In the Hybrid model, some spoke virtual networks and subscriptions follow the centralized model as a default approach for most of the spoke virtual network, and the decentralized model for a small number of spoke virtual networks that need a special degree of isolation and control.
There are two conditions that must be true to allow application teams the freedom to create any Azure PaaS resources they want in their subscription:
The following sections describe how the central platform team can enable these conditions by using Azure Policy. We will use Azure Storage as the Azure service that application teams need to deploy in our example below, but the same principle can be applied to most Azure services that support private link.
Create the Private DNS Zone in the central connectivity subscription for blob storage account private link services as per documentation. In our case, as we will use Storage account with blob as our example, it translates to us creating a privatelink.blob.core.windows.net private DNS zone in the connectivity subscription. You can use the ARM template available on Azure Quickstart Templates to create the
By default, storage accounts accept connections from clients on any network. To limit access to selected networks, you must first change the default action. Hence, as documented at Change the default network access rule, the first step to guarantee that a storage account can only be accessed via a private endpoint is setting its default access rule to Deny via PowerShell, Azure CLI, ARM template or via the Azure Portal by choosing Selected networks under the Firewall and virtual networks blade
In addition to the Private DNS zones, you also need to [create a set of custom Azure Policy definitions](https://docs.microsoft.com/en-us/azure/governance/policy/tutorials/create-custom-policy-definition) to enforce the use of private endpoints and automate the DNS record creation in the DNS zone we just created. You can deploy and use the following policies to find the storage accounts that expose public endpoints and report them as non-compliant, deny the creation of storage accounts with public endpoints enabled or automatically to create the PrivateDnsZoneGroup whenever a user deploys a private endpoint for a storage account. Let's look at each policy.
The following policy can be used to find all the storage account that are not properly configured to use a private endpoint.
{
"properties": {
"name": "storage-accounts-should-use-a-private-endpoint-policy-definition",
"displayName": "Storage Accounts should use a private endpoint.",
"description": "Audits any Storage Account not configured to use a private endpoint.",
"policyType": "Custom",
"metadata": {
"class": "Samples"
},
"mode": "All",
"parameters": {
"effect": {
"type": "string",
"defaultValue": "auditIfNotExists",
"allowedValues": [
"auditIfNotExists",
"disabled"
],
"metadata": {
"displayName": "Effect",
"description": "Enable or disable the execution of the policy"
}
}
},
"policyRule": {
"if": {
"field": "type",
"equals": "Microsoft.Storage/storageAccounts"
},
"then": {
"effect": "[parameters('effect')]",
"details": {
"type": "Microsoft.Storage/storageAccounts/privateEndpointConnections",
"existenceCondition": {
"field": "Microsoft.Storage/storageAccounts/privateEndpointConnections/
privateLinkServiceConnectionState.status",
"equals": "Approved"
}
}
}
}
}
}
The following policy can be used to set the default access rule of a storage account to Deny whenever a resource of this type is created or updated.
{
"properties": {
"displayName": "Storage Account default network access rule should be set to Deny.",
"description": "Sets the value of the networkAcls.defaultAction property of a storage account to Deny.",
"policyType": "Custom",
"metadata": {
"class": "Samples"
},
"mode": "All",
"parameters": {
"effect": {
"type": "string",
"defaultValue": "append",
"allowedValues": [
"append",
"disabled"
],
"metadata": {
"displayName": "Effect",
"description": "Enable or disable the execution of the policy"
}
}
},
"policyRule": {
"if": {
"allOf": [{
"field": "type",
"equals": "Microsoft.Storage/storageAccounts"
},
{
"field": "Microsoft.Storage/storageAccounts/networkAcls.defaultAction",
"notEquals": "Deny"
}
]
},
"then": {
"effect": "[parameters('effect')]",
"details": [{
"field": "Microsoft.Storage/storageAccounts/networkAcls.defaultAction",
"value": "Deny"
}]
}
}
}
}
The following Azure Policy checks the networkAcls.defaultAction property of storage accounts and determine whether requests from the public network are allowed or not by default. The policy uses the Deny effect to impede the creation of the Microsoft.Storage/storageAccounts resource type if the value of the networkAcls.defaultAction property is not equal to Deny.
{
"properties": {
"displayName": "Deny Storage Account default with network access rule other than Deny",
"description": "Denies the creation of storage accounts with the default network
access rule not equal to deny.",
"policyType": "Custom",
"metadata": {
"class": "Samples"
},
"mode": "All",
"parameters": {
"effect": {
"type": "string",
"defaultValue": "deny",
"allowedValues": [
"deny",
"disabled"
],
"metadata": {
"displayName": "Effect",
"description": "Enable or disable the execution of the policy"
}
}
},
"policyRule": {
"if": {
"allOf": [
{
"field": "type",
"equals": "Microsoft.Storage/storageAccounts"
},
{
"field": "Microsoft.Storage/storageAccounts/networkAcls.defaultAction",
"notEquals": "Deny"
}
]
},
"then": {
"effect": "[parameters('effect')]"
}
}
}
}
The following policy can be used to deny the creation of Private DNS Zone in any subscription other than the central networking subscription that hosts the hub virtual network.
{
"Description": "This policy restrict creation Private DNS Zones with the private link prefix",
"DisplayName": "Deny-PrivateDNSZone-PrivateLink",
"Mode": "All",
"Parameters": null,
"PolicyRule": {
"if": {
"allOf": [{
"field": "type",
"equals": "Microsoft.Network/privateDnsZones"
},
{
"field": "name",
"like": "privatelink*"
}
]
},
"then": {
"effect": "Deny"
}
}
}
This policy will be triggered if a private endpoint resource is created with a service-specific groupId. The groupId is the id of the group obtained from the remote resource (service) that this private endpoint should connect to. We then trigger a deployment of a PrivateDnsZoneGroup within the private endpoint, which is used to associate the private endpoint with our Azure Private DNS zone. For storage accounts, the groupId is equal to blob, for ADLS id equal to dfs and so on. For more information on Private Endpoints, Private DNS Zones and groupIds , see Azure Private Endpoint DNS configuration. When the policy finds that groupId in the Private Endpoint created, it will deploy a PrivateDnsZoneGroup within the private endpoint, and it will linked to the Private DNS Zone resource ID that is specified as parameter. For our example, the Private DNS Zone resource id would be like:
/subscriptions/<subscription-id>/resourceGroups/<resourceGroupName>/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net
An A record will automatically be created in the Private DNS Zone for the current private endpoint. Should the private endpoint be deleted, the A record will be automatically removed from the Private DNS Zone using the identity of the network provider. Note that this policy is generic and can be used for any kind of private endpoint: the groupId and resource id of the target Private DNS Zone must be specified with a parameter value when deploying the policy assignment.
{
"name": "CreatePrivateDnsZoneGroup",
"properties": {
"displayName": "Private Endpoints should use a PrivateDNSZoneGroup",
"description": "This policy automatically deploys a PrivateDnsZoneGroup
Azure Private Endpoints of a given type.",
"policyType": "Custom",
"mode": "Indexed",
"metadata": {
"version": "1.0.0",
"category": "Network Democratization",
"assignedBy": "Microsoft AGCI CE Team"
},
"parameters": {
"effect": {
"type": "string",
"defaultValue": "deployIfNotExists",
"allowedValues": [
"deployIfNotExists",
"disabled"
],
"metadata": {
"displayName": "Effect",
"description": "Enable or disable the execution of the policy."
}
},
"privateDnsZoneId": {
"type": "string",
"metadata": {
"displayName": "Specifies the resource id of the PrivateDnsZone
referenced by the PrivateDnsZoneGroup resource.",
"strongType": "Microsoft.Network/privateDnsZones"
}
},
"groupId": {
"type": "string",
"metadata": {
"displayName": "Specifies the group id of the private link sub resource."
}
}
},
"policyRule": {
"if": {
"allOf": [{
"field": "type",
"equals": "Microsoft.Network/privateEndpoints"
},
{
"count": {
"field": "Microsoft.Network/privateEndpoints/privateLinkServiceConnections[*].groupIds[*]",
"where": {
"field": "Microsoft.Network/privateEndpoints/privateLinkServiceConnections[*].groupIds[*]",
"equals": "[parameters('groupId')]"
}
},
"greaterOrEquals": 1
}
]
},
"then": {
"effect": "[parameters('effect')]",
"details": {
"type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups",
"roleDefinitionIds": [
"/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c"
],
"existenceCondition": {
"allOf": [{
"field": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/
privateDnsZoneConfigs[*].privateDnsZoneId",
"equals": "[parameters('privateDnsZoneId')]"
}]
},
"deployment": {
"properties": {
"mode": "incremental",
"parameters": {
"privateDnsZoneId": {
"value": "[parameters('privateDnsZoneId')]"
},
"privateEndpointName": {
"value": "[field('name')]"
},
"location": {
"value": "[field('location')]"
}
},
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"privateDnsZoneId": {
"type": "string"
},
"privateEndpointName": {
"type": "string"
},
"location": {
"type": "string"
}
},
"resources": [{
"name": "[concat(parameters('privateEndpointName'), '/deployedByPolicy')]",
"type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups",
"apiVersion": "2020-05-01",
"location": "[parameters('location')]",
"properties": {
"privateDnsZoneConfigs": [{
"name": "privateDnsZoneConfig",
"properties": {
"privateDnsZoneId": "[parameters('privateDnsZoneId')]"
}
}]
}
}]
}
}
}
}
}
}
}
}
Here is a policy assignment for blob storage accounts:
{
"name": " CreatePDZG4BlobStoragePE",
"location": "westeurope",
"Identity": {
"type": "SystemAssigned"
},
"properties": {
"displayName": "Blob Storage Accounts Private Endpoints should use a
PrivateDNSZoneGroup ",
"description": "This policy automatically deploys a PrivateDnsZoneGroup
for Blob Storage Accounts Private Endpoints.",
"metadata": {
"version": "1.0.0",
"category": "Network Democratization",
"assignedBy": "Microsoft AGCI CE Team"
},
"policyDefinitionId": "/providers/Microsoft.Management/managementgroups/
Contoso/providers/Microsoft.Authorization/policyDefinitions/OP-CreatePrivateDnsZoneGroup",
"scope": "/providers/Microsoft.Management/managementgroups/Production",
"notScopes": [],
"parameters": {
"privateDnsZoneId": {
"value": "/subscriptions/e123a06e-e380-42e6-80c9-1aa1f5db8b1a/
resourceGroups/network-ne/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net"
},
"groupId": {
"value": "blob"
}
}
}
}
The conclusion of our investigation is that the centralized model for managing Private DNS Zones should be used as a reference approach, while the decentralized model should only be adopted when application teams have very strict requirements in terms of workload isolation, sandboxing, and security of private endpoints. The lifecycle of the DNS records required by private endpoints can be entirely managed via Azure Policy and PrivateDnsZoneGroup as explained above.
These articles have been used to support our analysis: