Azure Analytics End to End with Azure Synapse - Deployment Accelerator

Overview

This is a deployment accelerator based on the reference architecture described in the Azure Architecture Center article Analytics end-to-end with Azure Synapse. This deployment accelerator aims to automate not only the deployment of the services covered by the reference architecture, but also to fully automate the configuration and permissions required for the services to work together. The deployed architecture enables the end-to-end analytics platform capable of handling the most common uses cases for most organizations.

The implementation of this deployment accelerator is done through the use of Azure Bicep, a domain-specific language (DSL) that uses declarative syntax to deploy Azure resources.

Deploy

Before you hit the deploy button, make sure you review the details about the services deployed.

Note: The "Deploy to Azure" button above will redirect you to the Azure Portal with a reference to the resulting ARM template file generated by the build of the Bicep code. Please refer to Bicep files for the true source of the code for this accelerator.

You can also use Azure CLI to deploy the services:

For a full deployment of all workloads with public endpoints use the command below:

az deployment group create --resource-group resource-group-name --template-file ./AzureAnalyticsE2E.bicep --parameters synapseSqlAdminPassword=use-complex-password-here

For a full deployment of all workloads with vNet integrated endpoints use the command below:

az deployment group create --resource-group resource-group-name --template-file ./AzureAnalyticsE2E.bicep --parameters networkIsolationMode=vNet synapseSqlAdminPassword=use-complex-password-here

You can have more control over the deployment by providing values to optional template parameters in the form of:

az deployment group create --resource-group resource-group-name --template-file ./AzureAnalyticsE2E.bicep --parameters synapseSqlAdminPassword=use-complex-password-here param1=value1 param2=value2...

Important: This deployment accelerator is meant to be executed under no interference from Azure Policies that deny certain configurations as they might prevent the its successful completion. Please use a sandbox environment if you need to validate the deployment resulting configuration before you run it against other environments under Azure Policies.

Important: This deployment accelerator implements some service features that are still in Public Preview. Please consider those before you plan for a production deployment.

Required Resource Providers

The target subscription for the deployment accelerator needs to have the following resource providers enabled before the deployment execution:

• Microsoft.Synapse
• Microsoft.Purview
• Microsoft.MachineLearningServices
• Microsoft.ContainerRegistry
• Microsoft.Network
• Microsoft.DataShare
• Microsoft.Authorization
• Microsoft.CognitiveServices
• Microsoft.ManagedIdentity
• Microsoft.KeyVault
• Microsoft.Storage
• Microsoft.StreamAnalytics
• Microsoft.Devices
• Microsoft.Insights
• Microsoft.EventHub
• Microsoft.DocumentDB

Deployment Details

The deployment accelerator can be deployed in two network isolation modes: default or vNet.

Azure Services Provisioned

The scope of this deployment accelerator is illustrated in the diagram below.

Important: All services are deployed in a single resource group and in the same region as the resource group. Before creating the resource group that will host the workloads, check the Azure Products by Region and select a region that has all selected services available.

Currently the following regions support the deployment of all services:

• Southeast Asia
• Australia East
• Canada Central
• North Europe
• West Europe
• Central India
• UK South
• East US
• East US 2
• South Central US
• West US 2

Other Azure regions are supported for partial deployments. The deployment will fail if any of the services selected is not available in the chosen Azure region.

Important: For a fully automated deployment and configuration of Synapse Analytics and Purview the deployment accelerator makes use of post-deployment PowerShell scripts to perform data plane operations. The operations executed by these scripts are to execute operations to complement the final environment configuration as not every setting is available through Bicep. Because of these imperative actions executed by the scripts, the template is no longer idempotent and should only be used for initial deployment and configuration. For more details about the scripts see the deployment accelerator documentation.

The default pricing tier for all services are provisioned are their lowest possible to meet the initial deployment requirements. If you choose to provide different different values to the input parameters, please observe the pricing information for each service in the table below.

If explicit names are not provided, all services names will be appended with a unique 5-letter suffix to ensure name uniqueness in Azure.

The Azure services used in the architecture above have been divided into workloads (see workload tables below) that can be conditionally deployed based on input parameters. The only mandatory workload is Synapse Analytics represented in the grey box in the diagram above.

Platform Services

Synapse Analytics

Data Governance

Artificial Intelligence (AI)

Data Sharing

Streaming

Operational Databases

Integration and Permissions

Beyond the deployment of the services that make up the reference architecture, this template also automates the configuration of connections and permissions between the services in order for the to work properly. Every arrow you see in the diagram above represents a configuration step that has been automated for you saving you a lot of time to get to insights.

Each connection and permission in the list below has been implemented following the technical documentation for the services involved below. Check the reference documentation links below for more information about them.

Service Connections

These are the service connections explicitly defined in deployment accelerator template. These connections represent the necessary configuration for the services to be fully integrated and work well together. Note that these connections may result in implicit RBAC permissions set between resources participating in the connection that are not in the permission list below. Check the reference documentation of each service connection below for more information.

ID	From Service	To Service	Connection Type	Reference Documentation
	azsynapsewkssuffix	azwksdatalakesuffix	Linked Service	Control storage account access for serverless SQL pool in Azure Synapse Analytics
	azsynapsewkssuffix	azpurviewsuffix	Workspace Connection	Connect a Synapse workspace to an Azure Purview account
	azsynapsewkssuffix	azkeyvaultsuffix	Linked Service	Store credential in Azure Key Vault
	azsynapsewkssuffix	azanomalydetectorsuffix, aztextanalyticssuffix	Linked Service	Configure prerequisites for using Cognitive Services in Azure Synapse Analytics
	azsynapsewkssuffix	azmlwkssuffix	Linked Service	Create a new Azure Machine Learning linked service in Synapse
	azsynapsewkssuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Linked Service	Control storage account access for serverless SQL pool in Azure Synapse Analytics
	azpurviewsuffix	azkeyvaultsuffix	Service Connection	Credentials for source authentication in Azure Purview
	azpurviewsuffix	azdatasharesuffix	Service Connection	How to connect Azure Data Share and Azure Purview
	azpurviewsuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Data Source	Connect to Azure Data Lake Gen2 in Azure Purview
	azmlwkssuffix	azmlstoragesuffix	Linked Service	Connect to storage services on Azure
	azmlwkssuffix	azmlappinsightssuffix	Linked Service	Monitor and collect data from ML web service endpoints
	azmlwkssuffix	azmlcontainerregsuffix	Linked Service	Manage Azure Machine Learning workspaces in the portal or with the Python SDK
	azmlwkssuffix	azkeyvaultsuffix	Linked Service	Use authentication credential secrets in Azure Machine Learning training runs
	azpurviewsuffix	azsynapsewkssuffix	Data Source	Connect to and manage Azure Synapse Analytics workspaces in Azure Purview
	azmlwkssuffix	azsynapsewkssuffix	Linked Service	Link Azure Synapse Analytics and Azure Machine Learning workspaces and attach Apache Spark pools
	azmlwkssuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Datastore	Connect to storage services on Azure
	azeventhubnssuffix	azrawdatalakesuffix	Event Capture	Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage
	azsynapsewkssuffix	azcosmosdb suffix	Linked Service	Connect to Azure Synapse Link for Azure Cosmos DB

Azure Role Based Access Control (RBAC) Permissions

Beyond the service connections created above, the deployment accelerator template defined Azure RBAC permissions between the services. These are the minimum level of permissions granted to their system-assigned identity (MSI) for the integration to function properly. These are the Azure RBAC permissions explicitly set by the template and the reason for these permissions to exist is describer in the reference documentation for each one of them.

ID	Granted To Service	Granted On Service	Permission Level	Reference Documentation
	azsynapsewkssuffix	azwksdatalakesuffix	Storage Blob Data Contributor	Grant permissions to workspace managed identity
	azpurviewsuffix	azsynapsewkssuffix	Reader	Connect to and manage Azure Synapse Analytics workspaces in Azure Purview
	azsynapsewkssuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Storage Blob Data Contributor	Grant permissions to workspace managed identity
	azsynapsewkssuffix	azmlwkssuffix	Contributor	Create a new Azure Machine Learning linked service in Synapse
	azpurviewsuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Storage Blob Data Reader	Connect to Azure Data Lake Gen2 in Azure Purview
	azdatasharesuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Storage Blob Data Reader	Roles and requirements for Azure Data Share
	azmlwkssuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Storage Blob Data Reader	Connect to storage by using identity-based data access
	azstreamjobsuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Storage Blob Data Contributor	Use Managed Identity to authenticate your Azure Stream Analytics job to Azure Blob Storage
	aziothubsuffix	azrawdatalakesuffix, azcurateddatalakesuffix	Storage Blob Data Contributor
	azstreamjobsuffix	azeventhubnssuffix	Event Hub Data Owner	Use managed identities to access Event Hub from an Azure Stream Analytics job
	azstreamjobsuffix	aziothubsuffix	IoT Hub Data Receiver	Control access to IoT Hub by using Azure Active Directory
	azpurviewsuffix	Resource Group	Storage Blob Data Reader	Connect to and manage Azure Synapse Analytics workspaces in Azure Purview

Data Plane Permissions

ID	Granted to Service	Granted On Service	Permission Level	Reference Documentation
	azsynapsewkssuffix	azkeyvaultsuffix	Get and List Secrets	Use Azure Key Vault secrets in pipeline activities
	azpurviewsuffix	azkeyvaultsuffix	Get and List Secrets	Credentials for source authentication in Azure Purview
	azmlwkssuffix	azsynapsewkssuffix	Synapse Apache Spark Administrator	Link Azure Synapse Analytics and Azure Machine Learning workspaces and attach Apache Spark pools
	azsynapewkssuffix	azpurviewsuffix	Data Curator	Connect a Synapse workspace to an Azure Purview account
	azdatasharesuffix	azpurviewsuffix	Data Curator	How to connect Azure Data Share and Azure Purview
	azsynapsewkssuffix	azcosmosdbsuffix	Cosmos DB Built In Data Contributor	HConfigure role-based access control with Azure Active Directory for your Azure Cosmos DB account

Networking Architecture

If you choose for a 'vNet Integrated' network isolation mode then the following applies:

• The Synapse Workspace will be deployed with a Managed Virtual Network.
• Managed private endpoints for some of the services will be created in the Synapse Workspace managed virtual network.
• Either a new or an existing virtual network will be used to deploy the private endpoints for all services in the architecture that support Private Link.
• Public access will be disabled and firewall rules will be set to restrict connectivity to and from the virtual network and between the services in the architecture.
• Private DNS zones required by the different private link domains can be optionally deployed and linked to the selected virtual network.

The following extra services will be deployed to support the private connectivity configuration:

Beyond the extra services above required to support the network isolation mode, the following network settings are applied to the services:

Workload	Name	Type	Network Settings	Notes	Reference Documentation
Platform Services	azkeyvaultsuffix	Key vault		'Allow Azure Services' required for access from Azure Purview and Azure ML	Configure Azure Key Vault networking settings
Synapse Analytics	azsynapsewkssuffix	Synapse workspace		Managed Virtual Network enabled	Understanding Azure Synapse Private Endpoints
Synapse Analytics	azwksdatalakesuffix	Storage account			Configure Azure Storage firewalls and virtual networks
Synapse Analytics	azrawdatalakesuffix	Storage account		'Allow Azure Services' enabled only when deploying Streaming workloads with Event Hubs	Configure Azure Storage firewalls and virtual networks
Synapse Analytics	azcurateddatalakesuffix	Storage account		'Allow Azure Services' enabled only when deploying Streaming workloads with Event Hubs	Configure Azure Storage firewalls and virtual networks
Data Governance	azpurviewsuffix	Purview account			Connect to your Azure Purview and scan data sources privately and securely
AI	azanomalydetectorsuffix	Anomaly detector			Configure Azure Cognitive Services virtual networks
AI	aztextanalyticssuffix	Language			Configure Azure Cognitive Services virtual networks
AI	azmlwkssuffix	Machine learning workspace			Secure Azure Machine Learning workspace resources using virtual networks (VNets)
AI	azmlstoragesuffix	Storage account			Secure an Azure Machine Learning workspace with virtual networks
AI	azmlcontainerregsuffix	Container registry			Secure an Azure Machine Learning workspace with virtual networks
Streaming	azeventhubnssuffix	Event Hub namespace			Network security for Azure Event Hubs
Streaming	aziothubsuffix	IoT Hub			IoT Hub support for virtual networks with Private Link and Managed Identity
Streaming	azstreamjobsuffix	Stream Analytics job		Stream Analytics Jobs don't support vNet integration. For that you should use Stream Analytics Clusters
Operational DB	azcosmosdbsuffix	Cosmos DB account			Configure Azure Private Link for Azure Cosmos DB analytical store

Contributing

If you would like to contribute to the solution (log bugs, issues, or add code) we have details on how to do that in our CONTRIBUTING.md file.

License

Details on licensing for the project can be found in the LICENSE file.