MLOps Gify Visual (Allow 20 Seconds To Load)

Version History And Updates

Azure ML Integration Now Live For GitHub Deployment.

Development for Azure DevOps Deployment in Progress

MLOps for Databricks with CI/CD (GitHub Actions)

MLOps Architecture

Features to be included in future releases:

• Model testing
• Metrics & Monitoring

Youtube Demo - Slightly Outdated

The deployment instructions for the video are slightly outdated (albeit still usefull). Please follow instructions below instead. The video still provides useful content for concepts outwith the deployment.

About This Repository

This Repository contains an Azure Databricks Continuous Deployment and Continuous Development Framework for delivering Data Engineering/Machine Learning projects based on the below Azure Technologies:

Azure Databricks is a powerful technology, used by Data Engineers and Scientists ubiquitously. However, operationalizing it within a Continuous Integration and Deployment setup that is fully automated, may prove challenging.

The net effect is a disproportionate amount of the Data Scientist/Engineers time contemplating DevOps matters. This Repository's guiding vision is to automate as much of the infrastructure as possible.

Prerequisites

Details of The Solution Accelerator

• Creation of four environments:
  • Sandbox
  • Development
  • User Acceptance Testing (UAT)
  • Production
• Full CI/CD between environments
• Infrastructure-as-Code for interacting with Databricks API and also CLI
• Azure Service Principal Authentication
• Azure resource deployment using BICEP
• Databricks Feature Store + MLFlow Tracking + Model Registry + Model Experiments
• DBX by Data Labs for Continuous Deployment of Jobs/Workflows (source code/ parameters files packaged within DBFS)

Deployment Instructions

Create Repository

Login To Azure

• All Code Throughout To Go Into VS Code PowerShell Terminal

az login

# If There Are Multiple Tenants In Your Subscription, Ensure You Specify The Correct Tenant "az login --tenant"

# ** Microsoft Employees Use: az login --tenant fdpo.onmicrosoft.com (New Non Prod Tenant )

GitHub Account

echo "Enter Your Git Username... "
# Example: "Ciaran28"
$Git_Configuration = "GitHub_Username"

GitHub Repos Within Databricks

echo "Enter Your Git Repo Url (this could be any Repository In Your Account )... "
# Example: "https://github.com/ciaran28/dstoolkit-mlops-databricks" 
$Repo_ConfigurationURL = ""

Updates Parameter Files & Git Push To Remote

echo "From root execute... "

./setup.ps1

Create Environments

Follow the naming convention (case sensitive)

Secrets

For each environment create GitHub Secrets entitled ARM_CLIENT_ID, ARM_CLIENT_SECRET and ARM_TENANT_ID using the output in VS Code PowerShell Terminal from previous step. (Note: The Service Principal below was destroyed, and therefore the credentials are useless )

In addition generate a GitHub Personal Access Token and use it to create a secret named PAT_GITHUB:

Final Snapshot of GitHub Secrets

Secrets in GitHub should look exactly like below. The secrets are case sensitive, therefore be very cautious when creating.

Deploy The Azure Environments

• In GitHub you can manually run the pipeline to deploy the environments to Azure using "onDeploy.yaml" found here. Use the instructions below to run the workflow.

• Azure Resources created (Production Environment snapshot - For speed I have hashed out all environment deployments except Sandbox. Update onDeploy.yaml to deploy all environments)

Running Pipelines

• The end to end machine learning pipleine will be pre-configured in the "workflows" section in databricks. This utilises a Job Cluster which will automatically upload the necessary dependencies contained within a python wheel file

• If you wish to run the machine learning scripts from the Notebook instead, first upload the dependencies (automatic upload is in development). Simply navigate to python wheel file contained within the dist/ folder. Manually upload the python wheel file to the cluster that you wish to run for the Notebook.

Continuous Deployment And Branching Strategy

The Branching Strategy I have chosen is configured automatically as part of the accelerator. It follows a GitHub Flow paradigm in order to facilitate rapid Continuous Integration, with some nuances. (see Footnote 1 which contains the SST Git Flow Article written by Willie Ahlers for the Data Science Toolkit - This provides a narrative explaining the numbers below)[^1]

The branching strategy is easy to change via updating the "if conditions" within .github/workflows/onRelease.yaml.

• Pull Request from Feature Branch to Main Branch: C.I Tests
• Pull Request approved from Feature Branch to Main Branch: C.D. to Development Environment
• Pull Request from Main Branch to Release Branch: C.I. Test
• Pull Request approved from Main Branch to Release Branch: C.D. to User Acceptance Testing (UAT) Environment
• Tag Version and Push to Release Branch: C.D. to Production Environment
• Naming conventions for branches (to ensure the CD pipelines will deploy - onRelease.yaml for more details ):
  • Feature Branches: "feature/"
  • Main Branch: "main"
  • Release branch "release/"

MLOps Paradigm: Deploy Code, not Models

In most situations, Databricks recommends that during the ML development process, you promote code, rather than models, from one environment to the next. Moving project assets this way ensures that all code in the ML development process goes through the same code review and integration testing processes. It also ensures that the production version of the model is trained on production code. For a more detailed discussion of the options and trade-offs, see Model deployment patterns.

https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns

Feature Store Integration

In an organization, thousands of features are buried in different scripts and in different formats; they are not captured, organized, or preserved, and thus cannot be reused and leveraged by teams other than those who generated them.

Because feature engineering is so important for machine learning models and features cannot be shared, data scientists must duplicate their feature engineering efforts across teams.

To solve those problems, a concept called feature store was developed, so that:

• Features are centralized in an organization and can be reused
• Features can be served in real-time with low latency