How to Build a Dev → Staging → Production Release Pipeline for an AI Application on Azure
A successful container deployment proves that an application can run. It does not prove that production received the same artifact tested in staging, that environment credentials are isolated, that an authorized person reviewed the release, or that operators can identify and restore a known-good version.
In this tutorial, we extend the existing document-insight-api Azure CI/CD project into an enterprise-style promotion pipeline. Azure DevOps builds the Docker image once, stores it in Azure Container Registry (ACR), resolves its immutable manifest digest, and promotes that exact digest through development, staging, and production Azure Container Apps. Development and staging deploy automatically. Production waits on an approval check owned by the document-insight-production Azure DevOps Environment.
Each runtime has its own Azure resource group, Container Apps environment, Log Analytics workspace, Key Vault, managed identity, and environment configuration. A tested rollback script creates a new revision from a recorded known-good digest rather than rebuilding old source.
The sample application performs deterministic text summarization and does not call a paid model. That keeps the tutorial focused on release governance. A real AI service must add model evaluation, data classification, prompt and retrieval versioning, tool authorization, privacy controls, and workload-specific reliability engineering.
What You Will Build
The release path is:
Reviewed GitHub commit
↓
Tests and release-control validation
↓
Build one Docker image
↓
ACR commit tag → resolved sha256 digest → locked tag
↓
Development deployment job + live verification
↓
Staging deployment job + live verification
↓
Azure DevOps production Environment approval
↓ approved only
Production deployment job + live verification
The implementation demonstrates:
One container build per release.
Deployment by registry/repository@sha256:digest, never latest.
Automatic, ordered dev and staging promotion.
A production approval gate that cannot be removed by editing pipeline YAML alone.
Azure DevOps Environment deployment history and commit/work-item traceability.
Separate resource groups, runtime identities, Key Vaults, Container Apps environments, and logging workspaces.
Version-pinned Key Vault secret references resolved by managed identity.
Environment-specific scaling without rebuilding the image.
Live verification before the next environment is allowed to run.
A controlled rollback plan based on a known-good digest.
Why This Matters in Production
Rebuilding for each environment creates three artifacts that may differ because dependency indexes, base images, or build tools can change between runs. Tag-only promotion is also weak when a registry tag can be overwritten. Deploying the resolved digest makes the artifact identity explicit.
AI systems have another source of drift: behavior can change through model versions, system prompts, retrieval collections, safety policies, provider endpoints, or secret rotation even when the container is unchanged. The release record therefore needs both artifact identity and environment-configuration evidence.
Human approval is useful only when it protects a real boundary. Azure DevOps approvals and checks are managed by resource owners and are not defined in the YAML file. A check on the production Environment pauses a stage before it can consume that resource. A pipeline editor cannot silently delete that check through the same pull request. Microsoft documents this separation in Approvals and checks.
An approval is not a substitute for automated validation. In this design, the reviewer receives test results, the image digest, successful dev and staging verification, pinned secret metadata, and rollback information before making the production decision.
Target Architecture
The registry is shared so the release moves without copying or rebuilding the image. Runtime access remains separate: each user-assigned identity receives AcrPull on ACR and Key Vault Secrets User only on its own vault. The deployment service connections are also environment-specific.
The repository uses empty Azure DevOps Environments as logical deployment targets. Microsoft recommends this pattern when you want deployment history even when the managed service is not registered as a VM or Kubernetes environment resource. Environment history can show the pipeline runs, newly deployed commits, and associated work items. See Create and target Azure DevOps Environments.
What We Reused from the Existing Azure Project
The first Azure tutorial already established the application and container baseline. This project reuses:
The document-insight-api FastAPI service.
/healthz, /readyz, /version, and /summarize.
Structured JSON logging without request or response bodies.
A Python 3.13 multi-stage Docker build.
Non-root UID 10001, port 8080, and the container health check.
Unit-test, local container, and remote smoke-test patterns.
Reuse is intentional: an environment-promotion tutorial should not introduce an unrelated application. The provider-specific delivery layer changes substantially. The new project adds digest capture, three deployment jobs, separate infrastructure, versioned Key Vault references, Environment history, production checks, release metadata, and rollback tooling.
Prerequisites
Prepare these items before changing Azure:
An approved Azure sandbox subscription.
An Azure DevOps organization and project connected to the GitHub repository.
Permission to create subscription deployments, resource groups, ACR, Container Apps environments, Key Vaults, managed identities, role assignments, and Log Analytics workspaces.
Permission to create Azure DevOps Environments, checks, pipeline permissions, and service connections.
Azure CLI, the Container Apps extension, Bicep, Docker, PowerShell 7, Git, and Python 3.13.
A globally unique lowercase alphanumeric ACR name.
Authorized release approvers who are not automatically the same people who edit the pipeline.
A release record or ticket where digest, evaluation, configuration, approval, and rollback evidence can be retained.
The tutorial uses eastus2, prefix docai, and public HTTPS endpoints for validation. Confirm service availability, organization policy, networking, and cost for your chosen region. Do not place credentials, secret values, customer data, production URLs, or tenant identifiers in the repository or screenshots.
Step 1: Validate the Reused AI API and Release Controls
Clone or copy the companion repository, then run:
cd .\examples\azure-ai-release-pipeline
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txt
python -m pytest -q
python .\scripts\check_release_config.py
The six tests cover liveness, readiness, request IDs, deterministic summarization, release metadata, secret non-disclosure, and invalid input. The offline validator checks the ordered stages, main-only publishing, digest capture, tag lock, three Environment names, production lock behavior, deployment template, Key Vault reference, managed identities, RBAC role IDs, live checks, and rollback assets.
The application exposes useful release evidence without exposing secrets:
{
"service": "document-insight-api",
"environment": "staging",
"model": "synthetic-summary-v1",
"source_revision": "<FULL_GIT_SHA>",
"release_id": "ado-142",
"container_app_revision": "document-insight-api-staging--r142-a1",
"model_secret": "configured"
}
During authoring, all six tests and the offline configuration validators passed. The portable smoke tester also verified /readyz, /version, and /summarize against a locally running FastAPI process. This proves the local application contract and required configuration markers; it does not prove the Docker image or an Azure deployment.
Step 2: Create Isolated Azure Foundations
The root Bicep template runs at subscription scope so it can create a shared registry resource group and three environment resource groups:
$SubscriptionId = '<AZURE_SUBSCRIPTION_ID>'
$AcrName = '<GLOBALLY_UNIQUE_LOWERCASE_ACR_NAME>'
az login
.\scripts\bootstrap-azure.ps1 `
-SubscriptionId $SubscriptionId `
-AcrName $AcrName `
-NamePrefix docai `
-Location eastus2
The expected resource layout is:
Scope | Important resources |
rg-docai-shared | Private ACR with administrator and anonymous pull disabled |
rg-docai-dev | Dev Container Apps environment, identity, Key Vault, Log Analytics |
rg-docai-staging | Staging Container Apps environment, identity, Key Vault, Log Analytics |
rg-docai-prod | Production Container Apps environment, identity, Key Vault, Log Analytics |
ach runtime identity receives the Azure built-in AcrPull role on the shared registry. It also receives Key Vault Secrets User on only its environment vault. Those role identifiers are declared in infrastructure/modules/environment.bicep, not replaced with broad Owner permissions.
The Container Apps themselves are created by the first deployment because the release image does not exist during foundation provisioning.
Step 3: Store and Pin Environment Configuration in Key Vault
Create model-api-key separately in each Key Vault through your approved secret-provisioning workflow. Use synthetic values in the tutorial. Do not reuse a production credential in dev or staging.
Read the identifier of the active version without printing the secret value:
az keyvault secret show `
--vault-name '<DEV_KEY_VAULT>' `
--name model-api-key `
--query id `
--output tsv
Repeat for staging and production. Put each versioned URI into the correct keyVaultSecretUri parameter in azure-pipelines.yml:
keyVaultSecretUri: 'https://<VAULT>.vault.azure.net/secrets/model-api-key/<VERSION>'
Container Apps supports Key Vault secret references using managed identity. Microsoft specifies that the identity needs secret access, and the CLI reference format combines keyvaultref:<URI> with identityref:<IDENTITY_ID>. The deployment template follows that format. See Manage secrets in Azure Container Apps.
The version segment is important. A versionless reference automatically follows the latest secret and can restart active revisions after rotation. That may be desirable for emergency rotation, but it is also configuration change outside the normal artifact promotion. This tutorial pins versions so each environment change is reviewed explicitly.
TODO: VERIFY Confirm each runtime identity can resolve only its own Key Vault reference and that the /version endpoint says model_secret: configured. Also verify that no log or API response contains the value.
Step 4: Configure Azure DevOps Service and Environment Boundaries
Create an ACR Docker Registry connection, a shared platform Azure Resource Manager connection, and one Azure Resource Manager connection per environment:
Connection | Responsibility |
sc-document-insight-acr | Push the built image |
sc-document-insight-platform | Resolve the digest and lock the tag |
sc-document-insight-dev | Deploy only to dev |
sc-document-insight-staging | Deploy only to staging |
sc-document-insight-production | Deploy only to production |
Prefer workload identity federation and resource-group scope. Grant only the additional permissions required to assign the existing runtime identity and manage the Container App. Keep the production connection authorization distinct from dev and staging.
Next, open Azure DevOps > Pipelines > Environments and pre-create:
document-insight-dev
document-insight-staging
document-insight-production
Restrict which pipelines can use each Environment and who can administer it. Pre-creation avoids relying on Azure Pipelines to auto-create an environment from YAML and lets owners establish the security boundary before execution.
Detailed setup is in docs/configure-azure-devops-environments.md.
Step 5: Build, Resolve, and Lock One Release Artifact
Pull requests run the test stage only. BuildAndPush contains a branch condition so publishing occurs only from refs/heads/main:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
The build uses the full commit SHA as its tag and embeds that revision in the image:
- task: Docker@2
inputs:
command: build
containerRegistry: sc-document-insight-acr
repository: $(imageRepository)
tags: |
$(Build.SourceVersion)
arguments: --build-arg BUILD_REVISION=$(Build.SourceVersion)
After push, the pipeline resolves the registry digest and emits it as an Azure Pipelines output variable:
DIGEST="$(az acr repository show \
--name '$(acrName)' \
--image '$(imageRepository):$(imageTag)' \
--query digest --output tsv)"
echo "##vso[task.setvariable variable=imageDigest;isOutput=true]$DIGEST"
It also disables write and delete operations for the release tag. The deployments still use the digest, so tag locking is defense in depth rather than the only immutability mechanism.
Step 6: Promote Automatically Through Development and Staging
Both lower environments use Azure DevOps deployment jobs and the same template. Each stage imports the output from BuildAndPush:
variables:
imageDigest: $[stageDependencies.BuildAndPush.BuildAndPush.outputs['CaptureImage.imageDigest']]
The deployment template constructs:
<acr>.azurecr.io/document-insight-api@sha256:<digest>
It assigns the environment's user identity, configures its ACR and Key Vault references, creates or updates the Container App, waits for the expected revision to become healthy, confirms the revision image equals the digest reference, and executes scripts/smoke_test.py.
The verifier must see:
APP_ENV=dev or APP_ENV=staging as appropriate.
The full Build.SourceVersion.
RELEASE_ID=ado-<Build.BuildId>.
The expected Container Apps revision name.
model_secret=configured.
A successful synthetic summary from the same source revision.
Staging depends on both Build and Dev. A failed dev deployment or verification prevents staging. Production similarly depends on the successful staging stage.
Step 7: Protect Production with Environment-Owned Approval
Open document-insight-production, select Approvals and checks, and add:
An Approvals check with the authorized group.
A timeout aligned with the change window.
Self-approval disabled where separation of duties requires it.
An Exclusive lock check.
The production stage contains:
lockBehavior: sequential
The exclusive lock prevents two production stages from changing the target concurrently. sequential queues runs in order rather than keeping only the newest run. Decide whether that policy is appropriate: an older approved release may no longer be desirable after a newer one exists.
Do not put approver identities in the repository. The approval configuration is resource-owned state in Azure DevOps. Before approving, use docs/production-approval-checklist.md to review the exact commit and digest, dev/staging evidence, AI evaluations, production secret version, resource diff, monitoring, and known-good rollback target.
Test both paths:
Reject one sandbox release and prove the production deployment job never begins.
Run a corrected release, approve it, and prove deployment begins only afterward.
Step 8: Verify Production History and Rehearse Rollback
After approval, the production deployment job uses sc-document-insight-production, deploys the same digest with production configuration, waits for a healthy revision, compares the live image reference, and smoke-tests the API.
Open Pipelines > Environments > document-insight-production > Deployments. The Environment history records which pipeline run targeted production and provides associated commit and work-item traceability. Drill into the job to connect the approval with the deployment result.
Verify the live revision directly:
az containerapp revision list `
--name document-insight-api-prod `
--resource-group rg-docai-prod `
--query "[].{name:name,image:properties.template.containers[0].image,health:properties.healthState,active:properties.active}" `
--output table
For rollback, first run the supplied script without -Execute:
.\scripts\rollback-azure.ps1 `
-Environment production `
-KnownGoodImage '<ACR>.azurecr.io/document-insight-api@sha256:<64_HEX_DIGEST>' `
-KnownGoodSourceRevision '<40_HEX_GIT_SHA>'
The dry run prints the exact target and creates nothing. After an authorized rollback decision, repeat with -Execute -ConfirmProduction. The script creates a new revision from the known-good digest and verifies its release metadata. Azure CLI also supports copying a previous Container Apps revision, but this tutorial deliberately anchors rollback to the recorded image digest; see Azure Container Apps revision commands.
Verify the Implementation
A publishable result should satisfy this evidence matrix:
Control | Evidence | Expected result |
Pull-request isolation | PR pipeline run | Test runs; image publication and deployment do not |
Artifact identity | ACR plus Build log | Full Git tag resolves to one sha256 digest |
Dev promotion | Deployment job and /version | Correct dev metadata and same digest |
Staging promotion | Deployment job and /version | Correct staging metadata and same digest |
Production gate | Pending/rejected run | Production job has not begun before approval |
Production release | Environment history, revision, and smoke log | Approved run, correct digest, healthy revision, correct metadata |
Secret isolation | Container App reference, RBAC, and API output | Versioned Key Vault URI; value absent from source, logs, and responses |
Rollback | Rehearsal record | Known-good digest creates and verifies a new revision |
Run the offline checks again before committing:
python -m pytest -q
python .\scripts\check_release_config.py
python ..\validate_projects.py
Then inspect the environment logs with observability/release-evidence.kql. Run it in each environment's Log Analytics workspace and correlate release_id, source_revision, Container App revision, pipeline run, and request ID.
Production Considerations
Security and Access Control
Use separate identities for image publication, shared-registry governance, and each environment deployment. Give runtime identities only AcrPull and secret-read access to their own vault. Restrict Environment administration, pipeline use, and service-connection authorization. For stronger blast-radius control, put production in a different subscription and update the Bicep/module scopes accordingly.
Public ingress exists only to make the tutorial verifiable. Add Microsoft Entra authentication, API Management, private ingress, Private Link, WAF and rate controls, caller authorization, and egress policy according to the workload.
AI Release Evidence
Container equality does not guarantee behavioral equality. Record model ID and version, prompt/template commit, retrieval index or dataset version, safety-policy version, evaluation suite and thresholds, provider configuration, and any human review. Fail production promotion when the exact release combination fails its approved evaluations.
Do not pass real prompts or customer data through a tutorial API. The sample omits bodies from logs, but teams still need data classification, telemetry review, retention, deletion, and incident procedures.
Reliability and Rollback
The reference uses single-revision mode. A higher-risk service may need multiple revision mode, labels, blue/green releases, canary traffic, metric checks, and automated abort. Test the chosen strategy under failure rather than assuming the platform default is sufficient.
Keep a release catalog mapping Git SHA, ACR digest, configuration versions, Container Apps revision, evaluation result, approval, and timestamp. A rollback is unsafe when operators must guess which image was good.
Auditability and Operations
Azure DevOps Environment history is one part of the audit trail. Retain branch reviews, build results, digest evidence, check history, deployment jobs, Key Vault audit logs, Azure Activity Log, application telemetry, incident links, and rollback decisions according to policy.
Configure alerts for failed revisions, elevated 5xx responses, latency, replica pressure, secret-resolution failures, and budget anomalies. Assign an owner and response action to each alert.
Cost and Scaling
Three Container Apps environments, three workspaces, Key Vault operations, ACR storage, telemetry ingestion, and hosted pipeline minutes may incur charges. Production uses one minimum replica in the sample, while dev and staging can scale to zero. These are teaching defaults, not sizing recommendations.
Confirm current Azure pricing before publication. Apply budgets, tags, log retention, artifact retention, and scheduled cleanup. Avoid leaving sandbox production replicas running without a reason.
Clean Up the Tutorial Resources
First retain any pipeline, approval, image, log, Key Vault metadata, and rollback evidence required by your organization. Review every target before deletion:
az resource list --resource-group rg-docai-dev --output table
az resource list --resource-group rg-docai-staging --output table
az resource list --resource-group rg-docai-prod --output table
az resource list --resource-group rg-docai-shared --output table
For a dedicated sandbox, delete the environment resource groups before the shared ACR group through your approved process. Resource-group deletion is irreversible. Soft-deleted Key Vault names may remain reserved during retention.
Azure resource cleanup does not remove Azure DevOps Environments, checks, service connections, pipeline definitions, GitHub authorization, or external release records. Remove or retain them deliberately.
Reference Implementation
The companion repository is examples/azure-ai-release-pipeline. It contains:
FastAPI source and six tests.
A production-oriented non-root Dockerfile.
A five-stage Azure Pipeline.
A reusable Azure Container Apps deployment-job template.
Subscription and environment Bicep modules.
Key Vault and ACR managed-identity RBAC.
Portable release smoke tests.
A release-control validator.
Azure DevOps Environment instructions and approval checklist.
A dry-run-first rollback script.
A Log Analytics release-evidence query.
Before publishing it to GitHub, replace the service connection, ACR, and versioned Key Vault URI placeholders; deploy only to an authorized sandbox; capture live evidence; and tag the reviewed repository version used by this article.
How Codersarts Can Help
CodersArts helps organizations turn containerized AI services into controlled delivery systems with environment isolation, federated CI/CD identity, immutable artifacts, secret governance, approval policy, automated evaluations, observability, and tested rollback runbooks.
Explore another CodersArts open-source project: Identity Verification API.
Contact us at Email: contact@codersarts.com
Conclusion
This release path builds one image, promotes one digest, verifies every environment, and places production authorization outside the pipeline file. Separate identities and Key Vaults reduce cross-environment access, Azure DevOps Environments preserve deployment history, and the rollback workflow starts from recorded evidence instead of a rebuild.
The next production step is to deploy the draft in an authorized Azure sandbox, exercise approval and rejection, rehearse rollback, capture the four evidence screenshots, and then layer in the real application's authentication, private networking, AI evaluations, policy, monitoring, and recovery requirements.
Comments