How to Build a Dev, Staging, Production Release Pipeline for an AI Application on GCP
Deploying an AI API once is not the same as operating a dependable release process. An enterprise team must be able to show what was built, which version reached each environment, who authorized production, how configuration and secrets were isolated, and what will happen when a release needs to be reversed.
In this tutorial, we extend the earlier containerized FastAPI project into a controlled Google Cloud release pipeline. Cloud Build tests the source and builds one Docker image. Artifact Registry stores that image under the Git commit and digest. Cloud Deploy then promotes the same release through separate development, staging, and production Cloud Run targets. Production has a required approval gate.
The sample document-insight-api performs deterministic text summarization rather than calling a paid model. This lets us test the release mechanics without sending data to a model or introducing variable output. The same pattern can later carry a Vertex AI or approved third-party integration, but a real AI workload also needs model evaluation, data governance, safety, latency, and cost controls.
What You Will Build
The completed release path is:
Protected GitHub main branch
↓
Cloud Build: test → build → push → resolve digest
↓
Artifact Registry: immutable commit tag and digest
↓
Cloud Deploy release
↓
Dev Cloud Run target
↓ deliberate promotion
Staging Cloud Run target
↓ deliberate promotion
Production rollout waiting for approval
↓ approve or reject
Production Cloud Run target
The reference design demonstrates:
One build artifact promoted across every environment.
Separate GCP projects for tooling, development, staging, and production.
Separate build, deployment, runtime, and approver identities.
Environment-specific configuration supplied by Cloud Deploy target parameters.
Environment-specific secrets resolved from Secret Manager at runtime.
A private Cloud Run service in each runtime project.
A production target with requireApproval: true.
Release verification through /version and a deterministic inference smoke test.
A Cloud Deploy rollback path to a known-good release.
Why This Matters in Production
Rebuilding an image for each environment weakens release evidence. Even when three builds start from the same Git commit, dependency downloads, base tags, timestamps, or build-system changes can produce different digests. The safer promotion unit is the image that already passed the earlier environment, not a new approximation of it.
Environment separation addresses a different risk. Development identities and experimentation should not silently inherit production data, secrets, quotas, or deployment authority. Separate projects provide clearer IAM, billing, audit, quota, and lifecycle boundaries than three loosely named services in one project.
Approval is also more than a button. The approver needs enough evidence to make a decision: source revision, image digest, earlier rollout status, AI evaluation results, rendered manifest changes, incident ownership, and a rollback candidate. Cloud Deploy records releases and rollouts and supports approval on a target, but the organization still owns the quality of the approval policy.
Target Architecture
Cloud Deploy targets must be registered in the same project and region as their delivery pipeline, but the Cloud Run services they reference can be in other projects and regions when the execution identity has access. Google recommends a different project for each Cloud Run environment in this pattern. See Deploy a Cloud Run service using Cloud Deploy.
The tooling project owns build and release control-plane resources. Each environment project owns its Cloud Run runtime identity and Secret Manager secret. Cloud Deploy parameters set non-sensitive environment values after manifest rendering; the secret value itself never becomes a deploy parameter or Git value.
What We Reused from the First GCP Project
The earlier gcp-containerized-ai-api-cicd project already proved a useful cloud-neutral application boundary:
FastAPI validation and deterministic /summarize behavior.
/healthz, /readyz, and /version endpoints.
Structured JSON logs without prompt or response-body logging.
A multi-stage Python 3.13 image.
Non-root runtime UID 10001.
Port 8080 and a container health check.
Unit tests and local smoke-test logic.
Those assets are reused because the business capability has not changed. The release system has.
The original direct Cloud Run deployment step is not reused. Cloud Build now stops after testing, building, pushing, resolving the digest, and creating a Cloud Deploy release. Cloud Deploy owns the rollout to all three environments. This separation prevents CI from becoming a second, competing production deployment path.
The companion repository is examples/gcp-ai-release-pipeline.
Prerequisites
Prepare these resources and decisions first:
Four billing-enabled GCP projects: tooling, dev, staging, and production.
Google Cloud CLI 541.0.0 or later. The current Cloud Deploy Cloud Run target guide specifies that minimum.
Docker Desktop or Docker Engine, Git, PowerShell 7, and Python 3.13.
A GitHub repository you can connect to Cloud Build.
Permission to enable APIs, create service accounts and secrets, register Cloud Deploy resources, and grant cross-project IAM bindings.
An administrator for foundation setup and a separate production approver identity or group.
An agreed region. The example uses us-central1 for the repository, build, delivery pipeline, and targets.
A sandbox rollout window and an owner for cost, logs, alerts, evidence retention, and cleanup.
Do not begin in shared production projects. Never commit project credentials, service-account keys, access tokens, model keys, prompt data, customer records, or generated responses.
Step 1: Validate the Reused Application and Container Contract
Clone or copy the companion repository, then run the application and release-configuration tests before creating cloud resources:
cd .\examples\gcp-ai-release-pipeline
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txt
python -m pytest -q
python .\scripts\check_release_config.py
python -m compileall -q app tests scripts
The test suite checks liveness, readiness, request-ID propagation, input validation, deterministic inference output, environment metadata, and source revision reporting. check_release_config.py then checks controls that are easy to remove accidentally: commit-based image tags, digest resolution, the three-target order, the production approval flag, environment parameters, the Secret Manager reference, and the Cloud Run Skaffold deployer.
During the authoring pass, all four application tests passed. The release-control check, project-wide repository check, mocked four-resource Cloud Deploy render, Python syntax check, and PowerShell parser also passed. These offline results validate the repository structure and logic; they do not prove a successful Cloud Build, Cloud Deploy, Secret Manager, IAM, or Cloud Run operation.
Build and exercise the container locally when Docker is available:
docker build `
--build-arg BUILD_REVISION=local-smoke `
--tag document-insight-api:local `
.
.\scripts\local-smoke.ps1 -Image document-insight-api:local
The local smoke script adds a read-only filesystem, drops Linux capabilities, enables no-new-privileges, waits for the image health check, calls the API, and removes the temporary container.
Step 2: Prepare the Four-Project Foundation
Choose explicit project IDs:
$ToolingProject = '<TOOLING_PROJECT_ID>'
$DevProject = '<DEV_PROJECT_ID>'
$StagingProject = '<STAGING_PROJECT_ID>'
$ProductionProject = '<PRODUCTION_PROJECT_ID>'
$Region = 'us-central1'
gcloud auth login
gcloud config set project $ToolingProject
Review scripts/bootstrap-gcp.ps1, then run it with an approved administrator identity:
.\scripts\bootstrap-gcp.ps1 `
-ToolingProjectId $ToolingProject `
-DevProjectId $DevProject `
-StagingProjectId $StagingProject `
-ProductionProjectId $ProductionProject `
-Region $Region
The script creates the following separation:
Identity | Location | Responsibility |
document-insight-build | Tooling project | Test, push the image, and create a Cloud Deploy release |
document-insight-deploy | Tooling project | Render and deploy through Cloud Deploy |
document-insight-runtime | Each runtime project | Run that environment's Cloud Run revision and read only its local secret |
Release approver | Workforce or approved group | Approve or reject the production rollout |
It also creates ai-api-images with immutable tags and allows each environment's Cloud Run service agent to pull the shared image. The build identity receives Artifact Registry write and Cloud Deploy release permissions; it does not receive the production approval role.
The script uses predefined roles for a readable tutorial. Derive approved custom roles, conditions, resource scopes, and organization-policy changes from the real operating model before adopting it broadly.
Step 3: Separate Non-Sensitive Parameters from Secrets
The repository uses one Cloud Run service manifest for all three environments. The differences are applied by Cloud Deploy after rendering:
annotations:
autoscaling.knative.dev/minScale: "0" # from-param: ${min_scale}
autoscaling.knative.dev/maxScale: "2" # from-param: ${max_scale}
run.googleapis.com/secrets: "model-api-key:projects/000000000000/secrets/document-insight-model-api-key" # from-param: ${secret_resource}
spec:
serviceAccountName: document-insight-runtime@example.iam.gserviceaccount.com # from-param: ${runtime_service_account}
containers:
- image: document-insight-api
env:
- name: APP_ENV
value: dev # from-param: ${app_env}
- name: MODEL_ID
value: synthetic-summary-v1 # from-param: ${model_id}
Each Cloud Deploy target supplies values such as app_env, scaling bounds, runtime identity, and the local secret resource. This keeps ordinary environment configuration visible in review while preserving a single manifest.
MODEL_API_KEY uses secretKeyRef. The configuration renderer pins the newest enabled version in each environment project when the pipeline is registered. Cloud Run's YAML format requires the secret location in the run.googleapis.com/secrets annotation and the environment variable to reference that lookup name. See Configure secrets for Cloud Run services.
Pinning a numeric version makes each rendered target deterministic. A production secret-rotation design must define version creation, configuration re-rendering, staged testing, rollout, revocation, incident access, and audit ownership. Do not replace the placeholder with a real value through command history.
Step 4: Register the Delivery Pipeline and Targets
The bootstrap renders clouddeploy/clouddeploy.template.yaml with the real project IDs and numbers, writes the ignored generated/clouddeploy.yaml, and applies it:
gcloud deploy apply `
--file .\generated\clouddeploy.yaml `
--project $ToolingProject `
--region $Region
The serial pipeline is intentionally short:
serialPipeline:
stages:
- targetId: dev
- targetId: staging
- targetId: prod
Only the production target requires approval:
metadata:
name: prod
requireApproval: true
run:
location: projects/<PRODUCTION_PROJECT_ID>/locations/us-central1
Grant roles/clouddeploy.approver to the release-manager identity or group, not to the build service account. Google Cloud also supports an IAM condition using the rollout-target attribute so approval authority can be limited to prod. See Use IAM to restrict Cloud Deploy access.
Cloud Deploy snapshots the delivery-pipeline configuration for a release. Later pipeline edits do not silently rewrite that existing release, so inspect mismatch warnings before promoting an older release.
Step 5: Build Once and Create the Dev Release
Connect the GitHub repository to Cloud Build, create a trigger for the protected main branch, and select cloudbuild.yaml. Configure the trigger to use:
document-insight-build@<TOOLING_PROJECT_ID>.iam.gserviceaccount.com
The important boundary in cloudbuild.yaml is that it does not call gcloud run deploy. After the tests pass, it builds and pushes the commit-tagged image, resolves the digest, and creates a release:
gcloud deploy releases create "rel-$SHORT_SHA" \
--delivery-pipeline="document-insight-release" \
--images="document-insight-api=$IMAGE_URI@$DIGEST"
The unqualified image: document-insight-api in the service manifest is replaced with that full Artifact Registry digest during rendering. The default release behavior creates the initial rollout to the first target, dev. Google documents this CI integration and image mapping in Integrating Cloud Deploy with your CI system.
Protect main with required review and CI checks. Treat pull-request code as untrusted; do not make production secrets or deployment credentials available to arbitrary contributor code.
Step 6: Verify Dev and Promote the Same Release to Staging
Get the commit used by the release and call the private dev service with an authorized identity:
$CommitSha = '<FULL_GIT_COMMIT_SHA>'
$ShortSha = $CommitSha.Substring(0, 7)
$Release = "rel-$ShortSha"
.\scripts\verify-environment.ps1 `
-ProjectId $DevProject `
-Environment dev `
-ExpectedRevision $CommitSha `
-Region $Region
The script checks /healthz, /version, and /summarize. The version response must report the expected Git SHA and dev environment.
Promote the existing release:
gcloud deploy releases promote `
--project $ToolingProject `
--region $Region `
--delivery-pipeline document-insight-release `
--release $Release `
--to-target staging
Cloud Deploy creates a new rollout for the staging target. It does not rerun the Docker build. After rollout success, run the same verification with -ProjectId $StagingProject -Environment staging and confirm that the source revision is unchanged.
Step 7: Request Production Promotion and Apply the Approval Gate
Complete docs/production-approval-checklist.md before opening the production rollout. At minimum, confirm:
The Git commit and Artifact Registry digest are expected.
Repository CI and Cloud Build tests passed.
Dev and staging report the same source revision.
Required AI evaluations passed for the release configuration.
The rendered production manifest diff contains only intended changes.
Runtime identity, secret reference, scaling, monitoring, and rollback ownership are acceptable.
Create the production rollout:
gcloud deploy releases promote `
--project $ToolingProject `
--region $Region `
--delivery-pipeline document-insight-release `
--release $Release `
--to-target prod
Because the target has requireApproval: true, the rollout waits instead of deploying. A principal with roles/clouddeploy.approver can review the rendered manifest diff and approve or reject it. Google documents both the role and the CLI/console workflow in Promote releases and manage approvals.
CLI approval is available when it fits the organization's controlled workflow:
gcloud deploy rollouts approve '<ROLLOUT_NAME>' `
--project $ToolingProject `
--region $Region `
--delivery-pipeline document-insight-release `
--release $Release
Reject when evidence is incomplete. A rejected rollout cannot later be approved; the release must be promoted again after the issue is resolved.
The approval should be performed with an identity distinct from the automated build identity. Whether the same human can merge code and approve production is an organizational separation-of-duties decision, not something Cloud Deploy decides automatically.
Step 8: Verify Production and Rehearse Rollback
After approval and rollout success, verify the production service:
.\scripts\verify-environment.ps1 `
-ProjectId $ProductionProject `
-Environment production `
-ExpectedRevision $CommitSha `
-Region $Region
The successful response should prove four things together:
The private Cloud Run service is reachable by an approved caller.
The runtime reports production, not a copied dev value.
The source revision matches the release promoted through staging.
The application contract still produces the expected response.
Then rehearse the rollback process in a sandbox. Cloud Deploy can create a new rollout from the last known-good release:
gcloud deploy targets rollback prod `
--project $ToolingProject `
--region $Region `
--delivery-pipeline document-insight-release
Use --release '<KNOWN_GOOD_RELEASE>' when the incident decision explicitly names the recovery version. The Cloud Deploy rollback guide notes that rollback creates another rollout; it does not erase history.
Confirm how production approval applies to rollback in your configuration and document who can authorize an emergency restoration. Cloud Run can also move traffic to an earlier revision, but an out-of-band traffic change creates delivery-state drift and must be recorded and reconciled.
Production Considerations
IAM and Separation of Duties
Keep four authorities distinct: code review, build/release creation, deployment execution, and production approval. Grant roles/clouddeploy.approver only to release managers or an integrated approval system. Use IAM conditions, groups, temporary elevation, and audit review where appropriate.
The example execution identity uses predefined Cloud Run and Cloud Deploy roles. A platform team should derive custom roles from observed permissions and scope them to named services, pipelines, secrets, and repositories where supported.
Secret and Configuration Governance
Deploy parameters are not a secret store. Use them for names, environment labels, model identifiers, resource limits, and other reviewable configuration. Keep credentials and sensitive endpoints in Secret Manager, grant the runtime identity access only to the required secret, and test rotation before expiring an old version.
For AI applications, version more than the container. Record the prompt template, model ID, retrieval index or dataset version, policy bundle, evaluation suite, and safety configuration associated with each release.
Reliability and Rollback
A successful deployment only proves that the service reached a healthy platform state. Add application SLOs, latency and error alerts, dependency health, model-provider failure handling, concurrency tests, and post-deployment evaluation. Define objective rollback triggers and an incident owner before the production gate is used.
Consider progressive Cloud Run traffic for high-risk changes. If using Cloud Deploy canary features, document how traffic phases interact with approval and evaluation. Do not assume an automatic rollback policy is safe for every AI behavior regression.
Observability and Auditability
The sample writes structured request metadata, duration, status, request ID, environment, and Cloud Run revision to standard output. It does not log prompt or response bodies. Configure Cloud Logging retention, exclusions, sinks, alerting, and access based on data classification.
Retain the Git review, test output, build provenance, image digest, Cloud Deploy release, rollout history, approval decision, rendered manifest diff, and verification results as one release record.
Cost and Scaling
Cloud Build minutes, Artifact Registry storage and transfer, Cloud Deploy operations, Cloud Run compute and requests, Secret Manager access, and log ingestion or retention can incur cost. The tutorial's production target sets one minimum instance, so it can incur idle cost.
Profile the actual inference path before setting CPU, memory, concurrency, timeout, minimum instances, or maximum instances. A synthetic summarizer says nothing about the capacity required by a hosted model, local model, retrieval system, or tool-using agent.
Clean Up the Tutorial Resources
Retain required evidence before cleanup. Remove resources in dependency-aware order:
Delete the production, staging, and dev Cloud Run services.
Delete Cloud Deploy rollouts and releases when retention policy permits, then delete the targets and delivery pipeline.
Delete each environment's secret and runtime service account.
Delete the Cloud Build trigger and dedicated GitHub connection.
Delete the Artifact Registry repository only when no retained release needs the image.
Delete the build and deploy service accounts and remove cross-project bindings.
Review Cloud Build logs, Cloud Logging buckets, Cloud Deploy source-staging buckets, audit logs, billing exports, and retained images separately.
Do not delete projects or disable APIs that are shared with other workloads. Deleting Artifact Registry images or Cloud Deploy history can remove incident, audit, or rollback evidence.
How Codersarts Can Help
CodersArts can help teams turn a working AI service into a governed delivery platform: container hardening, Cloud Build automation, Artifact Registry policy, Cloud Deploy promotion, Cloud Run environment isolation, Secret Manager integration, IAM design, AI evaluation gates, observability, and rollback runbooks.
Explore another open-source implementation: CodersArts Identity Verification API.
Contact Email: contact@codersarts.com
Conclusion
This design changes the release question from “can we deploy the container?” to “can we prove that the reviewed, tested artifact moved through controlled environments and was authorized for production?”
Cloud Build produces the artifact, Artifact Registry preserves its identity, Cloud Deploy manages promotion, Secret Manager keeps sensitive configuration out of Git, separate projects isolate environments, and the production target enforces a visible decision point. The remaining work for a real AI system is to connect that software evidence to model-quality evaluation, data controls, operational monitoring, and an exercised incident process.
Comments