AutoFix Services — GCP Deployment Reference¶

Purpose. Everything you need to deploy, redeploy, and operate the Mainframe Assist production services on GCP. Open this first for any deploy or incident.

1. What is deployed¶

Mainframe Assist runs as three isolated services on Cloud Run in project mainframe-modernization-495412 (region us-central1). Everything under the autofix-* prefix is one system; nothing else in the project touches it.

Service	Cloud Run name	What it does
Orchestrator	`autofix-orchestrator`	Public FastAPI: lifecycle state machine, plan validation/gating, HITL chat, self-learning KB, cost ledger. Only owner of Firestore + GCS.
Bridge	`autofix-bridge`	Private, `min=max=1`. Only service that reaches z/OSMF (via Zowe CLI). Runs the deterministic executor and the agentic brain (Claude Sonnet 4.5 on Azure AI Foundry).
UI	`lucent-ui`	Next.js dashboard behind the load balancer at `mainframe-assist.itsaiplatform.com`. Login, `/remediations`, `/admin`.

Supporting services still live:

incident-intake — auth backend for the UI (/api/auth/*).
file-upload — API-key-gated CLI upload endpoint.

2. Critical invariants¶

Do not break these without maintainer sign-off.

Invariant	Enforced by
Bridge is the only service with mainframe access	Cloud Run networking + code isolation (`autofix/*` imports nothing from `plan_agents/`, `services/`, `shared/`)
Orchestrator is the sole Firestore + GCS owner	Service accounts, code review
Every plan is HITL-approved before execution	Executor refuses any plan whose hash doesn't match the approved version
Every automated change is reversible	Executor logs the inverse of every resource action; compensates on failure
Per-incident USD ceiling	`autofix_config/loop.usd_ceiling` (default $0.30). Bridge halts to `needs_human` above cap

3. GCP resources (fixed values)¶

Setting	Value
Project ID	`mainframe-modernization-495412`
Region	`us-central1`
App domain	`mainframe-assist.itsaiplatform.com`
Wiki domain	`wiki-mainframe-assist.itsaiplatform.com`
Artifact Registry	`us-central1-docker.pkg.dev/mainframe-modernization-495412/lucent-images`
DNS zone	`mainframe-assist-zone` (in Cloud DNS)
Firestore database	`(default)`
Compute SA	`<project-number>-compute@developer.gserviceaccount.com`

Secret Manager¶

Secret	Purpose	Consumed by
`autofix-azure-api-key`	Anthropic-on-Azure key for the agentic brain	`autofix-bridge` (env `AZURE_API_KEY`)
`azure-api-key`	Azure OpenAI key (legacy plan brain / admin tools)	`incident-intake`, other services
`acs-connection-string`	Azure Communication Services (email)	`incident-intake`
`file-upload-api-key`	Shared Bearer key for CLI uploads	`file-upload` (env `FILE_UPLOAD_API_KEY`)

Never confuse autofix-azure-api-key with azure-api-key — they are different keys serving different systems.

Firestore collections¶

autofix_runs — the run state machine (one doc per remediation).
autofix_kb — self-learning lessons.
autofix_config — admin toggles (brain, learning, loop, spool).
autofix_chat — HITL chat log per run.

GCS buckets (per project prefix)¶

<project>-autofix-incidents/ — captured spool bundles.
<project>-autofix-artifacts/ — execution artifacts (post-fix spools, verify reports).
<project>-cli-uploads/ — CLI/file-upload landing zone.

4. Codebase structure¶

autofix/
  orchestrator/          Cloud Run control plane
    app.py               FastAPI: lifecycle state machine
    plangen.py           Plan generation wrapper
    schema.py            Pydantic Plan/Action/Criterion models
    store.py             Firestore accessors
    kb.py                Self-learning KB engine
    learning_config.py   Admin ON/OFF for learning
    brain_config.py      plan | agent brain selector
    cloudbuild.yaml      Build spec

  bridge/                Cloud Run mainframe-hands service
    app.py               FastAPI mainframe-ops layer (/mf/*)
    executor.py          Deterministic plan applier
    capture.py           Failure watcher, spool bundle builder
    spool_digest.py      Token-budgeted spool reduction
    agent/               Agentic brain (Claude Sonnet 4.5)
      loop.py            Reason-act loop
      tools.py           Read-only mainframe tools
      prompts.py         Vendored taxonomy + fix rules
      skills.py          Skill library (routing hints)
      model.py           Azure AI Foundry client
    cloudbuild.yaml      Build spec

  ui_cloudbuild.yaml     Build spec for lucent-ui

services/
  incident_intake/       Auth API (register/login/sessions/admin)

ui/                      Next.js 15 app (App Router)

Deploy scripts (scripts/_autofix_*.sh) live locally / in CI, not in the repo.

5. Deployment flow¶

5.1 Prerequisites (one-time, per developer)¶

# WSL Ubuntu recommended (documented setup path)
export PATH="$HOME/google-cloud-sdk/bin:$PATH"

gcloud auth login
gcloud auth application-default login
gcloud config set project mainframe-modernization-495412

# Sanity
gcloud config get-value project           # → mainframe-modernization-495412
gcloud auth list                          # one account with *
gcloud auth application-default print-access-token >/dev/null

Docker is not required — all image builds go through Cloud Build.

5.2 Redeploy a service¶

Every service has its own Cloud Build spec:

Change	Redeploy
`autofix/orchestrator/**`	`autofix-orchestrator` (`autofix/orchestrator/cloudbuild.yaml`)
`autofix/bridge/` (including `autofix/bridge/agent/`)	`autofix-bridge` (`autofix/bridge/cloudbuild.yaml`)
`ui/**`	`lucent-ui` (`autofix/ui_cloudbuild.yaml`)
`services/incident_intake/**`	`incident-intake`
`shared/**` broadly consumed	Everything that imports it

Recommended workflow:

Edit code.
Run the matching scripts/_autofix_*.sh script (or gcloud builds submit --config=<cloudbuild.yaml>).
Watch build output — each service prints SUCCESS on completion.
Run the smoke checks in §7.

5.3 First-time provisioning¶

If you ever need to bring up a fresh stack in a new GCP project:

Enable APIs: Vertex AI, Cloud Run, Cloud Build, Artifact Registry, Firestore, Secret Manager, Cloud DNS, Compute, Cloud Logging.
Create the Artifact Registry: us-central1-docker.pkg.dev/<PROJECT>/lucent-images.
Create the three GCS buckets listed in §3.
Create Firestore in Native mode (default DB).
Populate Secret Manager with the secrets in §3.
Reserve a global static IP + create the URL map/backend/SSL cert for the app domain (same pattern as wiki deploy).
Bind Cloud Run identity to roles/secretmanager.secretAccessor and roles/datastore.user.
Deploy each service (§5.2).

6. Common operations¶

6.1 Add or remove an admin¶

Self-service (preferred): sign in as an existing admin → /admin → Admin access → grant. Firestore is the source of truth, ~30s TTL cache.
Bootstrap (no admin exists yet): set the ADMIN_EMAILS env var on incident-intake. Comma-delimited values need the ^|^ gcloud delimiter:

gcloud run services update incident-intake \
  --region=us-central1 --project=mainframe-modernization-495412 \
  --update-env-vars="^|^ADMIN_EMAILS=<admin1@example.com>,<admin2@example.com>" \
  --quiet

6.2 Toggle the agentic brain¶

Admin UI: /admin/models → Brain → plan or agent. Backed by Firestore autofix_config/brain, ~30s TTL. Also requires AUTOFIX_AGENT_ENABLED=1 on the bridge:

gcloud run services update autofix-bridge \
  --region=us-central1 --project=mainframe-modernization-495412 \
  --update-env-vars="AUTOFIX_AGENT_ENABLED=1" --quiet

Revert: {"brain":"plan"} + AUTOFIX_AGENT_ENABLED=0.

6.3 Toggle self-learning¶

Admin UI: /admin/models → Self-learning & Knowledge Base → ON/OFF. Firestore autofix_config/learning, default OFF. OFF → agent prompt is byte-identical to pre-learning behaviour.

6.4 Tune the per-incident USD ceiling¶

# Read
gcloud firestore documents get autofix_config/loop \
  --project=mainframe-modernization-495412

# Update via admin UI or POST /admin/loop-config {"usd_ceiling": 0.50}

Default 0.30. Bridge halts to needs_human once cumulative LLM cost on a run crosses this.

6.5 Rotate a secret¶

# Add a new version
printf "%s" "<NEW_VALUE>" | gcloud secrets versions add <secret-name> \
  --data-file=- --project=mainframe-modernization-495412

# Force a new revision so the service picks up :latest
gcloud run services update <service> --region=us-central1 \
  --project=mainframe-modernization-495412 \
  --update-env-vars="ROTATION_NONCE=$(date +%s)" --quiet

# Destroy the old version once all consumers are on the new one
gcloud secrets versions destroy <OLD_VERSION> \
  --secret=<secret-name> --project=mainframe-modernization-495412

6.6 Rollback¶

# List recent revisions
gcloud run revisions list --service=<service> --region=us-central1 \
  --project=mainframe-modernization-495412 --limit=5

# Route 100% traffic to a prior revision
gcloud run services update-traffic <service> --region=us-central1 \
  --to-revisions=<old-revision>=100 \
  --project=mainframe-modernization-495412

6.7 Update env additively¶

# Simple value
gcloud run services update <service> --region=us-central1 \
  --update-env-vars "KEY=VALUE" --project=mainframe-modernization-495412 --quiet

# Value with commas (e.g. ADMIN_EMAILS list)
gcloud run services update <service> --region=us-central1 \
  --update-env-vars="^|^KEY=v1,v2,v3" \
  --project=mainframe-modernization-495412 --quiet

# Attach a secret
gcloud run services update <service> --region=us-central1 \
  --update-secrets "ENV_NAME=<secret-name>:latest" \
  --project=mainframe-modernization-495412 --quiet

Never use --set-env-vars unless you intend to wipe every other env var.

7. Smoke test after deploy¶

# 1. All services Ready
for s in autofix-orchestrator autofix-bridge lucent-ui incident-intake file-upload; do
  printf "%-22s " "$s"
  gcloud run services describe $s --region=us-central1 \
    --project=mainframe-modernization-495412 \
    --format="value(status.latestReadyRevisionName,status.conditions[0].status)"
done

# 2. Error scan (last 30 min)
gcloud logging read \
  'resource.type=cloud_run_revision AND severity>=ERROR' \
  --project=mainframe-modernization-495412 --limit=20 \
  --format="value(timestamp,resource.labels.service_name,textPayload)" \
  --freshness=30m

# 3. Auth gate (should return 401 unauthenticated)
curl -sS -o /dev/null -w "%{http_code}\n" \
  https://mainframe-assist.itsaiplatform.com/api/auth/me

Then run one interactive failure through the UI to prove end-to-end.

8. Gotchas & lessons learned¶

Gotcha	Why	Fix
`--update-env-vars "KEY=v1,v2"` fails	Comma is the pair separator	Use `--update-env-vars="^\|^KEY=v1,v2"`
`--set-env-vars` wiped `ACS_CONNECTION_STRING` once	Set replaces, not merges	Always use `--update-env-vars`
Foreground `gcloud run deploy` hits a tool timeout wall	10-minute limit on our async tool	Run as background or use CI
Bridge deploy suffix collides on same image digest	Deterministic revision name	Add a unique suffix or bump env
Concurrent gcloud builds serialize on the cred lock	Single credential per host	Deploy services sequentially
Foreign HLQ writes rejected on Z Xplore	ZXPACS ACS policy	Executor's `_foreign_hlq` guard enforces owner-HLQ scope
Small student spool evicts old jobs fast	Limited spool space	Capture immediately after failure

9. Where to look first¶

If you want to…	Read
Change plan validation rules	`autofix/orchestrator/plangen.py` + `schema.py`
Add or edit a fix skill	`autofix/bridge/agent/skills.py` and `prompts.py`
Add a mainframe tool the agent can call	`autofix/bridge/agent/tools.py` + `tool_schemas.py`
Change the executor's action vocabulary	`autofix/bridge/executor.py`
Tune the spool reducer	`autofix/bridge/spool_digest.py` + `autofix_config/spool`
Change UI state polling / cards	`ui/app/remediations/`, `ui/components/`
Add an admin toggle	`autofix/orchestrator/_config.py` + `ui/app/admin/*`

Redeploy checklist: - [ ] Code review passed - [ ] Local tests green - [ ] Build succeeds - [ ] Smoke checks green (§7) - [ ] Rollback recipe (§6.6) confirmed for the changed service