Agentic ERP: deployment infrastructure
The pipeline that puts the copilot on live infrastructure, plus the 6 checks that have to pass before any release reaches a user.
Two-repo CI/CD architecture
The development repo runs a 6-job GitHub Actions pipeline on every push to master: backend quality gates (ruff, black, mypy, bandit, pytest unit), frontend tsc, integration tests, a red-team job (promptfoo and pytest), and a build that pushes the API and frontend images to Azure Container Registry. A repository_dispatch event then fires this deployment repo.
This deployment repo picks up the dispatch and runs a deploy job that waits for manual approval through a GitHub environment. Once approved, the job applies the manifests with kubectl apply -k, sets the new image tags, and waits for the rollout. The 6-check smoke test then runs as its own job. The same workflow can also be started by hand with explicit image tags. Keeping deployment in its own repo draws the production boundary where a reviewer can see it: no commit reaches production without a second pair of eyes.
Infrastructure as code: Bicep modules
| Module | Detail |
|---|---|
| AKS | Kubernetes 1.31 on a system pool of Standard_B2s nodes (2 in the production parameters), with Container Insights on. The OIDC issuer and Workload Identity are enabled on the cluster, and an AcrPull role lets the nodes pull images without a registry password. |
| Azure Container Registry | Basic SKU. The CI build pushes the API and frontend images here, tagged with the commit. |
| Postgres (Flexible Server) | Burstable Standard_B1ms, holding the ERP data and the pgvector contract embeddings. A firewall rule admits Azure services. |
| Redis Cache | Basic C0, TLS 1.2 only, with LRU eviction. Backs the semantic cache and the human-gate decision records (24-hour TTL). |
| Cosmos DB | Gremlin API on the free tier, meant for the knowledge graph. East US had no capacity at deployment time, so the graph runs on Neo4j 5.26 inside the cluster, on a 5 Gi volume. |
| Azure Monitor / App Insights | A Log Analytics workspace and Application Insights, fed by the OTel DaemonSet. Alert rules live in alerts.bicep. |
| Azure AI Search | Free tier (50 MB, 3 indexes). |
Kustomize overlays
One base set of Kubernetes manifests is shared across environments: Deployments and Services for the API, the frontend, and Neo4j, a ConfigMap, the ingress, and a job that seeds the graph. The overlays patch only what differs. Staging adds a name prefix, runs 1 API replica, and sets its own host and log level. Production sets its host, log level, and CORS origin. No manifest is duplicated. The deploy workflow applies the base today.
The NGINX ingress controller routes /api, /ws, /docs, /openapi.json, and /health to the FastAPI backend, and / to the frontend. The ingress proxies over HTTP/1.1 with a 1-hour read timeout, so the WebSocket that carries the approval banner can upgrade and stay open.
Observability: the OTel DaemonSet
An OpenTelemetry Collector DaemonSet runs on every AKS node, receives OTLP traces and metrics from the application pods over gRPC or HTTP, and forwards them to Azure Monitor through the azuremonitor exporter. A dashboard definition in the repo charts request rate, error rate, P95 latency, LangGraph node latency by intent, solver cache hit rate, and pending human approvals.
Smoke test: 6 checks
After kubectl rollout status reports healthy, the smoke-test.sh script runs 6 checks in sequence and fails the job if any of them fails:
- Health endpoint. GET /health returns 200.
- OpenAPI docs. GET /docs returns 200, which confirms FastAPI started correctly.
- WebSocket reachability. A WebSocket upgrade request to /ws/chat gets an answer (101, 400, 403, or 404) rather than a timeout.
- Human-gate router. GET /api/approve/ with an unknown decision ID returns 404 rather than 500, which shows the approval route is wired.
- Injection probe. A POST to the approval endpoint carries a prompt injection asking for database credentials, and the response must contain none of password, secret, api_key, token, or credential.
- Frontend reachability. GET / returns 200 from the frontend.
Security posture
The workflows read their Azure access from GitHub secrets, and the AKS nodes pull images through an AcrPull role instead of a registry password. Workload Identity is enabled on the cluster, but the pods still read service credentials from a Kubernetes secret, a gap the repo's deviation log records. Redis accepts TLS only. Postgres admits Azure services through a firewall rule, and the data services keep public network access on, so private endpoints are the next step. The demo ran over plain HTTP on a nip.io hostname, and TLS through cert-manager is still to be added.