Skip to content

Deployment and Operations

Production uses the central GitOps model based on mago-onprem-infra. The Skill Runtime repository owns the application code, the Helm chart, container image builds, and the deployed image tag; the infrastructure repository manages the Argo CD Application and per-environment values.

The deployment flow is:

  1. A push to develop, or a manual run, starts the CD (on-prem) workflow.
  2. GitHub Actions pushes the image to GAR tagged <first 12 characters of the commit SHA>.
  3. The workflow updates deploy/values-onprem-image.yaml in this service repository.
  4. Review the changed image tag and manifest diff in Argo CD.
  5. An administrator manually syncs the sohri-skill-runtime-onprem Application.

This separates application and operations permissions, and keeps change approval, auditing, and rollback clear.

Docker Compose

Run from the project root. The host port is 9213 and the container port is 59213.

docker compose up --build

Helm values files

  • deploy/values-dev.yaml: GKE development
  • deploy/values-prd.yaml: GKE production
  • deploy/values-onprem.yaml: on-premises Kubernetes

Render check:

helm lint deploy
helm template sohri-skill-runtime deploy -f deploy/values-dev.yaml
helm template sohri-skill-runtime deploy -f deploy/values-prd.yaml
helm template sohri-skill-runtime deploy -f deploy/values-onprem.yaml
helm template sohri-skill-runtime deploy \
  -f deploy/values-onprem.yaml \
  -f deploy/values-onprem-image.yaml

Install directly only for development or emergency recovery. Production deployments go through Argo CD.

helm upgrade --install sohri-skill-runtime deploy \
  --namespace mago-prod \
  --create-namespace \
  -f deploy/values-onprem.yaml

CI/CD

GitHub Actions

  • .github/workflows/ci.yml: Python, tests, Helm, Docker, secret scan
  • .github/workflows/cd.yml: manual GKE fallback deployment
  • .github/workflows/onprem-cd.yml: on push to develop, pushes the GAR image and updates the service repo's image tag

On-premises prerequisites

The following Secret must be available to the Skill Runtime GitHub repository:

  • GCP_ARTIFACT_REGISTRY_WRITER_KEY: GCP credential allowed to push images to GAR

If you use an Organization Secret, add holamago/sohri-skill-runtime to each Secret's repository access. Never write Secret values into source code, Helm values, or workflow logs.

The Kubernetes mago-prod namespace needs a GAR pull Secret.

kubectl -n mago-prod get secret gcp-artifact-registry

The GitOps repository needs the following:

gitops/apps/sohri-skill-runtime-onprem.yaml
gitops/apps/values/sohri-skill-runtime-onprem/overrides.yaml
gitops/projects/mago-prod.yaml

The mago-prod AppProject's sourceRepos must allow this repository:

https://github.com/holamago/sohri-skill-runtime.git

The Application references both the Skill Runtime deploy chart and the infrastructure repository's values. Start the initial on-premises rollout with 1 replica and HPA/PDB disabled, then scale out once load and node capacity have been checked.

On-premises deployment procedure

  1. Merge the Skill Runtime change into develop and run the CD (on-prem) workflow.
  2. Confirm the workflow updated deploy/values-onprem-image.yaml.
  3. Merge the Application, AppProject, and values changes in the infrastructure repository into main.
  4. In Argo CD, Refresh mago-projects and then mago-root, and sync them manually.
  5. Confirm GAR has an image tag in this format:
asia-northeast3-docker.pkg.dev/mago-services/mago/sohri-skill-runtime:<SHA12>
  1. Confirm the workflow updated this file in the service repository:
deploy/values-onprem-image.yaml
  1. Hard Refresh sohri-skill-runtime-onprem in Argo CD.
  2. Review the image tag and Kubernetes manifest diff, then sync manually.
  3. Check the rollout and API health.
kubectl -n mago-prod rollout status deployment/sohri-skill-runtime
kubectl -n mago-prod get pod,service | grep sohri-skill-runtime
kubectl -n mago-prod port-forward service/sohri-skill-runtime 59213:59213

In another terminal:

curl --fail http://127.0.0.1:59213/health/live
curl --fail http://127.0.0.1:59213/health/ready

Troubleshooting

  • GAR authentication failure: check the repository access of GCP_ARTIFACT_REGISTRY_WRITER_KEY and its Artifact Registry Writer role.
  • Image tag push failure: check the workflow's contents: write permission and the service repository's branch protection rules.
  • Application missing in Argo CD: Refresh mago-root and confirm the Application manifest is on main in the infrastructure repository.
  • ComparisonError: check the AppProject's allowed source repositories and the Application's targetRevision and values paths.
  • ImagePullBackOff: check the gcp-artifact-registry Secret and that the image tag exists.
  • Readiness failure: check the Pod log and the /health/ready response.

Configuration ownership

The service repository manages the chart and image tag; mago-onprem-infra manages the AppProject, the Application, and per-cluster overrides. Argo CD renders both repositories together, and the final deployment is always a manual sync by an administrator.

Running helm upgrade directly from GitHub Actions would need a self-hosted runner that can reach the on-premises cluster, plus Kubernetes credentials. That concentrates production credentials on the CI runner, so it is not recommended as the default deployment method.

GKE fallback

GKE fallback repository variables:

  • AR_REPOSITORY_URL
  • GCP_WORKLOAD_IDENTITY_PROVIDER
  • GCP_DEPLOYER_SERVICE_ACCOUNT

Cloud Build

cloudbuild.yaml uses the same image build/push and per-environment values tag bump contract as Graph Builder. The infrastructure trigger must inject these substitutions:

  • _AR_REPOSITORY_URL
  • _IMAGE_NAME=sohri-skill-runtime
  • _VALUES_FILE
  • _ENV_NAME
  • _GITHUB_REPO_SLUG=holamago/sohri-skill-runtime
  • _GITHUB_PUSH_TOKEN_SECRET_VERSION

Probes

  • /health/live: checks that the process can serve requests
  • /health/ready: checks that the Skill Engine URL is configured
  • timeout: 5–10 seconds
  • replicas: at least 2
  • do not log request bodies

The Runtime never retries or caches execution POSTs, to avoid duplicate side effects.