Skip to content

Deployment (k8s + GitLab CI)

Production layout mirrors SoccerPredictAI: platform manifests (cae-platform) + app Helm chart (cae-copilot).

Domains

Surface URL
UI https://cae-copilot.dmitryivanov.dev
API (Swagger) https://api.cae-copilot.dmitryivanov.dev/docs
API base https://api.cae-copilot.dmitryivanov.dev
Docs https://docs.cae-copilot.dmitryivanov.dev (GitLab Pages)

Namespaces

Namespace Contents
cae-platform Qdrant (PVC, internal Service)
cae-copilot FastAPI, Streamlit, Ingress, Secrets

Secrets workflow

cp .env.example .env          # fill CAEC_LLM_API_KEY etc.
make gen-secrets                # .env → values-dev.yaml
make encrypt                    # SOPS encrypt .env + k8s secrets
git commit values-dev.enc.yaml .env.enc ...

Required GitLab CI variables (same server as soccer):

  • SERVER_SSH_HOSTNAME, SERVER_SSH_USER, SERVER_SSH_PORT, SERVER_SSH_PRIVATE_KEY
  • SSH_KNOWN_HOSTS, AGE_KEY

CI/CD stages

  1. linting — ruff, helm lint
  2. test — pytest unit
  3. build — push api, streamlit images (:$CI_COMMIT_SHA)
  4. deploy — kubectl apply Qdrant; helm upgrade chart
  5. deploy-images — bump image tags only
  6. pages — MkDocs → GitLab Pages

Deploy runs only on default branch, never from MR pipelines.

Qdrant storage (healserver)

The cluster has no default StorageClass. Qdrant uses storageClassName: manual with a hostPath PV — same pattern as SoccerPredictAI (k8s/manifests/ns_ds/).

Before first apply:

sudo mkdir -p /mnt/nextcloud_data/cae_platform/qdrant
sudo chown -R 1000:1000 /mnt/nextcloud_data/cae_platform/qdrant
kubectl apply -f k8s/manifests/ns_cae-platform/
kubectl get pvc -n cae-platform   # qdrant-data → Bound
kubectl get pods -n cae-platform    # qdrant → Running

If Qdrant stays Pending, check: kubectl describe pod -n cae-platform -l app=qdrant and kubectl describe pvc qdrant-data -n cae-platform.

Embedding model (offline prod)

The API image bakes intfloat/multilingual-e5-base into /app/models/e5-base at Docker build time (scripts/bake_embedding_model.py). Pods set CAEC_EMBEDDING_MODEL=/app/models/e5-base and HF_HUB_OFFLINE=1 — no HuggingFace egress at runtime.

If ingest/RAG fails with No modules.json found or LocalEntryNotFoundError, the running image is stale (hub cache only, not a saved bundle) — rebuild image-api and redeploy.

Workaround before rebuild — ingest from a dev machine via port-forward (host has internet; cluster Qdrant is reached locally):

# terminal 1 — tunnel to healserver Qdrant
ssh -L 6333:127.0.0.1:6333 healserver \
  "kubectl port-forward -n cae-platform svc/qdrant 6333:6333"

# terminal 2 — local mamba env (Rocky manifest ingest)
mamba activate cae-copilot
cp data/cae/ansys_rocky/v26r1/docs/manifest.example.yaml \
   data/cae/ansys_rocky/v26r1/docs/manifest.yaml
CAEC_QDRANT_URL=http://127.0.0.1:6333 pdm run python scripts/ingest.py \
  --manifest data/cae/ansys_rocky/v26r1/docs/manifest.yaml

First production deploy

# 1. Secrets
make gen-secrets && make encrypt && git push

# 2. GitLab pipeline on master builds images + deploys

# 3. Seed Qdrant (once per fresh PVC; requires API image with baked embeddings)
# Option A — Helm ingest Job (recommended; Rocky PDFs on kb-data volume):
kubectl delete job -n cae-copilot -l app.kubernetes.io/component=ingest 2>/dev/null || true
helm upgrade cae-copilot ~/k8s/helm/ns_cae-copilot -n cae-copilot \
  -f ~/k8s/helm/ns_cae-copilot/values.yaml \
  -f ~/k8s/helm/ns_cae-copilot/values-dev.yaml \
  --set ingest.enabled=true --set ingest.runId=2 --wait

# Option B — one-off exec (manifest on mounted kb-data volume):
kubectl exec -n cae-copilot deploy/fastapi -- \
  python scripts/ingest.py --manifest /app/data/cae/ansys_rocky/v26r1/docs/manifest.yaml

# 4. Smoke
SMOKE_BASE_URL=https://api.cae-copilot.dmitryivanov.dev make smoke

Local parity

make dev-stack
make ingest
make smoke

Manual Helm (on server)

helm upgrade --install cae-copilot ~/k8s/helm/ns_cae-copilot \
  --namespace cae-copilot --create-namespace \
  -f ~/k8s/helm/ns_cae-copilot/values.yaml \
  -f ~/k8s/helm/ns_cae-copilot/values-dev.yaml

Troubleshooting

Symptom Likely cause Fix
/ask returns 503 CAEC_DEMO_ENABLED=false Set to true in secrets / Helm values
RAG empty, low confidence Qdrant not seeded Run Helm ingest job; verify GET /kb/stats shows chunks > 0
No modules.json found API image missing baked embeddings Rebuild and redeploy API image (see Embedding model)
Ingest job fails Missing PDFs or manifest.yaml on kb-data volume Check kb.hostPath and Nextcloud sync path
Rate limit errors Ingress or app middleware Adjust nginx ingress annotations or CAEC_RATE_LIMIT_PER_MINUTE