American Cloud Deployment & CI/CD Guide¶

Golden Age Hub deploys its production workloads onto the secure, U.S.-sovereign American Cloud Kubernetes Service (ACKS). This document outlines our Kubernetes configuration, A2 Object Storage setups, and CI/CD pipelines, mapping the operational steps required to keep the production stack running.

Live Production Endpoints¶

• Main Caregiver Dashboard: https://dashboard.goldenage.americancloud.com/
• API Ingestion Gateway: https://api.goldenage.americancloud.com/
• API Swagger Documentation: https://api.goldenage.americancloud.com/docs

Architecture Overview¶

The deployment architecture consists of: 1. American Cloud Kubernetes Service (ACKS): Orchesrates our API Gateway, Celery Pipeline worker, and Policy service containers. 2. A2 Object Storage (S3-Compatible): HIPAA-compliant, encrypted-at-rest WORM archive for event verification records. 3. GitHub Actions CI/CD Pipeline: Automates building multi-architecture Docker images, provisioning A2 storage resources via Terraform, installing cluster-level ingress/cert-manager controllers, and executing rolling updates in Kubernetes.

Prerequisites¶

1. Cloud Infrastructure¶

Before starting the deployment, ensure the following resources are provisioned on American Cloud: 1. ACKS Cluster: Set up a Kubernetes cluster via the American Cloud CMP portal and download the cluster connection configuration (kube.conf). 2. Managed Database Instances: - Managed PostgreSQL (with TimescaleDB extensions enabled). - Managed Redis (for broker queue traffic). 3. Domain Name: A registered domain (e.g., goldenage.com) configured with a DNS A record pointing api.goldenage.yourdomain.com to the public IP of your Kubernetes Ingress LoadBalancer (which is automatically provisioned during the deployment step).

2. Local CLI Tools¶

For manual deployment, local configuration, and Terraform execution: - kubectl: Kubernetes command line tool. - helm: Kubernetes package manager. - Terraform: Required to provision A2 Storage resources.

Installing Terraform on Ubuntu/Debian (Official APT Repository)¶

# Install prerequisites
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl

# Add the official HashiCorp GPG key
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null

# Add the official HashiCorp repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list

# Install Terraform
sudo apt-get update && sudo apt-get install -y terraform

# Verify installation
terraform -version

Alternatively, you can install Terraform via Snap:

sudo snap install terraform --classic

Installing Helm on Ubuntu/Debian (Official APT Repository)¶

# Install prerequisites
sudo apt-get update && sudo apt-get install -y curl gpg apt-transport-https

# Add the official Helm GPG key
curl -fsSL https://packages.buildkite.com/helm-linux/helm-debian/gpgkey | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null

# Add the official Helm repository
echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://packages.buildkite.com/helm-linux/helm-debian/any/ any main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

# Install Helm
sudo apt-get update && sudo apt-get install -y helm

# Verify installation
helm version

Alternatively, you can install Helm via Snap:

sudo snap install helm --classic

Database Setup: Kubernetes-based PostgreSQL with TimescaleDB¶

The Golden Age Hub ingestion pipeline relies on the TimescaleDB extension for high-performance storage and query capabilities of time-series event data (e.g., vitals and IMU readings).

Rather than hosting PostgreSQL on a dedicated standalone VM, PostgreSQL is deployed directly inside the ACKS Kubernetes cluster under the goldenage namespace as a StatefulSet with persistent volume storage.

1. StatefulSet Orchestration¶

The database resources are managed using postgres-statefulset.yaml, which: - Spins up a single replica running timescale/timescaledb-ha:pg17. - Automatically requests a 10Gi PersistentVolumeClaim for storage persistence at /var/lib/postgresql/data. - Exposes a cluster-internal headless service (postgres-service) resolving to port 5432 for other pods in the namespace.

2. Automatic Database Initialization¶

When the StatefulSet spins up for the first time, it configures: - The database name from ConfigMap (POSTGRES_DB). - The application user (ga_prod_user). - The connection password from Secret (POSTGRES_PASSWORD).

[!NOTE] Automated Schema Provisioning Once the database is ready, the API Gateway startup script (startup_gateway.sh) will automatically execute the Python script setup_timescale.py upon pod initialization.

This script automatically: 1. Converts the events table to a time-partitioned hypertable. 2. Sets up ordering and index optimizations. 3. Enforces an at-rest compression policy (compressing events older than 7 days). 4. Enforces a WORM retention policy (dropping events only after 7 years for HIPAA auditing compliance).

1. Configure GitHub Repository Secrets¶

The GitHub Actions workflow requires access tokens and connection strings to authenticate with American Cloud and configure application runtimes.

Add the following Secrets to your GitHub repository (Settings -> Secrets and variables -> Actions):

2. CI/CD Deployment Flow¶

Deployments are automated through the Deploy to American Cloud workflow.

Automated Trigger¶

The workflow triggers automatically whenever a new version release tag is pushed to the repository:

git tag v1.0.0
git push origin v1.0.0

Manual Trigger¶

You can trigger a deployment manually at any time: 1. Go to the Actions tab in the GitHub repository. 2. Select the Deploy to American Cloud workflow from the left sidebar. 3. Click the Run workflow dropdown, choose the target branch, choose the environment (production or staging), and click Run workflow.

3. Manual Deployment (Optional / Local Dry-Run)¶

If you need to provision resources or deploy manifests manually from your local command-line interface, follow the steps below:

Step A: Provision A2 Storage with Terraform¶

Initialize and apply the Terraform configuration to create the S3-compatible compliance archive bucket:

cd infra/americancloud/terraform
terraform init
terraform apply \
  -var="american_cloud_access_key=$AMERICAN_CLOUD_ACCESS_KEY" \
  -var="american_cloud_secret_key=$AMERICAN_CLOUD_SECRET_KEY"

Step B: Configure Kubernetes Access¶

Ensure kubectl is pointing to the ACKS cluster by setting the KUBECONFIG environment variable:

export KUBECONFIG=/path/to/your/kube.conf
kubectl cluster-info

Step C: Apply ConfigMaps & Secrets¶

Create the goldenage namespace and configure application configuration parameters and credentials:

cd ../k8s

# Create namespace
kubectl apply -f namespace.yaml

# Apply ConfigMaps
kubectl apply -f configmap.yaml

# Create Application Secret
export $(grep -v '^#' ../../.env.prod | xargs) && \
  kubectl create secret generic app-secrets \
  --namespace=goldenage \
  --from-literal=POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
  --from-literal=REDIS_PASSWORD="$REDIS_PASSWORD" \
  --from-literal=AMERICAN_CLOUD_A2_ACCESS_KEY="$AMERICAN_CLOUD_ACCESS_KEY" \
  --from-literal=AMERICAN_CLOUD_A2_SECRET_KEY="$AMERICAN_CLOUD_SECRET_KEY" \
  --from-literal=JWT_SECRET="$JWT_SECRET" \
  --from-literal=API_KEY="$API_KEY" \
  --from-literal=POLICY_ADMIN_TOKEN="$POLICY_ADMIN_TOKEN" \
  --from-literal=HEALTH_API_KEY="$HEALTH_API_KEY"

export KUBECONFIG=../kube.conf
export $(grep -v '^#' ../../.env.prod | xargs) && \
kubectl create secret docker-registry registry-credentials \
  --namespace=goldenage \
  --docker-server=ghcr.io \
  --docker-username="$GITHUB_USERNAME" \
  --docker-password="$GITHUB_PAT"

Step D: Expose the Cluster (Helm Controllers)¶

Install the ingress controller and cert-manager:

# Ingress Controller
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace default --set controller.publishService.enabled=true

# Cert-Manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.12.0 --set installCRDs=true

Push Images to github¶

# 1. API Gateway Service
docker build -t ghcr.io/milesgray/goldenage-hub/gateway:latest -f backend/Dockerfile.gateway backend/
docker push ghcr.io/milesgray/goldenage-hub/gateway:latest

# 2. Celery Pipeline Worker
docker build -t ghcr.io/milesgray/goldenage-hub/pipeline:latest -f backend/Dockerfile.pipeline backend/
docker push ghcr.io/milesgray/goldenage-hub/pipeline:latest

# 3. Policy Service
docker build -t ghcr.io/milesgray/goldenage-hub/policy:latest -f backend/Dockerfile.policy .
docker push ghcr.io/milesgray/goldenage-hub/policy:latest

Step E: Apply Services & Deployments¶

Deploy the microservice resources and network policies:

# Network Boundaries
kubectl apply -f networkpolicy.yaml

# PostgreSQL Database (StatefulSet)
kubectl apply -f postgres-statefulset.yaml
kubectl rollout status statefulset/postgres -n goldenage

# Policy Service
kubectl apply -f policy-service.yaml
kubectl apply -f policy-deployment.yaml

# API Gateway Service
kubectl apply -f gateway-service.yaml
kubectl apply -f gateway-deployment.yaml

# Celery Ingestion pipeline
kubectl apply -f pipeline-deployment.yaml

# Cluster SSL Issuer & Ingress Routing
kubectl apply -f letsencrypt-issuer.yaml
kubectl apply -f ingress.yaml

Step F: Verify Deployment Status¶

Verify that all pods have spun up successfully:

kubectl rollout status statefulset/postgres -n goldenage
kubectl rollout status deployment/policy -n goldenage
kubectl rollout status deployment/gateway -n goldenage
kubectl rollout status deployment/pipeline -n goldenage

kubectl get svc -n default ingress-nginx-controller

4. Automatic Scaling & Kubernetes Best Practices (Celery)¶

The ingestion pipeline Celery workers are configured for production-grade reliability and autoscaling:

A. Celery Worker Autoscaling via KEDA¶

We use KEDA (Kubernetes Event-driven Autoscaling) to scale our Celery workers based on Redis queue depth rather than CPU/Memory. This avoids CPU idle-polling issues and ensures we scale out exactly when workload peaks.

Apply the KEDA configuration using:

kubectl apply -f keda-scaledobject.yaml

B. Graceful Shutdown¶

To prevent dropping tasks mid-execution, we use late-acknowledgement (task_acks_late=True in Celery configuration). Kubernetes is configured with a high terminationGracePeriodSeconds: 90 to allow active tasks to finish processing before receiving SIGKILL.

C. Readiness Probes¶

Our readiness probe uses celery inspect ping -d worker@$(HOSTNAME) so that individual workers are only marked ready when they have established a healthy connection with the Redis broker.

5. Manually Triggering Deployments¶

In addition to auto-deploying when a new tag is pushed (e.g., v1.0.0), you can manually trigger a deployment run to production or staging from your local machine using the provided helper script:

# Deploy the current branch to production (default)
./scripts/trigger-deployment.sh

# Deploy the current branch to staging
./scripts/trigger-deployment.sh --environment staging

# Deploy a specific branch/ref to production
./scripts/trigger-deployment.sh --ref feature-branch

6. Hosting the Documentation Site on Coolify VM¶

The static Zensical documentation site is hosted on a dedicated Coolify Virtual Machine (IP 50.117.53.134) on American Cloud, accessible at https://docs.goldenage.americancloud.com/.

This isolates the static documentation from the core ACKS Kubernetes cluster, reducing cluster load while serving the documentation under automatic Let's Encrypt SSL/TLS.

Prerequisites & DNS Configuration¶

To route traffic to the documentation site: 1. Ensure you have configured an A Record pointing docs.goldenage.americancloud.com to the Coolify VM's public IP: 50.117.53.134. 2. The VM has ports 80, 443, 8000 (Coolify Console), and 22 (SSH) open to all traffic (0.0.0.0/0) via the American Cloud firewall rules.

Deploying the Documentation¶

There are two primary methods to deploy or update the documentation:

Method A: Automated Deployment Script (Recommended)¶

We provide a bash helper script scripts/deploy-docs.sh that pulls the compiled image from the GitHub Container Registry (GHCR) and runs it on the VM's coolify Docker network with Traefik reverse-proxy labels.

1. Ensure the SSH key ~/.ssh/kube-key.pem is present locally.
2. Run the script:

   ./scripts/deploy-docs.sh
   

The script will connect as root@50.117.53.134, stop any existing docs-site container, pull the latest image, and spin it up with the following Traefik labels: * traefik.enable=true * traefik.http.routers.docs-site.rule=Host("docs.goldenage.americancloud.com") * traefik.http.routers.docs-site.entrypoints=https * traefik.http.routers.docs-site.tls=true * traefik.http.routers.docs-site.tls.certresolver=letsencrypt

Traefik (the built-in Coolify proxy) will automatically pick up the container, request a Let's Encrypt certificate, and route HTTPS traffic to the container's port 80.

Method B: Coolify Git Integration¶

Alternatively, you can configure Coolify to pull and build the repository automatically: 1. Log in to the Coolify Dashboard at http://50.117.53.134:8000. 2. Add a new Project and select Application -> Private Repository (authenticated via the repository token). 3. Set the Build Pack to Docker and configure the Dockerfile Path to Dockerfile.docs. 4. Under Domains, input https://docs.goldenage.americancloud.com. 5. Deploy the application. Coolify will build the Docker container and configure Traefik routing automatically.