type: create

description:
Currently laconic-so maps one stack to one deployment to one pod. All containers
in a stack's compose files become containers in a single k8s pod. This means:

- Can't upgrade doublezero without restarting agave-validator
- Can't restart monitoring without disrupting the validator
- Can't independently scale or lifecycle-manage components

The fix is stack composition. A meta-stack (e.g. biscayne-stack) composes
sub-stacks (agave, doublezero, agave-monitoring), each becoming its own
k8s Deployment with independent lifecycle.

title: Stack composition: deploy multiple stacks into one kind cluster

type: epic


description: Add laconic-so deployment prepare that creates cluster infrastructure without pods. Already implemented, needs review.

title: deployment prepare command

type: task


description: deployment stop on ANY deployment deletes the shared kind cluster. Should only delete its own namespace.

title: deployment stop should not destroy shared cluster

type: bug


description: Allow stack.yml to reference sub-stacks. Each sub-stack becomes its own k8s Deployment sharing namespace and PVs.

title: Add stacks: field to stack.yml for composition


description: Create three independent stacks from the monolithic agave-stack. Each gets its own compose file and independent lifecycle.

title: Split agave-stack into agave + doublezero + monitoring


title: deploy create should support external credential injection

type: feature

description: deploy create generates config.env but provides no mechanism to inject external credentials (API keys, tokens, etc.) at creation time. Operators must append to config.env after the fact, which mutates a build artifact. deploy create should accept --credentials-file or similar to include secrets in the generated config.env.


title: REGISTRY_TOKEN / imagePullSecret flow undocumented

description: create_registry_secret() exists in deployment_create.py and is called during up(), but REGISTRY_TOKEN is not documented in spec.yml or any user-facing docs. The restart command warns "Registry token env var REGISTRY_TOKEN not set, skipping registry secret" but doesn't explain how to set it. For GHCR private images, this is required and the flow from spec.yml -> config.env -> imagePullSecret needs documentation.


title: Stack path resolution differs between deploy create and deployment restart

description: deploy create resolves --stack as a relative path from cwd. deployment restart resolves --stack-path as absolute, then computes repo_root as 4 parents up (assuming stack_orchestrator/data/stacks/name structure). External stacks with different nesting depths (e.g. stack-orchestrator/stacks/name = 3 levels) get wrong repo_root, causing --spec-file resolution to fail. The two commands should use the same path resolution logic.


title: deployment restart should update in place, not delete/recreate

description:
deployment restart deletes the entire namespace then recreates everything from scratch. This causes:

1. **Downtime** — nothing serves traffic between delete and successful recreate
2. **No rollback** — deleting the namespace destroys ReplicaSet revision history
3. **Race conditions** — namespace may still be terminating when up() tries to create
4. **Cascading failures** — if ANY container fails to start, the entire site is down with no fallback

Fix: three changes needed.

**A. up() should create-or-update, not just create.** Use patch/apply semantics for Deployments, Services, Ingresses. When the pod spec changes (new env vars, new image), k8s creates a new ReplicaSet, scales it up, waits for readiness probes, then scales the old one down. Old pods serve traffic until new pods are healthy.

**B. down() should never delete the namespace on restart.** Only on explicit teardown. The namespace owns the revision history. Current code: _delete_namespace() on every down(). Should: delete individual resources by label for teardown, do nothing for restart (let update-in-place handle it).

**C. All containers need readiness probes.** Without them k8s considers pods ready immediately, defeating rolling update safety. laconic-so should generate readiness probes from the http-proxy routes in spec.yml (if a container has an http route, probe that port).

With these changes, k8s native rolling updates provide zero-downtime deploys and automatic rollback (if new pods fail readiness, rollout stalls, old pods keep serving).

Source files:
- deploy_k8s.py: up(), down(), _create_deployment(), _delete_namespace()
- cluster_info.py: pod spec generation (needs readiness probes)
- deployment.py: restart() orchestration


title: Add credentials-files spec key for on-disk credential injection

description:
deployment restart regenerates config.env from spec.yml, wiping credentials that were appended from on-disk files (e.g. ~/.credentials/*.env). Operators must append credentials after deploy create, which is fragile and breaks on restart.

Fix: New top-level spec key credentials-files. _write_config_file() reads each file and appends its contents to config.env after writing config vars. Files are read at deploy time from the deployment host.

Spec syntax:
  credentials-files:
    - ~/.credentials/dumpster-secrets.env
    - ~/.credentials/dumpster-r2.env

Files:
- deploy/spec.py: add get_credentials_files() returning list of paths
- deploy/deployment_create.py: in _write_config_file(), after writing config vars, read and append each credentials file (expand ~ to home dir)

Also update dumpster-stack spec.yml to use the new key and remove the ansible credential append workaround from woodburn_deployer (group_vars/all.yml credentials_env_files, stack_deploy role append tasks, restart_dumpster.yml credential steps). Those cleanups are in the woodburn_deployer repo.


title: Merge kind-mount-propagation branch — HostToContainer propagation for extraMounts

description: The kind-mount-root feature was cherry-picked to main (commit 8d03083d) but the mount propagation fix (commit 929bdab8 on branch enya-ac868cc4-kind-mount-propagation-fix) adds HostToContainer propagation so host submounts propagate into the Kind node. This is needed for ZFS child datasets and tmpfs mounts under the root. Cherry-pick 929bdab8 to main.


title: etcd cert backup not persisting across cluster deletion

description: The extraMount for etcd at data/cluster-backups/<id>/etcd is configured but after cluster deletion the directory is empty. Caddy TLS certificates stored in etcd are lost. Either etcd isn't writing to the host mount, or the cleanup code is deleting the backup. Investigate _clean_etcd_keeping_certs in helpers.py.


title: laconic-so should manage Caddy ingress image lifecycle

description: The Caddy ingress controller image is hardcoded in ingress-caddy-kind-deploy.yaml. There's no mechanism to update it without manual kubectl commands or cluster recreation. laconic-so should: 1) Allow spec.yml to specify a custom Caddy image, 2) Support updating the Caddy image as part of deployment restart, 3) Set strategy: Recreate on the Caddy Deployment (hostPort pods can't do RollingUpdate). This would let cryovial or similar tooling trigger Caddy updates through the normal deployment pipeline.


description:
The DockerDeployer.up() in stack_orchestrator/deploy/compose/deploy_docker.py accepts image_overrides as a parameter but silently drops it — only k8s mode (deploy_k8s.py) actually applies overrides.

Impact: the --image container=image CLI flag on 'laconic-so deployment start' is a no-op for compose-mode deployments. Spec-level image-overrides: keys are also ignored in compose mode (they reach up() via deployment.py but are never applied).

Use case: gorchain-stacks test scripts build :local images via build-containers, but compose files reference ghcr.io/gorbagana-dev/*:latest (so prod pulls work). Without image override support in compose mode, tests either need to docker tag the builds or the compose file needs to be rewritten before start — both ugly workarounds for what should be a first-class mechanism.

Fix sketch: in DockerDeployer.up(), when image_overrides is non-empty, write a temporary docker-compose.override.yml with {services: {name: {image: ref}}} and construct a new DockerClient with compose_files + [override_path]. Keeps k8s path untouched, reuses existing --image CLI flag and spec-level image-overrides: plumbing.

title: Compose deployer ignores image_overrides