14 changed files with 151 additions and 1469 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -8,7 +8,6 @@ NEVER assume your hypotheses are true without evidence
 ALWAYS clearly state when something is a hypothesis
 ALWAYS use evidence from the systems your interacting with to support your claims and hypotheses
 ALWAYS run `pre-commit run --all-files` before committing changes
 ## Key Principles
@ -44,76 +43,6 @@ This project follows principles inspired by literate programming, where developm
 This approach treats the human-AI collaboration as a form of **conversational literate programming** where understanding emerges through dialogue before code implementation.
 ## External Stacks Preferred
 When creating new stacks for any reason, **use the external stack pattern** rather than adding stacks directly to this repository.
 External stacks follow this structure:
 ```
 my-stack/
 └── stack-orchestrator/
    ├── stacks/
    │   └── my-stack/
    │       ├── stack.yml
    │       └── README.md
    ├── compose/
    │   └── docker-compose-my-stack.yml
    └── config/
        └── my-stack/
            └── (config files)
 ```
 ### Usage
 ```bash
 # Fetch external stack
 laconic-so fetch-stack github.com/org/my-stack
 # Use external stack
 STACK_PATH=~/cerc/my-stack/stack-orchestrator/stacks/my-stack
 laconic-so --stack $STACK_PATH deploy init --output spec.yml
 laconic-so --stack $STACK_PATH deploy create --spec-file spec.yml --deployment-dir deployment
 laconic-so deployment --dir deployment start
 ```
 ### Examples
 - `zenith-karma-stack` - Karma watcher deployment
 - `urbit-stack` - Fake Urbit ship for testing
 - `zenith-desk-stack` - Desk deployment stack
 ## Architecture: k8s-kind Deployments
 ### One Cluster Per Host
 One Kind cluster per host by design. Never request or expect separate clusters.
 - `create_cluster()` in `helpers.py` reuses any existing cluster
 - `cluster-id` in deployment.yml is an identifier, not a cluster request
 - All deployments share: ingress controller, etcd, certificates
 ### Stack Resolution
 - External stacks detected via `Path(stack).exists()` in `util.py`
 - Config/compose resolution: external path first, then internal fallback
 - External path structure: `stack_orchestrator/data/stacks/<name>/stack.yml`
 ### Secret Generation Implementation
 - `GENERATE_TOKEN_PATTERN` in `deployment_create.py` matches `$generate:type:length$`
 - `_generate_and_store_secrets()` creates K8s Secret
 - `cluster_info.py` adds `envFrom` with `secretRef` to containers
 - Non-secret config written to `config.env`
 ### Repository Cloning
 `setup-repositories --git-ssh` clones repos defined in stack.yml's `repos:` field. Requires SSH agent.
 ### Key Files (for codebase navigation)
 - `repos/setup_repositories.py`: `setup-repositories` command (git clone)
 - `deployment_create.py`: `deploy create` command, secret generation
 - `deployment.py`: `deployment start/stop/restart` commands
 - `deploy_k8s.py`: K8s deployer, cluster management calls
 - `helpers.py`: `create_cluster()`, etcd cleanup, kind operations
 - `cluster_info.py`: K8s resource generation (Deployment, Service, Ingress)
 ## Insights and Observations
 ### Design Principles
--- a/README.md
+++ b/README.md
@ -71,59 +71,6 @@ The various [stacks](/stack_orchestrator/data/stacks) each contain instructions
 - [laconicd with console and CLI](stack_orchestrator/data/stacks/fixturenet-laconic-loaded)
 - [kubo (IPFS)](stack_orchestrator/data/stacks/kubo)
 ## Deployment Types
 - **compose**: Docker Compose on local machine
 - **k8s**: External Kubernetes cluster (requires kubeconfig)
 - **k8s-kind**: Local Kubernetes via Kind - one cluster per host, shared by all deployments
 ## External Stacks
 Stacks can live in external git repositories. Required structure:
 ```
 <repo>/
  stack_orchestrator/data/
    stacks/<stack-name>/stack.yml
    compose/docker-compose-<pod-name>.yml
  deployment/spec.yml
 ```
 ## Deployment Commands
 ```bash
 # Create deployment from spec
 laconic-so --stack <path> deploy create --spec-file <spec.yml> --deployment-dir <dir>
 # Start (creates cluster on first run)
 laconic-so deployment --dir <dir> start
 # GitOps restart (git pull + redeploy, preserves data)
 laconic-so deployment --dir <dir> restart
 # Stop
 laconic-so deployment --dir <dir> stop
 ```
 ## spec.yml Reference
 ```yaml
 stack: stack-name-or-path
 deploy-to: k8s-kind
 network:
  http-proxy:
    - host-name: app.example.com
      routes:
        - path: /
          proxy-to: service-name:port
  acme-email: admin@example.com
 config:
  ENV_VAR: value
  SECRET_VAR: $generate:hex:32$   # Auto-generated, stored in K8s Secret
 volumes:
  volume-name:
 ```
 ## Contributing
 See the [CONTRIBUTING.md](/docs/CONTRIBUTING.md) for developer mode install.
--- a/TODO.md
+++ b/TODO.md
@ -7,25 +7,6 @@ We need an "update stack" command in stack orchestrator and cleaner documentatio
 **Context**: Currently, `deploy init` generates a spec file and `deploy create` creates a deployment directory. The `deployment update` command (added by Thomas Lackey) only syncs env vars and restarts - it doesn't regenerate configurations. There's a gap in the workflow for updating stack configurations after initial deployment.
 ## Bugs
 ### `deploy create` doesn't auto-generate volume mappings for new pods
 When a new pod is added to `stack.yml` (e.g. `monitoring`), `deploy create`
 does not generate default host path mappings in spec.yml for the new pod's
 volumes. The deployment then fails at scheduling because the PVCs don't exist.
 **Expected**: `deploy create` enumerates all volumes from all compose files
 in the stack and generates default host paths for any that aren't already
 mapped in the spec.yml `volumes:` section.
 **Actual**: Only volumes already in spec.yml get PVs. New volumes are silently
 missing, causing `FailedScheduling: persistentvolumeclaim not found`.
 **Workaround**: Manually add volume entries to spec.yml and create host dirs.
 **Files**: `deployment_create.py` (`_write_config_file`, volume handling)
 ## Architecture Refactoring
 ### Separate Deployer from Stack Orchestrator CLI
--- a/docs/deployment_patterns.md
+++ b/docs/deployment_patterns.md
@ -1,202 +0,0 @@
 # Deployment Patterns
 ## GitOps Pattern
 For production deployments, we recommend a GitOps approach where your deployment configuration is tracked in version control.
 ### Overview
 - **spec.yml is your source of truth**: Maintain it in your operator repository
 - **Don't regenerate on every restart**: Run `deploy init` once, then customize and commit
 - **Use restart for updates**: The restart command respects your git-tracked spec.yml
 ### Workflow
 1. **Initial setup**: Run `deploy init` once to generate a spec.yml template
 2. **Customize and commit**: Edit spec.yml with your configuration (hostnames, resources, etc.) and commit to your operator repo
 3. **Deploy from git**: Use the committed spec.yml for deployments
 4. **Update via git**: Make changes in git, then restart to apply
 ```bash
 # Initial setup (run once)
 laconic-so --stack my-stack deploy init --output spec.yml
 # Customize for your environment
 vim spec.yml  # Set hostname, resources, etc.
 # Commit to your operator repository
 git add spec.yml
 git commit -m "Add my-stack deployment configuration"
 git push
 # On deployment server: deploy from git-tracked spec
 laconic-so deploy create \
  --spec-file /path/to/operator-repo/spec.yml \
  --deployment-dir my-deployment
 laconic-so deployment --dir my-deployment start
 ```
 ### Updating Deployments
 When you need to update a deployment:
 ```bash
 # 1. Make changes in your operator repo
 vim /path/to/operator-repo/spec.yml
 git commit -am "Update configuration"
 git push
 # 2. On deployment server: pull and restart
 cd /path/to/operator-repo && git pull
 laconic-so deployment --dir my-deployment restart
 ```
 The `restart` command:
 - Pulls latest code from the stack repository
 - Uses your git-tracked spec.yml (does NOT regenerate from defaults)
 - Syncs the deployment directory
 - Restarts services
 ### Anti-patterns
 **Don't do this:**
 ```bash
 # BAD: Regenerating spec on every deployment
 laconic-so --stack my-stack deploy init --output spec.yml
 laconic-so deploy create --spec-file spec.yml ...
 ```
 This overwrites your customizations with defaults from the stack's `commands.py`.
 **Do this instead:**
 ```bash
 # GOOD: Use your git-tracked spec
 git pull  # Get latest spec.yml from your operator repo
 laconic-so deployment --dir my-deployment restart
 ```
 ## Private Registry Authentication
 For deployments using images from private container registries (e.g., GitHub Container Registry), configure authentication in your spec.yml:
 ### Configuration
 Add a `registry-credentials` section to your spec.yml:
 ```yaml
 registry-credentials:
  server: ghcr.io
  username: your-org-or-username
  token-env: REGISTRY_TOKEN
 ```
 **Fields:**
 - `server`: The registry hostname (e.g., `ghcr.io`, `docker.io`, `gcr.io`)
 - `username`: Registry username (for GHCR, use your GitHub username or org name)
 - `token-env`: Name of the environment variable containing your API token/PAT
 ### Token Environment Variable
 The `token-env` pattern keeps credentials out of version control. Set the environment variable when running `deployment start`:
 ```bash
 export REGISTRY_TOKEN="your-personal-access-token"
 laconic-so deployment --dir my-deployment start
 ```
 For GHCR, create a Personal Access Token (PAT) with `read:packages` scope.
 ### Ansible Integration
 When using Ansible for deployments, pass the token from a credentials file:
 ```yaml
 - name: Start deployment
  ansible.builtin.command:
    cmd: laconic-so deployment --dir {{ deployment_dir }} start
  environment:
    REGISTRY_TOKEN: "{{ lookup('file', '~/.credentials/ghcr_token') }}"
 ```
 ### How It Works
 1. laconic-so reads the `registry-credentials` config from spec.yml
 2. Creates a Kubernetes `docker-registry` secret named `{deployment}-registry`
 3. The deployment's pods reference this secret for image pulls
 ## Cluster and Volume Management
 ### Stopping Deployments
 The `deployment stop` command has two important flags:
 ```bash
 # Default: stops deployment, deletes cluster, PRESERVES volumes
 laconic-so deployment --dir my-deployment stop
 # Explicitly delete volumes (USE WITH CAUTION)
 laconic-so deployment --dir my-deployment stop --delete-volumes
 ```
 ### Volume Persistence
 Volumes persist across cluster deletion by design. This is important because:
 - **Data survives cluster recreation**: Ledger data, databases, and other state are preserved
 - **Faster recovery**: No need to re-sync or rebuild data after cluster issues
 - **Safe cluster upgrades**: Delete and recreate cluster without data loss
 **Only use `--delete-volumes` when:**
 - You explicitly want to start fresh with no data
 - The user specifically requests volume deletion
 - You're cleaning up a test/dev environment completely
 ### Shared Cluster Architecture
 In kind deployments, multiple stacks share a single cluster:
 - First `deployment start` creates the cluster
 - Subsequent deployments reuse the existing cluster
 - `deployment stop` on ANY deployment deletes the shared cluster
 - Other deployments will fail until cluster is recreated
 To stop a single deployment without affecting the cluster:
 ```bash
 laconic-so deployment --dir my-deployment stop --skip-cluster-management
 ```
 ## Volume Persistence in k8s-kind
 k8s-kind has 3 storage layers:
 - **Docker Host**: The physical server running Docker
 - **Kind Node**: A Docker container simulating a k8s node
 - **Pod Container**: Your workload
 For k8s-kind, volumes with paths are mounted from Docker Host → Kind Node → Pod via extraMounts.
 | spec.yml volume | Storage Location | Survives Pod Restart | Survives Cluster Restart |
 |-----------------|------------------|---------------------|-------------------------|
 | `vol:` (empty)  | Kind Node PVC    | ✅ | ❌ |
 | `vol: ./data/x` | Docker Host      | ✅ | ✅ |
 | `vol: /abs/path`| Docker Host      | ✅ | ✅ |
 **Recommendation**: Always use paths for data you want to keep. Relative paths
 (e.g., `./data/rpc-config`) resolve to `$DEPLOYMENT_DIR/data/rpc-config` on the
 Docker Host.
 ### Example
 ```yaml
 # In spec.yml
 volumes:
  rpc-config: ./data/rpc-config  # Persists to $DEPLOYMENT_DIR/data/rpc-config
  chain-data: ./data/chain       # Persists to $DEPLOYMENT_DIR/data/chain
  temp-cache:                    # Empty = Kind Node PVC (lost on cluster delete)
 ```
 ### The Antipattern
 Empty-path volumes appear persistent because they survive pod restarts (data lives
 in Kind Node container). However, this data is lost when the kind cluster is
 recreated. This "false persistence" has caused data loss when operators assumed
 their data was safe.
--- a/stack_orchestrator/constants.py
+++ b/stack_orchestrator/constants.py
@ -44,4 +44,3 @@ unlimited_memlock_key = "unlimited-memlock"
 runtime_class_key = "runtime-class"
 high_memlock_runtime = "high-memlock"
 high_memlock_spec_filename = "high-memlock-spec.json"
 acme_email_key = "acme-email"
--- a/stack_orchestrator/data/k8s/components/ingress/ingress-caddy-kind-deploy.yaml
+++ b/stack_orchestrator/data/k8s/components/ingress/ingress-caddy-kind-deploy.yaml
@ -93,7 +93,6 @@ rules:
      - get
      - create
      - update
      - delete
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
--- a/stack_orchestrator/deploy/deployment.py
+++ b/stack_orchestrator/deploy/deployment.py
@ -15,9 +15,7 @@
 import click
 from pathlib import Path
 import subprocess
 import sys
 import time
 from stack_orchestrator import constants
 from stack_orchestrator.deploy.images import push_images_operation
 from stack_orchestrator.deploy.deploy import (
@ -230,176 +228,3 @@ def run_job(ctx, job_name, helm_release):
    ctx.obj = make_deploy_context(ctx)
    run_job_operation(ctx, job_name, helm_release)
@command.command()
@click.option("--stack-path", help="Path to stack git repo (overrides stored path)")
@click.option(
    "--spec-file", help="Path to GitOps spec.yml in repo (e.g., deployment/spec.yml)"
 )
@click.option("--config-file", help="Config file to pass to deploy init")
@click.option(
    "--force",
    is_flag=True,
    default=False,
    help="Skip DNS verification",
 )
@click.option(
    "--expected-ip",
    help="Expected IP for DNS verification (if different from egress)",
 )
@click.pass_context
 def restart(ctx, stack_path, spec_file, config_file, force, expected_ip):
    """Pull latest code and restart deployment using git-tracked spec.
    GitOps workflow:
    1. Operator maintains spec.yml in their git repository
    2. This command pulls latest code (including updated spec.yml)
    3. If hostname changed, verifies DNS routes to this server
    4. Syncs deployment directory with the git-tracked spec
    5. Stops and restarts the deployment
    Data volumes are always preserved. The cluster is never destroyed.
    Stack source resolution (in order):
    1. --stack-path argument (if provided)
    2. stack-source field in deployment.yml (if stored)
    3. Error if neither available
    Note: spec.yml should be maintained in git, not regenerated from
    commands.py on each restart. Use 'deploy init' only for initial
    spec generation, then customize and commit to your operator repo.
    """
    from stack_orchestrator.util import get_yaml, get_parsed_deployment_spec
    from stack_orchestrator.deploy.deployment_create import create_operation
    from stack_orchestrator.deploy.dns_probe import verify_dns_via_probe
    deployment_context: DeploymentContext = ctx.obj
    # Get current spec info (before git pull)
    current_spec = deployment_context.spec
    current_http_proxy = current_spec.get_http_proxy()
    current_hostname = (
        current_http_proxy[0]["host-name"] if current_http_proxy else None
    )
    # Resolve stack source path
    if stack_path:
        stack_source = Path(stack_path).resolve()
    else:
        # Try to get from deployment.yml
        deployment_file = (
            deployment_context.deployment_dir / constants.deployment_file_name
        )
        deployment_data = get_yaml().load(open(deployment_file))
        stack_source_str = deployment_data.get("stack-source")
        if not stack_source_str:
            print(
                "Error: No stack-source in deployment.yml and --stack-path not provided"
            )
            print("Use --stack-path to specify the stack git repository location")
            sys.exit(1)
        stack_source = Path(stack_source_str)
    if not stack_source.exists():
        print(f"Error: Stack source path does not exist: {stack_source}")
        sys.exit(1)
    print("=== Deployment Restart ===")
    print(f"Deployment dir: {deployment_context.deployment_dir}")
    print(f"Stack source: {stack_source}")
    print(f"Current hostname: {current_hostname}")
    # Step 1: Git pull (brings in updated spec.yml from operator's repo)
    print("\n[1/4] Pulling latest code from stack repository...")
    git_result = subprocess.run(
        ["git", "pull"], cwd=stack_source, capture_output=True, text=True
    )
    if git_result.returncode != 0:
        print(f"Git pull failed: {git_result.stderr}")
        sys.exit(1)
    print(f"Git pull: {git_result.stdout.strip()}")
    # Determine spec file location
    # Priority: --spec-file argument > repo's deployment/spec.yml > deployment dir
    # Stack path is like: repo/stack_orchestrator/data/stacks/stack-name
    # So repo root is 4 parents up
    repo_root = stack_source.parent.parent.parent.parent
    if spec_file:
        # Spec file relative to repo root
        spec_file_path = repo_root / spec_file
    else:
        # Try standard GitOps location in repo
        gitops_spec = repo_root / "deployment" / "spec.yml"
        if gitops_spec.exists():
            spec_file_path = gitops_spec
        else:
            # Fall back to deployment directory
            spec_file_path = deployment_context.deployment_dir / "spec.yml"
    if not spec_file_path.exists():
        print(f"Error: spec.yml not found at {spec_file_path}")
        print("For GitOps, add spec.yml to your repo at deployment/spec.yml")
        print("Or specify --spec-file with path relative to repo root")
        sys.exit(1)
    print(f"Using spec: {spec_file_path}")
    # Parse spec to check for hostname changes
    new_spec_obj = get_parsed_deployment_spec(str(spec_file_path))
    new_http_proxy = new_spec_obj.get("network", {}).get("http-proxy", [])
    new_hostname = new_http_proxy[0]["host-name"] if new_http_proxy else None
    print(f"Spec hostname: {new_hostname}")
    # Step 2: DNS verification (only if hostname changed)
    if new_hostname and new_hostname != current_hostname:
        print(f"\n[2/4] Hostname changed: {current_hostname} -> {new_hostname}")
        if force:
            print("DNS verification skipped (--force)")
        else:
            print("Verifying DNS via probe...")
            if not verify_dns_via_probe(new_hostname):
                print(f"\nDNS verification failed for {new_hostname}")
                print("Ensure DNS is configured before restarting.")
                print("Use --force to skip this check.")
                sys.exit(1)
    else:
        print("\n[2/4] Hostname unchanged, skipping DNS verification")
    # Step 3: Sync deployment directory with spec
    print("\n[3/4] Syncing deployment directory...")
    deploy_ctx = make_deploy_context(ctx)
    create_operation(
        deployment_command_context=deploy_ctx,
        spec_file=str(spec_file_path),
        deployment_dir=str(deployment_context.deployment_dir),
        update=True,
        network_dir=None,
        initial_peers=None,
    )
    # Reload deployment context with updated spec
    deployment_context.init(deployment_context.deployment_dir)
    ctx.obj = deployment_context
    # Stop deployment
    print("\n[4/4] Restarting deployment...")
    ctx.obj = make_deploy_context(ctx)
    down_operation(
        ctx, delete_volumes=False, extra_args_list=[], skip_cluster_management=True
    )
    # Brief pause to ensure clean shutdown
    time.sleep(5)
    # Start deployment
    up_operation(
        ctx, services_list=None, stay_attached=False, skip_cluster_management=True
    )
    print("\n=== Restart Complete ===")
    print("Deployment restarted with git-tracked configuration.")
    if new_hostname and new_hostname != current_hostname:
        print(f"\nNew hostname: {new_hostname}")
        print("Caddy will automatically provision TLS certificate.")
--- a/stack_orchestrator/deploy/deployment_create.py
+++ b/stack_orchestrator/deploy/deployment_create.py
@ -15,12 +15,9 @@
 import click
 from importlib import util
 import json
 import os
 import re
 import base64
 from pathlib import Path
-from typing import List, Optional
+from typing import List
 import random
 from shutil import copy, copyfile, copytree, rmtree
 from secrets import token_hex
@ -471,10 +468,7 @@ def init_operation(
            else:
                volume_descriptors[named_volume] = f"./data/{named_volume}"
        if volume_descriptors:
-            # Merge with existing volumes from stack init()
+            spec_file_content["volumes"] = volume_descriptors
            # init() volumes take precedence over compose defaults
            orig_volumes = spec_file_content.get("volumes", {})
            spec_file_content["volumes"] = {**volume_descriptors, **orig_volumes}
        if configmap_descriptors:
            spec_file_content["configmaps"] = configmap_descriptors
@ -487,180 +481,15 @@ def init_operation(
        get_yaml().dump(spec_file_content, output_file)
-# Token pattern: $generate:hex:32$ or $generate:base64:16$
+def _write_config_file(spec_file: Path, config_env_file: Path):
 GENERATE_TOKEN_PATTERN = re.compile(r"\$generate:(\w+):(\d+)\$")
 def _generate_and_store_secrets(config_vars: dict, deployment_name: str):
    """Generate secrets for $generate:...$ tokens and store in K8s Secret.
    Called by `deploy create` - generates fresh secrets and stores them.
    Returns the generated secrets dict for reference.
    """
    from kubernetes import client, config as k8s_config
    secrets = {}
    for name, value in config_vars.items():
        if not isinstance(value, str):
            continue
        match = GENERATE_TOKEN_PATTERN.search(value)
        if not match:
            continue
        secret_type, length = match.group(1), int(match.group(2))
        if secret_type == "hex":
            secrets[name] = token_hex(length)
        elif secret_type == "base64":
            secrets[name] = base64.b64encode(os.urandom(length)).decode()
        else:
            secrets[name] = token_hex(length)
    if not secrets:
        return secrets
    # Store in K8s Secret
    try:
        k8s_config.load_kube_config()
    except Exception:
        # Fall back to in-cluster config if available
        try:
            k8s_config.load_incluster_config()
        except Exception:
            print(
                "Warning: Could not load kube config, secrets will not be stored in K8s"
            )
            return secrets
    v1 = client.CoreV1Api()
    secret_name = f"{deployment_name}-generated-secrets"
    namespace = "default"
    secret_data = {k: base64.b64encode(v.encode()).decode() for k, v in secrets.items()}
    k8s_secret = client.V1Secret(
        metadata=client.V1ObjectMeta(name=secret_name), data=secret_data, type="Opaque"
    )
    try:
        v1.create_namespaced_secret(namespace, k8s_secret)
        num_secrets = len(secrets)
        print(f"Created K8s Secret '{secret_name}' with {num_secrets} secret(s)")
    except client.exceptions.ApiException as e:
        if e.status == 409:  # Already exists
            v1.replace_namespaced_secret(secret_name, namespace, k8s_secret)
            num_secrets = len(secrets)
            print(f"Updated K8s Secret '{secret_name}' with {num_secrets} secret(s)")
        else:
            raise
    return secrets
 def create_registry_secret(spec: Spec, deployment_name: str) -> Optional[str]:
    """Create K8s docker-registry secret from spec + environment.
    Reads registry configuration from spec.yml and creates a Kubernetes
    secret of type kubernetes.io/dockerconfigjson for image pulls.
    Args:
        spec: The deployment spec containing image-registry config
        deployment_name: Name of the deployment (used for secret naming)
    Returns:
        The secret name if created, None if no registry config
    """
    from kubernetes import client, config as k8s_config
    registry_config = spec.get_image_registry_config()
    if not registry_config:
        return None
    server = registry_config.get("server")
    username = registry_config.get("username")
    token_env = registry_config.get("token-env")
    if not all([server, username, token_env]):
        return None
    # Type narrowing for pyright - we've validated these aren't None above
    assert token_env is not None
    token = os.environ.get(token_env)
    if not token:
        print(
            f"Warning: Registry token env var '{token_env}' not set, "
            "skipping registry secret"
        )
        return None
    # Create dockerconfigjson format (Docker API uses "password" field for tokens)
    auth = base64.b64encode(f"{username}:{token}".encode()).decode()
    docker_config = {
        "auths": {server: {"username": username, "password": token, "auth": auth}}
    }
    # Secret name derived from deployment name
    secret_name = f"{deployment_name}-registry"
    # Load kube config
    try:
        k8s_config.load_kube_config()
    except Exception:
        try:
            k8s_config.load_incluster_config()
        except Exception:
            print("Warning: Could not load kube config, registry secret not created")
            return None
    v1 = client.CoreV1Api()
    namespace = "default"
    k8s_secret = client.V1Secret(
        metadata=client.V1ObjectMeta(name=secret_name),
        data={
            ".dockerconfigjson": base64.b64encode(
                json.dumps(docker_config).encode()
            ).decode()
        },
        type="kubernetes.io/dockerconfigjson",
    )
    try:
        v1.create_namespaced_secret(namespace, k8s_secret)
        print(f"Created registry secret '{secret_name}' for {server}")
    except client.exceptions.ApiException as e:
        if e.status == 409:  # Already exists
            v1.replace_namespaced_secret(secret_name, namespace, k8s_secret)
            print(f"Updated registry secret '{secret_name}' for {server}")
        else:
            raise
    return secret_name
 def _write_config_file(
    spec_file: Path, config_env_file: Path, deployment_name: Optional[str] = None
 ):
    spec_content = get_parsed_deployment_spec(spec_file)
-    config_vars = spec_content.get("config", {}) or {}
+    # Note: we want to write an empty file even if we have no config variables
    # Generate and store secrets in K8s if deployment_name provided and tokens exist
    if deployment_name and config_vars:
        has_generate_tokens = any(
            isinstance(v, str) and GENERATE_TOKEN_PATTERN.search(v)
            for v in config_vars.values()
        )
        if has_generate_tokens:
            _generate_and_store_secrets(config_vars, deployment_name)
    # Write non-secret config to config.env (exclude $generate:...$ tokens)
    with open(config_env_file, "w") as output_file:
-        if config_vars:
+        if "config" in spec_content and spec_content["config"]:
-            for variable_name, variable_value in config_vars.items():
+            config_vars = spec_content["config"]
-                # Skip variables with generate tokens - they go to K8s Secret
+            if config_vars:
-                if isinstance(variable_value, str) and GENERATE_TOKEN_PATTERN.search(
+                for variable_name, variable_value in config_vars.items():
-                    variable_value
+                    output_file.write(f"{variable_name}={variable_value}\n")
                ):
                    continue
                output_file.write(f"{variable_name}={variable_value}\n")
 def _write_kube_config_file(external_path: Path, internal_path: Path):
@ -675,14 +504,11 @@ def _copy_files_to_directory(file_paths: List[Path], directory: Path):
        copy(path, os.path.join(directory, os.path.basename(path)))
-def _create_deployment_file(deployment_dir: Path, stack_source: Optional[Path] = None):
+def _create_deployment_file(deployment_dir: Path):
    deployment_file_path = deployment_dir.joinpath(constants.deployment_file_name)
    cluster = f"{constants.cluster_name_prefix}{token_hex(8)}"
    deployment_content = {constants.cluster_id_key: cluster}
    if stack_source:
        deployment_content["stack-source"] = str(stack_source)
    with open(deployment_file_path, "w") as output_file:
-        get_yaml().dump(deployment_content, output_file)
+        output_file.write(f"{constants.cluster_id_key}: {cluster}\n")
 def _check_volume_definitions(spec):
@ -690,14 +516,10 @@ def _check_volume_definitions(spec):
        for volume_name, volume_path in spec.get_volumes().items():
            if volume_path:
                if not os.path.isabs(volume_path):
-                    # For k8s-kind: allow relative paths, they'll be resolved
+                    raise Exception(
-                    # by _make_absolute_host_path() during kind config generation
+                        f"Relative path {volume_path} for volume {volume_name} not "
-                    if not spec.is_kind_deployment():
+                        f"supported for deployment type {spec.get_deployment_type()}"
-                        deploy_type = spec.get_deployment_type()
+                    )
                        raise Exception(
                            f"Relative path {volume_path} for volume "
                            f"{volume_name} not supported for {deploy_type}"
                        )
@click.command()
@ -791,15 +613,11 @@ def create_operation(
        generate_helm_chart(stack_name, spec_file, deployment_dir_path)
        return  # Exit early for helm chart generation
    # Resolve stack source path for restart capability
    stack_source = get_stack_path(stack_name)
    if update:
        # Sync mode: write to temp dir, then copy to deployment dir with backups
        temp_dir = Path(tempfile.mkdtemp(prefix="deployment-sync-"))
        try:
-            # Write deployment files to temp dir
+            # Write deployment files to temp dir (skip deployment.yml to preserve cluster ID)
            # (skip deployment.yml to preserve cluster ID)
            _write_deployment_files(
                temp_dir,
                Path(spec_file),
@ -807,14 +625,12 @@ def create_operation(
                stack_name,
                deployment_type,
                include_deployment_file=False,
                stack_source=stack_source,
            )
-            # Copy from temp to deployment dir, excluding data volumes
+            # Copy from temp to deployment dir, excluding data volumes and backing up changed files
-            # and backing up changed files.
+            # Exclude data/* to avoid touching user data volumes
-            # Exclude data/* to avoid touching user data volumes.
+            # Exclude config file to preserve deployment settings (XXX breaks passing config vars
-            # Exclude config file to preserve deployment settings
+            # from spec. could warn about this or not exclude...)
            # (XXX breaks passing config vars from spec)
            exclude_patterns = ["data", "data/*", constants.config_file_name]
            _safe_copy_tree(
                temp_dir, deployment_dir_path, exclude_patterns=exclude_patterns
@ -831,7 +647,6 @@ def create_operation(
            stack_name,
            deployment_type,
            include_deployment_file=True,
            stack_source=stack_source,
        )
    # Delegate to the stack's Python code
@ -852,7 +667,7 @@ def create_operation(
    )
-def _safe_copy_tree(src: Path, dst: Path, exclude_patterns: Optional[List[str]] = None):
+def _safe_copy_tree(src: Path, dst: Path, exclude_patterns: List[str] = None):
    """
    Recursively copy a directory tree, backing up changed files with .bak suffix.
@ -903,7 +718,6 @@ def _write_deployment_files(
    stack_name: str,
    deployment_type: str,
    include_deployment_file: bool = True,
    stack_source: Optional[Path] = None,
 ):
    """
    Write deployment files to target directory.
@ -913,8 +727,7 @@ def _write_deployment_files(
    :param parsed_spec: Parsed spec object
    :param stack_name: Name of stack
    :param deployment_type: Type of deployment
-    :param include_deployment_file: Whether to create deployment.yml (skip for update)
+    :param include_deployment_file: Whether to create deployment.yml file (skip for update)
    :param stack_source: Path to stack source (git repo) for restart capability
    """
    stack_file = get_stack_path(stack_name).joinpath(constants.stack_file_name)
    parsed_stack = get_parsed_stack_config(stack_name)
@ -925,15 +738,10 @@ def _write_deployment_files(
    # Create deployment file if requested
    if include_deployment_file:
-        _create_deployment_file(target_dir, stack_source=stack_source)
+        _create_deployment_file(target_dir)
    # Copy any config variables from the spec file into an env file suitable for compose
-    # Use stack_name as deployment_name for K8s secret naming
+    _write_config_file(spec_file, target_dir.joinpath(constants.config_file_name))
    # Extract just the name part if stack_name is a path ("path/to/stack" -> "stack")
    deployment_name = Path(stack_name).name.replace("_", "-")
    _write_config_file(
        spec_file, target_dir.joinpath(constants.config_file_name), deployment_name
    )
    # Copy any k8s config file into the target dir
    if deployment_type == "k8s":
@ -994,9 +802,8 @@ def _write_deployment_files(
                    )
        else:
            # TODO:
-            # This is odd - looks up config dir that matches a volume name,
+            # this is odd - looks up config dir that matches a volume name, then copies as a mount dir?
-            # then copies as a mount dir?
+            # AFAICT this is not used by or relevant to any existing stack - roy
            # AFAICT not used by or relevant to any existing stack - roy
            # TODO: We should probably only do this if the volume is marked :ro.
            for volume_name, volume_path in parsed_spec.get_volumes().items():
--- a/stack_orchestrator/deploy/dns_probe.py
+++ b/stack_orchestrator/deploy/dns_probe.py
@ -1,159 +0,0 @@
 # Copyright © 2024 Vulcanize
 # SPDX-License-Identifier: AGPL-3.0
 """DNS verification via temporary ingress probe."""
 import secrets
 import socket
 import time
 from typing import Optional
 import requests
 from kubernetes import client
 def get_server_egress_ip() -> str:
    """Get this server's public egress IP via ipify."""
    response = requests.get("https://api.ipify.org", timeout=10)
    response.raise_for_status()
    return response.text.strip()
 def resolve_hostname(hostname: str) -> list[str]:
    """Resolve hostname to list of IP addresses."""
    try:
        _, _, ips = socket.gethostbyname_ex(hostname)
        return ips
    except socket.gaierror:
        return []
 def verify_dns_simple(hostname: str, expected_ip: Optional[str] = None) -> bool:
    """Simple DNS verification - check hostname resolves to expected IP.
    If expected_ip not provided, uses server's egress IP.
    Returns True if hostname resolves to expected IP.
    """
    resolved_ips = resolve_hostname(hostname)
    if not resolved_ips:
        print(f"DNS FAIL: {hostname} does not resolve")
        return False
    if expected_ip is None:
        expected_ip = get_server_egress_ip()
    if expected_ip in resolved_ips:
        print(f"DNS OK: {hostname} -> {resolved_ips} (includes {expected_ip})")
        return True
    else:
        print(f"DNS WARN: {hostname} -> {resolved_ips} (expected {expected_ip})")
        return False
 def create_probe_ingress(hostname: str, namespace: str = "default") -> str:
    """Create a temporary ingress for DNS probing.
    Returns the probe token that the ingress will respond with.
    """
    token = secrets.token_hex(16)
    networking_api = client.NetworkingV1Api()
    # Create a simple ingress that Caddy will pick up
    ingress = client.V1Ingress(
        metadata=client.V1ObjectMeta(
            name="laconic-dns-probe",
            annotations={
                "kubernetes.io/ingress.class": "caddy",
                "laconic.com/probe-token": token,
            },
        ),
        spec=client.V1IngressSpec(
            rules=[
                client.V1IngressRule(
                    host=hostname,
                    http=client.V1HTTPIngressRuleValue(
                        paths=[
                            client.V1HTTPIngressPath(
                                path="/.well-known/laconic-probe",
                                path_type="Exact",
                                backend=client.V1IngressBackend(
                                    service=client.V1IngressServiceBackend(
                                        name="caddy-ingress-controller",
                                        port=client.V1ServiceBackendPort(number=80),
                                    )
                                ),
                            )
                        ]
                    ),
                )
            ]
        ),
    )
    networking_api.create_namespaced_ingress(namespace=namespace, body=ingress)
    return token
 def delete_probe_ingress(namespace: str = "default"):
    """Delete the temporary probe ingress."""
    networking_api = client.NetworkingV1Api()
    try:
        networking_api.delete_namespaced_ingress(
            name="laconic-dns-probe", namespace=namespace
        )
    except client.exceptions.ApiException:
        pass  # Ignore if already deleted
 def verify_dns_via_probe(
    hostname: str, namespace: str = "default", timeout: int = 30, poll_interval: int = 2
 ) -> bool:
    """Verify DNS by creating temp ingress and probing it.
    This definitively proves that traffic to the hostname reaches this cluster.
    Args:
        hostname: The hostname to verify
        namespace: Kubernetes namespace for probe ingress
        timeout: Total seconds to wait for probe to succeed
        poll_interval: Seconds between probe attempts
    Returns:
        True if probe succeeds, False otherwise
    """
    # First check DNS resolves at all
    if not resolve_hostname(hostname):
        print(f"DNS FAIL: {hostname} does not resolve")
        return False
    print(f"Creating probe ingress for {hostname}...")
    create_probe_ingress(hostname, namespace)
    try:
        # Wait for Caddy to pick up the ingress
        time.sleep(3)
        # Poll until success or timeout
        probe_url = f"http://{hostname}/.well-known/laconic-probe"
        start_time = time.time()
        last_error = None
        while time.time() - start_time < timeout:
            try:
                response = requests.get(probe_url, timeout=5)
                # For now, just verify we get a response from this cluster
                # A more robust check would verify a unique token
                if response.status_code < 500:
                    print(f"DNS PROBE OK: {hostname} routes to this cluster")
                    return True
            except requests.RequestException as e:
                last_error = e
            time.sleep(poll_interval)
        print(f"DNS PROBE FAIL: {hostname} - {last_error}")
        return False
    finally:
        print("Cleaning up probe ingress...")
        delete_probe_ingress(namespace)
--- a/stack_orchestrator/deploy/k8s/cluster_info.py
+++ b/stack_orchestrator/deploy/k8s/cluster_info.py
@ -31,7 +31,6 @@ from stack_orchestrator.deploy.k8s.helpers import (
    envs_from_environment_variables_map,
    envs_from_compose_file,
    merge_envs,
    translate_sidecar_service_names,
 )
 from stack_orchestrator.deploy.deploy_util import (
    parsed_pod_files_map_from_file_names,
@ -126,8 +125,7 @@ class ClusterInfo:
                                name=(
                                    f"{self.app_name}-nodeport-"
                                    f"{pod_port}-{protocol.lower()}"
-                                ),
+                                )
                                labels={"app": self.app_name},
                            ),
                            spec=client.V1ServiceSpec(
                                type="NodePort",
@ -210,9 +208,7 @@ class ClusterInfo:
            ingress = client.V1Ingress(
                metadata=client.V1ObjectMeta(
-                    name=f"{self.app_name}-ingress",
+                    name=f"{self.app_name}-ingress", annotations=ingress_annotations
                    labels={"app": self.app_name},
                    annotations=ingress_annotations,
                ),
                spec=spec,
            )
@ -242,10 +238,7 @@ class ClusterInfo:
        ]
        service = client.V1Service(
-            metadata=client.V1ObjectMeta(
+            metadata=client.V1ObjectMeta(name=f"{self.app_name}-service"),
                name=f"{self.app_name}-service",
                labels={"app": self.app_name},
            ),
            spec=client.V1ServiceSpec(
                type="ClusterIP",
                ports=service_ports,
@ -327,7 +320,7 @@ class ClusterInfo:
            spec = client.V1ConfigMap(
                metadata=client.V1ObjectMeta(
                    name=f"{self.app_name}-{cfg_map_name}",
-                    labels={"app": self.app_name, "configmap-label": cfg_map_name},
+                    labels={"configmap-label": cfg_map_name},
                ),
                binary_data=data,
            )
@ -359,15 +352,11 @@ class ClusterInfo:
                continue
            if not os.path.isabs(volume_path):
-                # For k8s-kind, allow relative paths:
+                print(
-                # - PV uses /mnt/{volume_name} (path inside kind node)
+                    f"WARNING: {volume_name}:{volume_path} is not absolute, "
-                # - extraMounts resolve the relative path to Docker Host
+                    "cannot bind volume."
-                if not self.spec.is_kind_deployment():
+                )
-                    print(
+                continue
                        f"WARNING: {volume_name}:{volume_path} is not absolute, "
                        "cannot bind volume."
                    )
                    continue
            if self.spec.is_kind_deployment():
                host_path = client.V1HostPathVolumeSource(
@ -384,53 +373,20 @@ class ClusterInfo:
            pv = client.V1PersistentVolume(
                metadata=client.V1ObjectMeta(
                    name=f"{self.app_name}-{volume_name}",
-                    labels={
+                    labels={"volume-label": f"{self.app_name}-{volume_name}"},
                        "app": self.app_name,
                        "volume-label": f"{self.app_name}-{volume_name}",
                    },
                ),
                spec=spec,
            )
            result.append(pv)
        return result
    def _any_service_has_host_network(self):
        for pod_name in self.parsed_pod_yaml_map:
            pod = self.parsed_pod_yaml_map[pod_name]
            for svc in pod.get("services", {}).values():
                if svc.get("network_mode") == "host":
                    return True
        return False
    def _resolve_container_resources(
        self, container_name: str, service_info: dict, global_resources: Resources
    ) -> Resources:
        """Resolve resources for a container using layered priority.
        Priority: spec per-container > compose deploy.resources
        > spec global > DEFAULT
        """
        # 1. Check spec.yml for per-container override
        per_container = self.spec.get_container_resources_for(container_name)
        if per_container:
            return per_container
        # 2. Check compose service_info for deploy.resources
        deploy_block = service_info.get("deploy", {})
        compose_resources = deploy_block.get("resources", {}) if deploy_block else {}
        if compose_resources:
            return Resources(compose_resources)
        # 3. Fall back to spec.yml global (already resolved with DEFAULT fallback)
        return global_resources
    # TODO: put things like image pull policy into an object-scope struct
    def get_deployment(self, image_pull_policy: Optional[str] = None):
        containers = []
        services = {}
-        global_resources = self.spec.get_container_resources()
+        resources = self.spec.get_container_resources()
-        if not global_resources:
+        if not resources:
-            global_resources = DEFAULT_CONTAINER_RESOURCES
+            resources = DEFAULT_CONTAINER_RESOURCES
        for pod_name in self.parsed_pod_yaml_map:
            pod = self.parsed_pod_yaml_map[pod_name]
            services = pod["services"]
@ -470,12 +426,6 @@ class ClusterInfo:
                    if "environment" in service_info
                    else self.environment_variables.map
                )
                # Translate docker-compose service names to localhost for sidecars
                # All services in the same pod share the network namespace
                sibling_services = [s for s in services.keys() if s != service_name]
                merged_envs = translate_sidecar_service_names(
                    merged_envs, sibling_services
                )
                envs = envs_from_environment_variables_map(merged_envs)
                if opts.o.debug:
                    print(f"Merged envs: {envs}")
@ -503,19 +453,6 @@ class ClusterInfo:
                if "command" in service_info:
                    cmd = service_info["command"]
                    container_args = cmd if isinstance(cmd, list) else cmd.split()
                # Add env_from to pull secrets from K8s Secret
                secret_name = f"{self.app_name}-generated-secrets"
                env_from = [
                    client.V1EnvFromSource(
                        secret_ref=client.V1SecretEnvSource(
                            name=secret_name,
                            optional=True,  # Don't fail if no secrets
                        )
                    )
                ]
                container_resources = self._resolve_container_resources(
                    container_name, service_info, global_resources
                )
                container = client.V1Container(
                    name=container_name,
                    image=image_to_use,
@ -523,7 +460,6 @@ class ClusterInfo:
                    command=container_command,
                    args=container_args,
                    env=envs,
                    env_from=env_from,
                    ports=container_ports if container_ports else None,
                    volume_mounts=volume_mounts,
                    security_context=client.V1SecurityContext(
@ -534,18 +470,13 @@ class ClusterInfo:
                        if self.spec.get_capabilities()
                        else None,
                    ),
-                    resources=to_k8s_resource_requirements(container_resources),
+                    resources=to_k8s_resource_requirements(resources),
                )
                containers.append(container)
        volumes = volumes_for_pod_files(
            self.parsed_pod_yaml_map, self.spec, self.app_name
        )
-        registry_config = self.spec.get_image_registry_config()
+        image_pull_secrets = [client.V1LocalObjectReference(name="laconic-registry")]
        if registry_config:
            secret_name = f"{self.app_name}-registry"
            image_pull_secrets = [client.V1LocalObjectReference(name=secret_name)]
        else:
            image_pull_secrets = []
        annotations = None
        labels = {"app": self.app_name}
@ -601,7 +532,6 @@ class ClusterInfo:
                    )
                )
        use_host_network = self._any_service_has_host_network()
        template = client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(annotations=annotations, labels=labels),
            spec=client.V1PodSpec(
@ -611,8 +541,6 @@ class ClusterInfo:
                affinity=affinity,
                tolerations=tolerations,
                runtime_class_name=self.spec.get_runtime_class(),
                host_network=use_host_network or None,
                dns_policy=("ClusterFirstWithHostNet" if use_host_network else None),
            ),
        )
        spec = client.V1DeploymentSpec(
--- a/stack_orchestrator/deploy/k8s/deploy_k8s.py
+++ b/stack_orchestrator/deploy/k8s/deploy_k8s.py
@ -29,7 +29,6 @@ from stack_orchestrator.deploy.k8s.helpers import (
 from stack_orchestrator.deploy.k8s.helpers import (
    install_ingress_for_kind,
    wait_for_ingress_in_kind,
    is_ingress_running,
 )
 from stack_orchestrator.deploy.k8s.helpers import (
    pods_in_deployment,
@ -96,7 +95,7 @@ class K8sDeployer(Deployer):
    core_api: client.CoreV1Api
    apps_api: client.AppsV1Api
    networking_api: client.NetworkingV1Api
-    k8s_namespace: str
+    k8s_namespace: str = "default"
    kind_cluster_name: str
    skip_cluster_management: bool
    cluster_info: ClusterInfo
@ -113,7 +112,6 @@ class K8sDeployer(Deployer):
    ) -> None:
        self.type = type
        self.skip_cluster_management = False
        self.k8s_namespace = "default"  # Will be overridden below if context exists
        # TODO: workaround pending refactoring above to cope with being
        # created with a null deployment_context
        if deployment_context is None:
@ -121,8 +119,6 @@ class K8sDeployer(Deployer):
        self.deployment_dir = deployment_context.deployment_dir
        self.deployment_context = deployment_context
        self.kind_cluster_name = compose_project_name
        # Use deployment-specific namespace for resource isolation and easy cleanup
        self.k8s_namespace = f"laconic-{compose_project_name}"
        self.cluster_info = ClusterInfo()
        self.cluster_info.int(
            compose_files,
@ -152,46 +148,6 @@ class K8sDeployer(Deployer):
        self.apps_api = client.AppsV1Api()
        self.custom_obj_api = client.CustomObjectsApi()
    def _ensure_namespace(self):
        """Create the deployment namespace if it doesn't exist."""
        if opts.o.dry_run:
            print(f"Dry run: would create namespace {self.k8s_namespace}")
            return
        try:
            self.core_api.read_namespace(name=self.k8s_namespace)
            if opts.o.debug:
                print(f"Namespace {self.k8s_namespace} already exists")
        except ApiException as e:
            if e.status == 404:
                # Create the namespace
                ns = client.V1Namespace(
                    metadata=client.V1ObjectMeta(
                        name=self.k8s_namespace,
                        labels={"app": self.cluster_info.app_name},
                    )
                )
                self.core_api.create_namespace(body=ns)
                if opts.o.debug:
                    print(f"Created namespace {self.k8s_namespace}")
            else:
                raise
    def _delete_namespace(self):
        """Delete the deployment namespace and all resources within it."""
        if opts.o.dry_run:
            print(f"Dry run: would delete namespace {self.k8s_namespace}")
            return
        try:
            self.core_api.delete_namespace(name=self.k8s_namespace)
            if opts.o.debug:
                print(f"Deleted namespace {self.k8s_namespace}")
        except ApiException as e:
            if e.status == 404:
                if opts.o.debug:
                    print(f"Namespace {self.k8s_namespace} not found")
            else:
                raise
    def _create_volume_data(self):
        # Create the host-path-mounted PVs for this deployment
        pvs = self.cluster_info.get_pvs()
@ -333,40 +289,22 @@ class K8sDeployer(Deployer):
        self.skip_cluster_management = skip_cluster_management
        if not opts.o.dry_run:
            if self.is_kind() and not self.skip_cluster_management:
-                # Create the kind cluster (or reuse existing one)
+                # Create the kind cluster
-                kind_config = str(
+                create_cluster(
-                    self.deployment_dir.joinpath(constants.kind_config_filename)
+                    self.kind_cluster_name,
                    str(self.deployment_dir.joinpath(constants.kind_config_filename)),
                )
-                actual_cluster = create_cluster(self.kind_cluster_name, kind_config)
+                # Ensure the referenced containers are copied into kind
-                if actual_cluster != self.kind_cluster_name:
+                load_images_into_kind(
-                    # An existing cluster was found, use it instead
+                    self.kind_cluster_name, self.cluster_info.image_set
                    self.kind_cluster_name = actual_cluster
                # Only load locally-built images into kind
                # Registry images (docker.io, ghcr.io, etc.) will be pulled by k8s
                local_containers = self.deployment_context.stack.obj.get(
                    "containers", []
                )
                if local_containers:
                    # Filter image_set to only images matching local containers
                    local_images = {
                        img
                        for img in self.cluster_info.image_set
                        if any(c in img for c in local_containers)
                    }
                    if local_images:
                        load_images_into_kind(self.kind_cluster_name, local_images)
                # Note: if no local containers defined, all images come from registries
            self.connect_api()
            # Create deployment-specific namespace for resource isolation
            self._ensure_namespace()
            if self.is_kind() and not self.skip_cluster_management:
                # Configure ingress controller (not installed by default in kind)
-                # Skip if already running (idempotent for shared cluster)
+                install_ingress_for_kind()
-                if not is_ingress_running():
+                # Wait for ingress to start
-                    install_ingress_for_kind(self.cluster_info.spec.get_acme_email())
+                # (deployment provisioning will fail unless this is done)
-                    # Wait for ingress to start
+                wait_for_ingress_in_kind()
                    # (deployment provisioning will fail unless this is done)
                    wait_for_ingress_in_kind()
                # Create RuntimeClass if unlimited_memlock is enabled
                if self.cluster_info.spec.get_unlimited_memlock():
                    _create_runtime_class(
@ -377,11 +315,6 @@ class K8sDeployer(Deployer):
        else:
            print("Dry run mode enabled, skipping k8s API connect")
        # Create registry secret if configured
        from stack_orchestrator.deploy.deployment_create import create_registry_secret
        create_registry_secret(self.cluster_info.spec, self.cluster_info.app_name)
        self._create_volume_data()
        self._create_deployment()
@ -426,30 +359,107 @@ class K8sDeployer(Deployer):
                    print("NodePort created:")
                    print(f"{nodeport_resp}")
-    def down(self, timeout, volumes, skip_cluster_management):
+    def down(self, timeout, volumes, skip_cluster_management):  # noqa: C901
        self.skip_cluster_management = skip_cluster_management
        self.connect_api()
        # Delete the k8s objects
        # PersistentVolumes are cluster-scoped (not namespaced), so delete by label
        if volumes:
-            try:
+            # Create the host-path-mounted PVs for this deployment
-                pvs = self.core_api.list_persistent_volume(
+            pvs = self.cluster_info.get_pvs()
-                    label_selector=f"app={self.cluster_info.app_name}"
+            for pv in pvs:
                )
                for pv in pvs.items:
                    if opts.o.debug:
                        print(f"Deleting PV: {pv.metadata.name}")
                    try:
                        self.core_api.delete_persistent_volume(name=pv.metadata.name)
                    except ApiException as e:
                        _check_delete_exception(e)
            except ApiException as e:
                if opts.o.debug:
-                    print(f"Error listing PVs: {e}")
+                    print(f"Deleting this pv: {pv}")
                try:
                    pv_resp = self.core_api.delete_persistent_volume(
                        name=pv.metadata.name
                    )
                    if opts.o.debug:
                        print("PV deleted:")
                        print(f"{pv_resp}")
                except ApiException as e:
                    _check_delete_exception(e)
-        # Delete the deployment namespace - this cascades to all namespaced resources
+            # Figure out the PVCs for this deployment
-        # (PVCs, ConfigMaps, Deployments, Services, Ingresses, etc.)
+            pvcs = self.cluster_info.get_pvcs()
-        self._delete_namespace()
+            for pvc in pvcs:
                if opts.o.debug:
                    print(f"Deleting this pvc: {pvc}")
                try:
                    pvc_resp = self.core_api.delete_namespaced_persistent_volume_claim(
                        name=pvc.metadata.name, namespace=self.k8s_namespace
                    )
                    if opts.o.debug:
                        print("PVCs deleted:")
                        print(f"{pvc_resp}")
                except ApiException as e:
                    _check_delete_exception(e)
        # Figure out the ConfigMaps for this deployment
        cfg_maps = self.cluster_info.get_configmaps()
        for cfg_map in cfg_maps:
            if opts.o.debug:
                print(f"Deleting this ConfigMap: {cfg_map}")
            try:
                cfg_map_resp = self.core_api.delete_namespaced_config_map(
                    name=cfg_map.metadata.name, namespace=self.k8s_namespace
                )
                if opts.o.debug:
                    print("ConfigMap deleted:")
                    print(f"{cfg_map_resp}")
            except ApiException as e:
                _check_delete_exception(e)
        deployment = self.cluster_info.get_deployment()
        if opts.o.debug:
            print(f"Deleting this deployment: {deployment}")
        if deployment and deployment.metadata and deployment.metadata.name:
            try:
                self.apps_api.delete_namespaced_deployment(
                    name=deployment.metadata.name, namespace=self.k8s_namespace
                )
            except ApiException as e:
                _check_delete_exception(e)
        service = self.cluster_info.get_service()
        if opts.o.debug:
            print(f"Deleting service: {service}")
        if service and service.metadata and service.metadata.name:
            try:
                self.core_api.delete_namespaced_service(
                    namespace=self.k8s_namespace, name=service.metadata.name
                )
            except ApiException as e:
                _check_delete_exception(e)
        ingress = self.cluster_info.get_ingress(use_tls=not self.is_kind())
        if ingress and ingress.metadata and ingress.metadata.name:
            if opts.o.debug:
                print(f"Deleting this ingress: {ingress}")
            try:
                self.networking_api.delete_namespaced_ingress(
                    name=ingress.metadata.name, namespace=self.k8s_namespace
                )
            except ApiException as e:
                _check_delete_exception(e)
        else:
            if opts.o.debug:
                print("No ingress to delete")
        nodeports: List[client.V1Service] = self.cluster_info.get_nodeports()
        for nodeport in nodeports:
            if opts.o.debug:
                print(f"Deleting this nodeport: {nodeport}")
            if nodeport.metadata and nodeport.metadata.name:
                try:
                    self.core_api.delete_namespaced_service(
                        namespace=self.k8s_namespace, name=nodeport.metadata.name
                    )
                except ApiException as e:
                    _check_delete_exception(e)
        else:
            if opts.o.debug:
                print("No nodeport to delete")
        if self.is_kind() and not self.skip_cluster_management:
            # Destroy the kind cluster
@ -587,7 +597,7 @@ class K8sDeployer(Deployer):
                log_data = ""
                for container in containers:
                    container_log = self.core_api.read_namespaced_pod_log(
-                        k8s_pod_name, namespace=self.k8s_namespace, container=container
+                        k8s_pod_name, namespace="default", container=container
                    )
                    container_log_lines = container_log.splitlines()
                    for line in container_log_lines:
--- a/stack_orchestrator/deploy/k8s/helpers.py
+++ b/stack_orchestrator/deploy/k8s/helpers.py
@ -14,13 +14,11 @@
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 from kubernetes import client, utils, watch
 from kubernetes.client.exceptions import ApiException
 import os
 from pathlib import Path
 import subprocess
 import re
 from typing import Set, Mapping, List, Optional, cast
 import yaml
 from stack_orchestrator.util import get_k8s_dir, error_exit
 from stack_orchestrator.opts import opts
@ -98,227 +96,16 @@ def _run_command(command: str):
    return result
 def _get_etcd_host_path_from_kind_config(config_file: str) -> Optional[str]:
    """Extract etcd host path from kind config extraMounts."""
    import yaml
    try:
        with open(config_file, "r") as f:
            config = yaml.safe_load(f)
    except Exception:
        return None
    nodes = config.get("nodes", [])
    for node in nodes:
        extra_mounts = node.get("extraMounts", [])
        for mount in extra_mounts:
            if mount.get("containerPath") == "/var/lib/etcd":
                return mount.get("hostPath")
    return None
 def _clean_etcd_keeping_certs(etcd_path: str) -> bool:
    """Clean persisted etcd, keeping only TLS certificates.
    When etcd is persisted and a cluster is recreated, kind tries to install
    resources fresh but they already exist. Instead of trying to delete
    specific stale resources (blacklist), we keep only the valuable data
    (caddy TLS certs) and delete everything else (whitelist approach).
    The etcd image is distroless (no shell), so we extract the statically-linked
    etcdctl binary and run it from alpine which has shell support.
    Returns True if cleanup succeeded, False if no action needed or failed.
    """
    db_path = Path(etcd_path) / "member" / "snap" / "db"
    # Check existence using docker since etcd dir is root-owned
    check_cmd = (
        f"docker run --rm -v {etcd_path}:/etcd:ro alpine:3.19 "
        "test -f /etcd/member/snap/db"
    )
    check_result = subprocess.run(check_cmd, shell=True, capture_output=True)
    if check_result.returncode != 0:
        if opts.o.debug:
            print(f"No etcd snapshot at {db_path}, skipping cleanup")
        return False
    if opts.o.debug:
        print(f"Cleaning persisted etcd at {etcd_path}, keeping only TLS certs")
    etcd_image = "gcr.io/etcd-development/etcd:v3.5.9"
    temp_dir = "/tmp/laconic-etcd-cleanup"
    # Whitelist: prefixes to KEEP - everything else gets deleted
    keep_prefixes = "/registry/secrets/caddy-system"
    # The etcd image is distroless (no shell). We extract the statically-linked
    # etcdctl binary and run it from alpine which has shell + jq support.
    cleanup_script = f"""
        set -e
        ALPINE_IMAGE="alpine:3.19"
        # Cleanup previous runs
        docker rm -f laconic-etcd-cleanup 2>/dev/null || true
        docker rm -f etcd-extract 2>/dev/null || true
        docker run --rm -v /tmp:/tmp $ALPINE_IMAGE rm -rf {temp_dir}
        # Create temp dir
        docker run --rm -v /tmp:/tmp $ALPINE_IMAGE mkdir -p {temp_dir}
        # Extract etcdctl binary (it's statically linked)
        docker create --name etcd-extract {etcd_image}
        docker cp etcd-extract:/usr/local/bin/etcdctl /tmp/etcdctl-bin
        docker rm etcd-extract
        docker run --rm -v /tmp/etcdctl-bin:/src:ro -v {temp_dir}:/dst $ALPINE_IMAGE \
            sh -c "cp /src /dst/etcdctl && chmod +x /dst/etcdctl"
        # Copy db to temp location
        docker run --rm \
            -v {etcd_path}:/etcd:ro \
            -v {temp_dir}:/tmp-work \
            $ALPINE_IMAGE cp /etcd/member/snap/db /tmp-work/etcd-snapshot.db
        # Restore snapshot
        docker run --rm -v {temp_dir}:/work {etcd_image} \
            etcdutl snapshot restore /work/etcd-snapshot.db \
                --data-dir=/work/etcd-data --skip-hash-check 2>/dev/null
        # Start temp etcd (runs the etcd binary, no shell needed)
        docker run -d --name laconic-etcd-cleanup \
            -v {temp_dir}/etcd-data:/etcd-data \
            -v {temp_dir}:/backup \
            {etcd_image} etcd \
                --data-dir=/etcd-data \
                --listen-client-urls=http://0.0.0.0:2379 \
                --advertise-client-urls=http://localhost:2379
        sleep 3
        # Use alpine with extracted etcdctl to run commands (alpine has shell + jq)
        # Export caddy secrets
        docker run --rm \
            -v {temp_dir}:/backup \
            --network container:laconic-etcd-cleanup \
            $ALPINE_IMAGE sh -c \
            '/backup/etcdctl get --prefix "{keep_prefixes}" -w json \
                > /backup/kept.json 2>/dev/null || echo "{{}}" > /backup/kept.json'
        # Delete ALL registry keys
        docker run --rm \
            -v {temp_dir}:/backup \
            --network container:laconic-etcd-cleanup \
            $ALPINE_IMAGE /backup/etcdctl del --prefix /registry
        # Restore kept keys using jq
        docker run --rm \
            -v {temp_dir}:/backup \
            --network container:laconic-etcd-cleanup \
            $ALPINE_IMAGE sh -c '
                apk add --no-cache jq >/dev/null 2>&1
                jq -r ".kvs[] | @base64" /backup/kept.json 2>/dev/null | \
                while read encoded; do
                    key=$(echo $encoded | base64 -d | jq -r ".key" | base64 -d)
                    val=$(echo $encoded | base64 -d | jq -r ".value" | base64 -d)
                    echo "$val" | /backup/etcdctl put "$key"
                done
            ' || true
        # Save cleaned snapshot
        docker exec laconic-etcd-cleanup \
            etcdctl snapshot save /etcd-data/cleaned-snapshot.db
        docker stop laconic-etcd-cleanup
        docker rm laconic-etcd-cleanup
        # Restore to temp location first to verify it works
        docker run --rm \
            -v {temp_dir}/etcd-data/cleaned-snapshot.db:/data/db:ro \
            -v {temp_dir}:/restore \
            {etcd_image} \
            etcdutl snapshot restore /data/db --data-dir=/restore/new-etcd \
            --skip-hash-check 2>/dev/null
        # Create timestamped backup of original (kept forever)
        TIMESTAMP=$(date +%Y%m%d-%H%M%S)
        docker run --rm -v {etcd_path}:/etcd $ALPINE_IMAGE \
            cp -a /etcd/member /etcd/member.backup-$TIMESTAMP
        # Replace original with cleaned version
        docker run --rm -v {etcd_path}:/etcd -v {temp_dir}:/tmp-work $ALPINE_IMAGE \
            sh -c "rm -rf /etcd/member && mv /tmp-work/new-etcd/member /etcd/member"
        # Cleanup temp files (but NOT the timestamped backup in etcd_path)
        docker run --rm -v /tmp:/tmp $ALPINE_IMAGE rm -rf {temp_dir}
        rm -f /tmp/etcdctl-bin
    """
    result = subprocess.run(cleanup_script, shell=True, capture_output=True, text=True)
    if result.returncode != 0:
        if opts.o.debug:
            print(f"Warning: etcd cleanup failed: {result.stderr}")
        return False
    if opts.o.debug:
        print("Cleaned etcd, kept only TLS certificates")
    return True
 def create_cluster(name: str, config_file: str):
    """Create or reuse the single kind cluster for this host.
    There is only one kind cluster per host by design. Multiple deployments
    share this cluster. If a cluster already exists, it is reused.
    Args:
        name: Cluster name (used only when creating the first cluster)
        config_file: Path to kind config file (used only when creating)
    Returns:
        The name of the cluster being used
    """
    existing = get_kind_cluster()
    if existing:
        print(f"Using existing cluster: {existing}")
        return existing
    # Clean persisted etcd, keeping only TLS certificates
    etcd_path = _get_etcd_host_path_from_kind_config(config_file)
    if etcd_path:
        _clean_etcd_keeping_certs(etcd_path)
    print(f"Creating new cluster: {name}")
    result = _run_command(f"kind create cluster --name {name} --config {config_file}")
    if result.returncode != 0:
        raise DeployerException(f"kind create cluster failed: {result}")
    return name
 def destroy_cluster(name: str):
    _run_command(f"kind delete cluster --name {name}")
 def is_ingress_running() -> bool:
    """Check if the Caddy ingress controller is already running in the cluster."""
    try:
        core_v1 = client.CoreV1Api()
        pods = core_v1.list_namespaced_pod(
            namespace="caddy-system",
            label_selector=(
                "app.kubernetes.io/name=caddy-ingress-controller,"
                "app.kubernetes.io/component=controller"
            ),
        )
        for pod in pods.items:
            if pod.status and pod.status.container_statuses:
                if pod.status.container_statuses[0].ready is True:
                    if opts.o.debug:
                        print("Caddy ingress controller already running")
                    return True
        return False
    except ApiException:
        return False
 def wait_for_ingress_in_kind():
    core_v1 = client.CoreV1Api()
    for i in range(20):
@ -345,7 +132,7 @@ def wait_for_ingress_in_kind():
    error_exit("ERROR: Timed out waiting for Caddy ingress to become ready")
-def install_ingress_for_kind(acme_email: str = ""):
+def install_ingress_for_kind():
    api_client = client.ApiClient()
    ingress_install = os.path.abspath(
        get_k8s_dir().joinpath(
@ -354,34 +141,7 @@ def install_ingress_for_kind(acme_email: str = ""):
    )
    if opts.o.debug:
        print("Installing Caddy ingress controller in kind cluster")
-
+    utils.create_from_yaml(api_client, yaml_file=ingress_install)
    # Template the YAML with email before applying
    with open(ingress_install) as f:
        yaml_content = f.read()
    if acme_email:
        yaml_content = yaml_content.replace('email: ""', f'email: "{acme_email}"')
        if opts.o.debug:
            print(f"Configured Caddy with ACME email: {acme_email}")
    # Apply templated YAML
    yaml_objects = list(yaml.safe_load_all(yaml_content))
    utils.create_from_yaml(api_client, yaml_objects=yaml_objects)
    # Patch ConfigMap with ACME email if provided
    if acme_email:
        if opts.o.debug:
            print(f"Configuring ACME email: {acme_email}")
        core_api = client.CoreV1Api()
        configmap = core_api.read_namespaced_config_map(
            name="caddy-ingress-controller-configmap", namespace="caddy-system"
        )
        configmap.data["email"] = acme_email
        core_api.patch_namespaced_config_map(
            name="caddy-ingress-controller-configmap",
            namespace="caddy-system",
            body=configmap,
        )
 def load_images_into_kind(kind_cluster_name: str, image_set: Set[str]):
@ -564,25 +324,6 @@ def _generate_kind_mounts(parsed_pod_files, deployment_dir, deployment_context):
    volume_host_path_map = _get_host_paths_for_volumes(deployment_context)
    seen_host_path_mounts = set()  # Track to avoid duplicate mounts
    # Cluster state backup for offline data recovery (unique per deployment)
    # etcd contains all k8s state; PKI certs needed to decrypt etcd offline
    deployment_id = deployment_context.id
    backup_subdir = f"cluster-backups/{deployment_id}"
    etcd_host_path = _make_absolute_host_path(
        Path(f"./data/{backup_subdir}/etcd"), deployment_dir
    )
    volume_definitions.append(
        f"  - hostPath: {etcd_host_path}\n" f"    containerPath: /var/lib/etcd\n"
    )
    pki_host_path = _make_absolute_host_path(
        Path(f"./data/{backup_subdir}/pki"), deployment_dir
    )
    volume_definitions.append(
        f"  - hostPath: {pki_host_path}\n" f"    containerPath: /etc/kubernetes/pki\n"
    )
    # Note these paths are relative to the location of the pod files (at present)
    # So we need to fix up to make them correct and absolute because kind assumes
    # relative to the cwd.
@ -942,41 +683,6 @@ def envs_from_compose_file(
    return result
 def translate_sidecar_service_names(
    envs: Mapping[str, str], sibling_service_names: List[str]
 ) -> Mapping[str, str]:
    """Translate docker-compose service names to localhost for sidecar containers.
    In docker-compose, services can reference each other by name (e.g., 'db:5432').
    In Kubernetes, when multiple containers are in the same pod (sidecars), they
    share the same network namespace and must use 'localhost' instead.
    This function replaces service name references with 'localhost' in env values.
    """
    import re
    if not sibling_service_names:
        return envs
    result = {}
    for env_var, env_val in envs.items():
        if env_val is None:
            result[env_var] = env_val
            continue
        new_val = str(env_val)
        for service_name in sibling_service_names:
            # Match service name followed by optional port (e.g., 'db:5432', 'db')
            # Handle URLs like: postgres://user:pass@db:5432/dbname
            # and simple refs like: db:5432 or just db
            pattern = rf"\b{re.escape(service_name)}(:\d+)?\b"
            new_val = re.sub(pattern, lambda m: f'localhost{m.group(1) or ""}', new_val)
        result[env_var] = new_val
    return result
 def envs_from_environment_variables_map(
    map: Mapping[str, str]
 ) -> List[client.V1EnvVar]:
--- a/stack_orchestrator/deploy/spec.py
+++ b/stack_orchestrator/deploy/spec.py
@ -98,17 +98,6 @@ class Spec:
    def get_image_registry(self):
        return self.obj.get(constants.image_registry_key)
    def get_image_registry_config(self) -> typing.Optional[typing.Dict]:
        """Returns registry auth config: {server, username, token-env}.
        Used for private container registries like GHCR. The token-env field
        specifies an environment variable containing the API token/PAT.
        Note: Uses 'registry-credentials' key to avoid collision with
        'image-registry' key which is for pushing images.
        """
        return self.obj.get("registry-credentials")
    def get_volumes(self):
        return self.obj.get(constants.volumes_key, {})
@ -120,27 +109,6 @@ class Spec:
            self.obj.get(constants.resources_key, {}).get("containers", {})
        )
    def get_container_resources_for(
        self, container_name: str
    ) -> typing.Optional[Resources]:
        """Look up per-container resource overrides from spec.yml.
        Checks resources.containers.<container_name> in the spec. Returns None
        if no per-container override exists (caller falls back to other sources).
        """
        containers_block = self.obj.get(constants.resources_key, {}).get(
            "containers", {}
        )
        if container_name in containers_block:
            entry = containers_block[container_name]
            # Only treat it as a per-container override if it's a dict with
            # reservations/limits nested inside (not a top-level global key)
            if isinstance(entry, dict) and (
                "reservations" in entry or "limits" in entry
            ):
                return Resources(entry)
        return None
    def get_volume_resources(self):
        return Resources(
            self.obj.get(constants.resources_key, {}).get(constants.volumes_key, {})
@ -211,9 +179,6 @@ class Spec:
    def get_deployment_type(self):
        return self.obj.get(constants.deploy_to_key)
    def get_acme_email(self):
        return self.obj.get(constants.network_key, {}).get(constants.acme_email_key, "")
    def is_kubernetes_deployment(self):
        return self.get_deployment_type() in [
            constants.k8s_kind_deploy_type,
--- a/tests/scripts/run-test-local.sh
+++ b/tests/scripts/run-test-local.sh
@ -1,53 +0,0 @@
 #!/bin/bash
 # Run a test suite locally in an isolated venv.
 #
 # Usage:
 #   ./tests/scripts/run-test-local.sh <test-script>
 #
 # Examples:
 #   ./tests/scripts/run-test-local.sh tests/webapp-test/run-webapp-test.sh
 #   ./tests/scripts/run-test-local.sh tests/smoke-test/run-smoke-test.sh
 #   ./tests/scripts/run-test-local.sh tests/k8s-deploy/run-deploy-test.sh
 #
 # The script creates a temporary venv, installs shiv, builds the laconic-so
 # package, runs the requested test, then cleans up.
 set -euo pipefail
 if [ $# -lt 1 ]; then
  echo "Usage: $0 <test-script> [args...]"
  exit 1
 fi
 TEST_SCRIPT="$1"
 shift
 if [ ! -f "$TEST_SCRIPT" ]; then
  echo "Error: $TEST_SCRIPT not found"
  exit 1
 fi
 REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
 VENV_DIR=$(mktemp -d /tmp/so-test-XXXXXX)
 cleanup() {
  echo "Cleaning up venv: $VENV_DIR"
  rm -rf "$VENV_DIR"
 }
 trap cleanup EXIT
 cd "$REPO_DIR"
 echo "==> Creating venv in $VENV_DIR"
 python3 -m venv "$VENV_DIR"
 source "$VENV_DIR/bin/activate"
 echo "==> Installing shiv"
 pip install -q shiv
 echo "==> Building laconic-so package"
 ./scripts/create_build_tag_file.sh
 ./scripts/build_shiv_package.sh
 echo "==> Running: $TEST_SCRIPT $*"
 exec "./$TEST_SCRIPT" "$@"