add e2e test for hook re-copy on restart
Covers two scenarios on a single Kind cluster: - Single-repo: deploy create copies commands.py into hooks/, deployment start runs it, mutating the stack-source working tree to v2 + deployment restart re-copies and re-executes v2. - Multi-repo: stack with two pod repos produces hooks/commands_0.py + commands_1.py, deployment start invokes both pod start() hooks. The test stages stack files into a temp git clone (bare + working) so restart's git pull has a real upstream. busybox pods keep the harness trivial. Phase 2 uses kubectl wait directly because deployment ps's substring filter (deploy_k8s.py:1366) doesn't list multi-pod stacks. Also tightens the _copy_hooks docstring to spell out that only call_stack_deploy_start loads from the copied location. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>pull/750/head
pranav
parent
befb548a6e
commit
d141546cdd
|
|
@ -16,6 +16,7 @@ on:
|
|||
- '.github/workflows/triggers/test-k8s-deploy'
|
||||
- '.github/workflows/test-k8s-deploy.yml'
|
||||
- 'tests/k8s-deploy/run-deploy-test.sh'
|
||||
- 'tests/k8s-deploy/run-restart-hook-test.sh'
|
||||
schedule:
|
||||
- cron: '3 15 * * *'
|
||||
|
||||
|
|
@ -46,3 +47,5 @@ jobs:
|
|||
run: ./tests/scripts/install-kubectl.sh
|
||||
- name: "Run k8s deployment test"
|
||||
run: ./tests/k8s-deploy/run-deploy-test.sh
|
||||
- name: "Run restart-hook k8s deployment test"
|
||||
run: ./tests/k8s-deploy/run-restart-hook-test.sh
|
||||
|
|
|
|||
|
|
@ -0,0 +1,5 @@
|
|||
services:
|
||||
test-restart-hook:
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "echo started && sleep infinity"]
|
||||
restart: always
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# test-restart-hook-multi
|
||||
|
||||
E2E test stack used by `tests/k8s-deploy/run-restart-hook-test.sh` to cover the
|
||||
multi-repo case: `pods:` references two pod repos, each shipping its own
|
||||
`deploy/commands.py`. `deploy create` should produce
|
||||
`<deployment>/hooks/commands_0.py` and `<deployment>/hooks/commands_1.py`, and
|
||||
`deployment start` should invoke both `start()` hooks (each writes its own
|
||||
marker file so neither overwrites the other).
|
||||
|
||||
The pod repos themselves are created by the test script as bare-repo +
|
||||
working-clone pairs under `$CERC_REPO_BASE_DIR/test-restart-hook-pod-{a,b}`;
|
||||
they are not committed to this repository. Each pod repo ships its own
|
||||
`docker-compose.yml` (resolved by `get_pod_file_path` for dict-form pods) and
|
||||
`stack/deploy/commands.py` — the stack repo only owns `stack.yml`.
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
version: "1.0"
|
||||
name: test-restart-hook-multi
|
||||
description: "E2E test stack: verifies hooks/commands_*.py multi-repo file naming and multi-hook invocation"
|
||||
pods:
|
||||
- name: test-restart-hook-multi-a
|
||||
repository: test-restart-hook-pod-a
|
||||
path: .
|
||||
- name: test-restart-hook-multi-b
|
||||
repository: test-restart-hook-pod-b
|
||||
path: .
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
# test-restart-hook
|
||||
|
||||
E2E test stack used by `tests/k8s-deploy/run-restart-hook-test.sh`.
|
||||
|
||||
The stack ships a single `start()` hook that writes a versioned marker file
|
||||
into the deployment directory. The test:
|
||||
|
||||
1. `deploy create` → asserts `commands.py` was copied into `<deployment>/hooks/`.
|
||||
2. `deployment start` → asserts the marker file contains the v1 string.
|
||||
3. Modifies `commands.py` in the stack-source working tree (v1 → v2).
|
||||
4. `deployment restart` → asserts the new commands.py was re-copied into
|
||||
`<deployment>/hooks/` and the marker file now contains the v2 string.
|
||||
|
||||
The pod uses a public `busybox` image that just sleeps; the start hook is the
|
||||
only thing under test.
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
# Copyright © 2026 Vulcanize
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
|
||||
|
||||
from stack_orchestrator.util import get_yaml
|
||||
from stack_orchestrator.deploy.deployment_context import DeploymentContext
|
||||
|
||||
default_spec_file_content = ""
|
||||
|
||||
|
||||
def init(command_context):
|
||||
return get_yaml().load(default_spec_file_content)
|
||||
|
||||
|
||||
def start(deployment_context: DeploymentContext):
|
||||
# Writes a marker file the e2e test asserts on. The test flips the
|
||||
# literal below from "v1" to "v2" in the stack-source working tree
|
||||
# before running 'deployment restart' to verify the updated hook is
|
||||
# copied into deployment_dir/hooks/ and re-executed.
|
||||
marker = deployment_context.deployment_dir / "start-hook-marker"
|
||||
marker.write_text("v1")
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
version: "1.0"
|
||||
name: test-restart-hook
|
||||
description: "E2E test stack: verifies hooks/commands.py is re-copied and re-loaded on deployment restart"
|
||||
pods:
|
||||
- test-restart-hook
|
||||
|
|
@ -83,9 +83,7 @@ class DeploymentContext:
|
|||
# Fallback to cluster-id for deployments created before the
|
||||
# deployment-id field was introduced. Keeps existing resource
|
||||
# names stable across this upgrade.
|
||||
self.deployment_id = obj.get(
|
||||
constants.deployment_id_key, self.id
|
||||
)
|
||||
self.deployment_id = obj.get(constants.deployment_id_key, self.id)
|
||||
# Handle the case of a legacy deployment with no file
|
||||
# Code below is intended to match the output from _make_default_cluster_name()
|
||||
# TODO: remove when we no longer need to support legacy deployments
|
||||
|
|
|
|||
|
|
@ -380,9 +380,7 @@ def _validate_host_path_mounts(parsed_pod_file, pod_name, pod_file_path):
|
|||
"content at runtime.\n\n"
|
||||
"See docs/deployment_patterns.md."
|
||||
)
|
||||
total = sum(
|
||||
p.stat().st_size for p in entries if p.is_file()
|
||||
)
|
||||
total = sum(p.stat().st_size for p in entries if p.is_file())
|
||||
if total > _HOST_PATH_CONFIGMAP_BUDGET_BYTES:
|
||||
raise DeployerException(
|
||||
f"Directory host-path bind '{volume_str}' in "
|
||||
|
|
@ -1113,13 +1111,25 @@ def _safe_copy_tree(src: Path, dst: Path, exclude_patterns: Optional[List[str]]
|
|||
|
||||
|
||||
def _copy_hooks(stack_name: str, target_dir: Path):
|
||||
"""Copy commands.py hooks into deployment_dir/hooks/ so the deployment is self-sufficient.
|
||||
"""Copy commands.py hooks into deployment_dir/hooks/ for self-sufficiency.
|
||||
|
||||
Single repo: hooks/commands.py
|
||||
Multi-repo: hooks/commands_0.py, hooks/commands_1.py, ... (not tested)
|
||||
Multi-repo: hooks/commands_0.py, hooks/commands_1.py, ... — indexed by
|
||||
plugin path order.
|
||||
|
||||
Note: the whole commands.py file is copied (init/setup/create/start), but
|
||||
at runtime only call_stack_deploy_start loads from this copied location.
|
||||
call_stack_deploy_init, call_stack_deploy_setup, and call_stack_deploy_create
|
||||
still resolve commands.py from the live stack source via
|
||||
get_plugin_code_paths — they only run at deploy create time when the source
|
||||
is guaranteed to be present, so they don't need to be self-sufficient.
|
||||
"""
|
||||
plugin_paths = get_plugin_code_paths(stack_name)
|
||||
sources = [p.joinpath("deploy", "commands.py") for p in plugin_paths if p.joinpath("deploy", "commands.py").exists()]
|
||||
sources = [
|
||||
p.joinpath("deploy", "commands.py")
|
||||
for p in plugin_paths
|
||||
if p.joinpath("deploy", "commands.py").exists()
|
||||
]
|
||||
if not sources:
|
||||
return
|
||||
hooks_dir = target_dir / "hooks"
|
||||
|
|
@ -1271,7 +1281,9 @@ def _write_deployment_files(
|
|||
else:
|
||||
source_config_dir = resolve_config_dir(stack_name, configmap_name)
|
||||
if os.path.exists(source_config_dir):
|
||||
destination_config_dir = target_dir.joinpath("configmaps", configmap_name)
|
||||
destination_config_dir = target_dir.joinpath(
|
||||
"configmaps", configmap_name
|
||||
)
|
||||
copytree(source_config_dir, destination_config_dir, dirs_exist_ok=True)
|
||||
|
||||
# Copy the job files into the target dir
|
||||
|
|
@ -1284,9 +1296,7 @@ def _write_deployment_files(
|
|||
if job_file_path and job_file_path.exists():
|
||||
parsed_job_file = yaml.load(open(job_file_path, "r"))
|
||||
if parsed_spec.is_kubernetes_deployment():
|
||||
_validate_host_path_mounts(
|
||||
parsed_job_file, job, job_file_path
|
||||
)
|
||||
_validate_host_path_mounts(parsed_job_file, job, job_file_path)
|
||||
_fixup_pod_file(parsed_job_file, parsed_spec, destination_compose_dir)
|
||||
with open(
|
||||
destination_compose_jobs_dir.joinpath(
|
||||
|
|
|
|||
|
|
@ -479,9 +479,7 @@ class ClusterInfo:
|
|||
if sanitized in seen:
|
||||
continue
|
||||
seen.add(sanitized)
|
||||
abs_src = resolve_host_path_for_kind(
|
||||
src, deployment_dir
|
||||
)
|
||||
abs_src = resolve_host_path_for_kind(src, deployment_dir)
|
||||
data = self._read_host_path_source(abs_src, mount_string)
|
||||
cm = client.V1ConfigMap(
|
||||
metadata=client.V1ObjectMeta(
|
||||
|
|
@ -495,9 +493,7 @@ class ClusterInfo:
|
|||
result.append(cm)
|
||||
return result
|
||||
|
||||
def _read_host_path_source(
|
||||
self, abs_src: Path, mount_string: str
|
||||
) -> dict:
|
||||
def _read_host_path_source(self, abs_src: Path, mount_string: str) -> dict:
|
||||
"""Read file or flat-directory content for a host-path ConfigMap.
|
||||
|
||||
Validates shape at read time as a defensive second check — the
|
||||
|
|
@ -517,9 +513,7 @@ class ClusterInfo:
|
|||
for entry in abs_src.iterdir():
|
||||
if entry.is_file():
|
||||
with open(entry, "rb") as f:
|
||||
data[entry.name] = base64.b64encode(f.read()).decode(
|
||||
"ASCII"
|
||||
)
|
||||
data[entry.name] = base64.b64encode(f.read()).decode("ASCII")
|
||||
return data
|
||||
|
||||
def get_pvs(self):
|
||||
|
|
@ -711,9 +705,7 @@ class ClusterInfo:
|
|||
volume_mounts = volume_mounts_for_service(
|
||||
parsed_yaml_map,
|
||||
service_name,
|
||||
Path(self.spec.file_path).parent
|
||||
if self.spec.file_path
|
||||
else None,
|
||||
Path(self.spec.file_path).parent if self.spec.file_path else None,
|
||||
)
|
||||
# Handle command/entrypoint from compose file
|
||||
# In docker-compose: entrypoint -> k8s command, command -> k8s args
|
||||
|
|
@ -1021,9 +1013,7 @@ class ClusterInfo:
|
|||
metadata=client.V1ObjectMeta(
|
||||
name=deployment_name,
|
||||
labels=self._stack_labels(
|
||||
{"app.kubernetes.io/component": pod_name}
|
||||
if multi_pod
|
||||
else None
|
||||
{"app.kubernetes.io/component": pod_name} if multi_pod else None
|
||||
),
|
||||
),
|
||||
spec=spec,
|
||||
|
|
@ -1071,9 +1061,7 @@ class ClusterInfo:
|
|||
container_ports[container].add(port)
|
||||
if maintenance_svc and ":" in maintenance_svc:
|
||||
maint_container, maint_port_str = maintenance_svc.split(":", 1)
|
||||
container_ports.setdefault(maint_container, set()).add(
|
||||
int(maint_port_str)
|
||||
)
|
||||
container_ports.setdefault(maint_container, set()).add(int(maint_port_str))
|
||||
|
||||
# Build map: pod_file -> set of service names in that pod
|
||||
pod_services_map: dict = {}
|
||||
|
|
|
|||
|
|
@ -219,10 +219,7 @@ class K8sDeployer(Deployer):
|
|||
)
|
||||
self.core_api.create_namespace(body=ns)
|
||||
if opts.o.debug:
|
||||
print(
|
||||
f"Created namespace {self.k8s_namespace} "
|
||||
f"owned by {my_dir}"
|
||||
)
|
||||
print(f"Created namespace {self.k8s_namespace} " f"owned by {my_dir}")
|
||||
return
|
||||
|
||||
annotations = (existing.metadata.annotations or {}) if existing.metadata else {}
|
||||
|
|
@ -1025,9 +1022,7 @@ class K8sDeployer(Deployer):
|
|||
|
||||
call_stack_deploy_start(self.deployment_context)
|
||||
|
||||
def down(
|
||||
self, timeout, volumes, skip_cluster_management, delete_namespace=False
|
||||
):
|
||||
def down(self, timeout, volumes, skip_cluster_management, delete_namespace=False):
|
||||
"""Tear down stack-labeled resources. Phases:
|
||||
|
||||
1. Delete namespaced resources (if namespace still exists).
|
||||
|
|
@ -1221,34 +1216,68 @@ class K8sDeployer(Deployer):
|
|||
listers = []
|
||||
if namespace_present:
|
||||
listers += [
|
||||
("deployment", lambda: self.apps_api.list_namespaced_deployment(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
("ingress", lambda: self.networking_api.list_namespaced_ingress(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
("job", lambda: self.batch_api.list_namespaced_job(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
("service", lambda: self.core_api.list_namespaced_service(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
("configmap", lambda: self.core_api.list_namespaced_config_map(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
("secret", lambda: self.core_api.list_namespaced_secret(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
("pod", lambda: self.core_api.list_namespaced_pod(
|
||||
namespace=namespace, label_selector=selector)),
|
||||
(
|
||||
"deployment",
|
||||
lambda: self.apps_api.list_namespaced_deployment(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
(
|
||||
"ingress",
|
||||
lambda: self.networking_api.list_namespaced_ingress(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
(
|
||||
"job",
|
||||
lambda: self.batch_api.list_namespaced_job(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
(
|
||||
"service",
|
||||
lambda: self.core_api.list_namespaced_service(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
(
|
||||
"configmap",
|
||||
lambda: self.core_api.list_namespaced_config_map(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
(
|
||||
"secret",
|
||||
lambda: self.core_api.list_namespaced_secret(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
(
|
||||
"pod",
|
||||
lambda: self.core_api.list_namespaced_pod(
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
),
|
||||
]
|
||||
if delete_volumes:
|
||||
listers.append(
|
||||
("persistentvolumeclaim",
|
||||
(
|
||||
"persistentvolumeclaim",
|
||||
lambda: self.core_api.list_namespaced_persistent_volume_claim(
|
||||
namespace=namespace, label_selector=selector))
|
||||
namespace=namespace, label_selector=selector
|
||||
),
|
||||
)
|
||||
)
|
||||
# PVs are cluster-scoped — wait for them even when the namespace
|
||||
# is already gone (orphaned from a prior --delete-namespace).
|
||||
if delete_volumes:
|
||||
listers.append(
|
||||
("persistentvolume",
|
||||
(
|
||||
"persistentvolume",
|
||||
lambda: self.core_api.list_persistent_volume(
|
||||
label_selector=selector))
|
||||
label_selector=selector
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
def remaining():
|
||||
|
|
@ -1276,8 +1305,7 @@ class K8sDeployer(Deployer):
|
|||
left = remaining()
|
||||
if left:
|
||||
print(
|
||||
f"Warning: resources still present after {timeout_seconds}s: "
|
||||
f"{left}"
|
||||
f"Warning: resources still present after {timeout_seconds}s: " f"{left}"
|
||||
)
|
||||
|
||||
def status(self):
|
||||
|
|
|
|||
|
|
@ -207,9 +207,7 @@ def _install_caddy_cert_backup(
|
|||
print("No kind-mount-root configured; caddy cert backup disabled")
|
||||
return
|
||||
manifest = os.path.abspath(
|
||||
get_k8s_dir().joinpath(
|
||||
"components", "ingress", "caddy-cert-backup.yaml"
|
||||
)
|
||||
get_k8s_dir().joinpath("components", "ingress", "caddy-cert-backup.yaml")
|
||||
)
|
||||
with open(manifest) as f:
|
||||
objects = list(yaml.safe_load_all(f))
|
||||
|
|
@ -233,9 +231,7 @@ def _parse_kind_extra_mounts(config_file: str) -> List[Dict[str, str]]:
|
|||
host_path = m.get("hostPath")
|
||||
container_path = m.get("containerPath")
|
||||
if host_path and container_path:
|
||||
mounts.append(
|
||||
{"hostPath": host_path, "containerPath": container_path}
|
||||
)
|
||||
mounts.append({"hostPath": host_path, "containerPath": container_path})
|
||||
return mounts
|
||||
|
||||
|
||||
|
|
@ -500,12 +496,9 @@ def install_ingress_for_kind(
|
|||
continue
|
||||
if (
|
||||
obj.get("kind") == "Deployment"
|
||||
and obj.get("metadata", {}).get("name")
|
||||
== "caddy-ingress-controller"
|
||||
):
|
||||
for c in (
|
||||
obj["spec"]["template"]["spec"].get("containers") or []
|
||||
and obj.get("metadata", {}).get("name") == "caddy-ingress-controller"
|
||||
):
|
||||
for c in obj["spec"]["template"]["spec"].get("containers") or []:
|
||||
if c.get("name") == "caddy-ingress-controller":
|
||||
c["image"] = caddy_image
|
||||
if opts.o.debug:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,265 @@
|
|||
#!/usr/bin/env bash
|
||||
set -e
|
||||
if [ -n "$CERC_SCRIPT_DEBUG" ]; then
|
||||
set -x
|
||||
echo "Environment variables:"
|
||||
env
|
||||
fi
|
||||
|
||||
# Helper functions: TODO move into a separate file (mirrors run-deploy-test.sh:10).
|
||||
wait_for_pods_started () {
|
||||
local dir=$1
|
||||
for i in {1..50}
|
||||
do
|
||||
local ps_output=$( $TEST_TARGET_SO deployment --dir $dir ps )
|
||||
|
||||
if [[ "$ps_output" == *"Running containers:"* ]]; then
|
||||
return
|
||||
else
|
||||
sleep 5
|
||||
fi
|
||||
done
|
||||
echo "waiting for pods to start: FAILED"
|
||||
cleanup_and_exit
|
||||
}
|
||||
|
||||
# Multi-pod stacks aren't visible to 'deployment ps' (deploy_k8s.py:1366
|
||||
# filters by app_name-deployment substring, which doesn't match
|
||||
# laconic-<id>-<podname>-deployment-<hash> names). Wait via kubectl.
|
||||
wait_for_k8s_pods_ready () {
|
||||
local ns=$1
|
||||
local timeout=240
|
||||
local waited=0
|
||||
# First wait for at least one pod to appear in the namespace.
|
||||
while [ $waited -lt $timeout ]; do
|
||||
local count=$(kubectl get pods -n "$ns" --no-headers 2>/dev/null | wc -l)
|
||||
if [ "$count" -gt 0 ]; then
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
waited=$((waited + 2))
|
||||
done
|
||||
if ! kubectl wait --for=condition=Ready pod --all \
|
||||
-n "$ns" --timeout=$((timeout - waited))s 2>&1; then
|
||||
echo "kubectl wait pods ready: FAILED (ns=$ns)"
|
||||
kubectl get pods -n "$ns" 2>&1 || true
|
||||
kubectl describe pods -n "$ns" 2>&1 | tail -80 || true
|
||||
cleanup_and_exit
|
||||
fi
|
||||
}
|
||||
|
||||
# Best-effort full teardown so CI runners don't leak namespaces/PVs/clusters
|
||||
# between runs. Variables may be unset depending on which phase tripped.
|
||||
cleanup_and_exit () {
|
||||
if [ -n "$DEP1" ] && [ -d "$DEP1" ]; then
|
||||
$TEST_TARGET_SO deployment --dir $DEP1 \
|
||||
stop --delete-volumes --delete-namespace --skip-cluster-management || true
|
||||
fi
|
||||
if [ -n "$DEP2" ] && [ -d "$DEP2" ]; then
|
||||
$TEST_TARGET_SO deployment --dir $DEP2 \
|
||||
stop --delete-volumes --delete-namespace --perform-cluster-management || true
|
||||
fi
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Make a clone usable for `git commit` without touching the runner's global config.
|
||||
configure_git_identity () {
|
||||
local repo_dir=$1
|
||||
git -C $repo_dir config user.email "test@stack-orchestrator.test"
|
||||
git -C $repo_dir config user.name "test"
|
||||
}
|
||||
|
||||
TEST_TARGET_SO=$( ls -t1 ./package/laconic-so* | head -1 )
|
||||
echo "Testing this package: $TEST_TARGET_SO"
|
||||
|
||||
WORK_DIR=~/stack-orchestrator-test/restart-hook
|
||||
# Multi-repo pod working clones land here; resolved by get_plugin_code_paths.
|
||||
export CERC_REPO_BASE_DIR=$WORK_DIR/repo-base
|
||||
rm -rf $WORK_DIR
|
||||
mkdir -p $WORK_DIR $CERC_REPO_BASE_DIR
|
||||
|
||||
# Source location of the test stacks shipped in this checkout. The test stages
|
||||
# them into a temp git repo so 'deployment restart' (which runs 'git pull' on
|
||||
# the stack source) has a real repo to pull from.
|
||||
DATA_DIR=stack_orchestrator/data
|
||||
|
||||
# ============================================================================
|
||||
# Phase 1 — single-repo restart cycle. Verifies that:
|
||||
# * deploy create copies commands.py into <deployment>/hooks/
|
||||
# * deployment start runs the copied start() hook
|
||||
# * mutating the stack-source commands.py and running 'deployment restart'
|
||||
# re-copies the new file into hooks/ and re-executes the new start()
|
||||
# ============================================================================
|
||||
echo "=== Phase 1: single-repo restart cycle ==="
|
||||
|
||||
BARE1=$WORK_DIR/stack-single.git
|
||||
CLONE1=$WORK_DIR/stack-single
|
||||
git init -b main --bare $BARE1
|
||||
git clone $BARE1 $CLONE1
|
||||
configure_git_identity $CLONE1
|
||||
|
||||
# External-stack layout: <repo>/stack-orchestrator/{stacks,compose}/...
|
||||
mkdir -p $CLONE1/stack-orchestrator/stacks $CLONE1/stack-orchestrator/compose
|
||||
cp -r $DATA_DIR/stacks/test-restart-hook $CLONE1/stack-orchestrator/stacks/
|
||||
cp $DATA_DIR/compose/docker-compose-test-restart-hook.yml $CLONE1/stack-orchestrator/compose/
|
||||
|
||||
git -C $CLONE1 add .
|
||||
git -C $CLONE1 commit -m "test-restart-hook v1"
|
||||
git -C $CLONE1 push -u origin main
|
||||
|
||||
STACK_PATH_SINGLE=$CLONE1/stack-orchestrator/stacks/test-restart-hook
|
||||
SPEC1=$WORK_DIR/spec-single.yml
|
||||
DEP1=$WORK_DIR/dep-single
|
||||
|
||||
$TEST_TARGET_SO --stack $STACK_PATH_SINGLE deploy --deploy-to k8s-kind init --output $SPEC1
|
||||
$TEST_TARGET_SO --stack $STACK_PATH_SINGLE deploy create --spec-file $SPEC1 --deployment-dir $DEP1
|
||||
|
||||
if [ ! -f "$DEP1/hooks/commands.py" ]; then
|
||||
echo "single-repo deploy create test: FAILED (hooks/commands.py missing)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
if ! grep -q '"v1"' "$DEP1/hooks/commands.py"; then
|
||||
echo "single-repo deploy create test: FAILED (hooks/commands.py does not contain v1 marker)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
echo "single-repo deploy create test: passed"
|
||||
|
||||
$TEST_TARGET_SO deployment --dir $DEP1 start --perform-cluster-management
|
||||
wait_for_pods_started $DEP1
|
||||
|
||||
# call_stack_deploy_start runs synchronously inside the start command
|
||||
# (deploy_k8s.py:1026), so the marker is on disk before 'start' returns.
|
||||
if [ ! -f "$DEP1/start-hook-marker" ]; then
|
||||
echo "single-repo start hook v1 test: FAILED (marker file missing)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
marker_v1=$(cat $DEP1/start-hook-marker)
|
||||
if [ "$marker_v1" != "v1" ]; then
|
||||
echo "single-repo start hook v1 test: FAILED (got: $marker_v1)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
echo "single-repo start hook v1 test: passed"
|
||||
|
||||
# Mutate the stack-source working tree v1 -> v2. No commit needed: 'deployment
|
||||
# restart' runs 'git pull' against the bare which is a no-op, and _copy_hooks
|
||||
# reads the working tree directly via get_plugin_code_paths.
|
||||
sed -i 's/"v1"/"v2"/' $STACK_PATH_SINGLE/deploy/commands.py
|
||||
|
||||
$TEST_TARGET_SO deployment --dir $DEP1 restart --stack-path $STACK_PATH_SINGLE
|
||||
|
||||
if ! grep -q '"v2"' "$DEP1/hooks/commands.py"; then
|
||||
echo "single-repo restart hook re-copy test: FAILED (hooks/commands.py still v1)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
echo "single-repo restart hook re-copy test: passed"
|
||||
|
||||
marker_v2=$(cat $DEP1/start-hook-marker)
|
||||
if [ "$marker_v2" != "v2" ]; then
|
||||
echo "single-repo restart hook re-execute test: FAILED (got: $marker_v2)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
echo "single-repo restart hook re-execute test: passed"
|
||||
|
||||
# Stop phase 1 deployment but keep the cluster for phase 2.
|
||||
$TEST_TARGET_SO deployment --dir $DEP1 \
|
||||
stop --delete-volumes --delete-namespace --skip-cluster-management
|
||||
|
||||
# ============================================================================
|
||||
# Phase 2 — multi-repo create + start. Verifies that a stack with N pods, each
|
||||
# from a separate repo, produces hooks/commands_0.py ... commands_{N-1}.py and
|
||||
# that call_stack_deploy_start invokes every module's start().
|
||||
# ============================================================================
|
||||
echo "=== Phase 2: multi-repo create + start ==="
|
||||
|
||||
# Pod repos: stack.yml's pods[].repository = 'cerc-io/test-restart-hook-pod-X'
|
||||
# resolves (via get_plugin_code_paths) to
|
||||
# $CERC_REPO_BASE_DIR/test-restart-hook-pod-X/<pod_path>/stack/...
|
||||
for label in a b; do
|
||||
POD_BARE=$WORK_DIR/pod-$label.git
|
||||
POD_CLONE=$CERC_REPO_BASE_DIR/test-restart-hook-pod-$label
|
||||
git init -b main --bare $POD_BARE
|
||||
git clone $POD_BARE $POD_CLONE
|
||||
configure_git_identity $POD_CLONE
|
||||
mkdir -p $POD_CLONE/stack/deploy
|
||||
# For dict-form pods, get_pod_file_path resolves the compose file at
|
||||
# <pod_repo>/<pod_path>/docker-compose.yml — owned by the pod repo, not
|
||||
# the stack repo. get_plugin_code_paths adds the trailing 'stack/', so
|
||||
# commands.py lives at <pod_repo>/<pod_path>/stack/deploy/commands.py.
|
||||
cat > $POD_CLONE/docker-compose.yml <<EOF
|
||||
services:
|
||||
test-restart-hook-multi-$label:
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "sleep infinity"]
|
||||
restart: always
|
||||
EOF
|
||||
# Each pod hook writes a distinct marker file so neither overwrites the
|
||||
# other when both start() hooks are loaded by call_stack_deploy_start.
|
||||
cat > $POD_CLONE/stack/deploy/commands.py <<EOF
|
||||
from stack_orchestrator.deploy.deployment_context import DeploymentContext
|
||||
|
||||
|
||||
def start(deployment_context: DeploymentContext):
|
||||
marker = deployment_context.deployment_dir / "start-hook-marker-$label"
|
||||
marker.write_text("v1")
|
||||
EOF
|
||||
git -C $POD_CLONE add .
|
||||
git -C $POD_CLONE commit -m "pod $label v1"
|
||||
git -C $POD_CLONE push -u origin main
|
||||
done
|
||||
|
||||
# Stack repo
|
||||
BARE2=$WORK_DIR/stack-multi.git
|
||||
CLONE2=$WORK_DIR/stack-multi
|
||||
git init -b main --bare $BARE2
|
||||
git clone $BARE2 $CLONE2
|
||||
configure_git_identity $CLONE2
|
||||
|
||||
# For multi-repo (dict-form pods), the stack repo only owns stack.yml — pod
|
||||
# compose files and hooks live in the per-pod repos under CERC_REPO_BASE_DIR.
|
||||
mkdir -p $CLONE2/stack-orchestrator/stacks
|
||||
cp -r $DATA_DIR/stacks/test-restart-hook-multi $CLONE2/stack-orchestrator/stacks/
|
||||
|
||||
git -C $CLONE2 add .
|
||||
git -C $CLONE2 commit -m "test-restart-hook-multi v1"
|
||||
git -C $CLONE2 push -u origin main
|
||||
|
||||
STACK_PATH_MULTI=$CLONE2/stack-orchestrator/stacks/test-restart-hook-multi
|
||||
SPEC2=$WORK_DIR/spec-multi.yml
|
||||
DEP2=$WORK_DIR/dep-multi
|
||||
|
||||
$TEST_TARGET_SO --stack $STACK_PATH_MULTI deploy --deploy-to k8s-kind init --output $SPEC2
|
||||
$TEST_TARGET_SO --stack $STACK_PATH_MULTI deploy create --spec-file $SPEC2 --deployment-dir $DEP2
|
||||
|
||||
# get_plugin_code_paths returns list(set(...)) so the index ordering is not
|
||||
# guaranteed; we assert presence of both files rather than mapping each to
|
||||
# a specific pod.
|
||||
if [ ! -f "$DEP2/hooks/commands_0.py" ] || [ ! -f "$DEP2/hooks/commands_1.py" ]; then
|
||||
echo "multi-repo deploy create test: FAILED (hooks/commands_{0,1}.py missing)"
|
||||
ls -la $DEP2/hooks/ || true
|
||||
cleanup_and_exit
|
||||
fi
|
||||
echo "multi-repo deploy create test: passed"
|
||||
|
||||
$TEST_TARGET_SO deployment --dir $DEP2 start --skip-cluster-management
|
||||
wait_for_k8s_pods_ready laconic-test-restart-hook-multi
|
||||
|
||||
for label in a b; do
|
||||
if [ ! -f "$DEP2/start-hook-marker-$label" ]; then
|
||||
echo "multi-repo start hook test: FAILED (start-hook-marker-$label missing)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
val=$(cat $DEP2/start-hook-marker-$label)
|
||||
if [ "$val" != "v1" ]; then
|
||||
echo "multi-repo start hook test: FAILED (start-hook-marker-$label content: $val)"
|
||||
cleanup_and_exit
|
||||
fi
|
||||
done
|
||||
echo "multi-repo start hook test: passed"
|
||||
|
||||
# Final teardown — destroy the cluster for the next CI run.
|
||||
$TEST_TARGET_SO deployment --dir $DEP2 \
|
||||
stop --delete-volumes --delete-namespace --perform-cluster-management
|
||||
|
||||
rm -rf $WORK_DIR
|
||||
|
||||
echo "Test passed"
|
||||
Loading…
Reference in New Issue