diff --git a/.pebbles/events.jsonl b/.pebbles/events.jsonl index 7d1397a3..b343c355 100644 --- a/.pebbles/events.jsonl +++ b/.pebbles/events.jsonl @@ -43,3 +43,10 @@ {"type":"create","timestamp":"2026-03-08T06:58:54.238136989Z","issue_id":"bar-b41","payload":{"description":"Automate the leapfrog recovery strategy documented in CLAUDE.md. When the\nvalidator is stuck in a repair-dependent gap, download a fresh snapshot past\nthe incomplete zone while preserving the existing ledger (which has turbine\nshreds at the tip). Needs: shred completeness check, snapshot slot targeting,\nselective wipe (accounts+snapshots only, keep ledger).","priority":"2","title":"snapshot leapfrog recovery playbook","type":"feature"}} {"type":"create","timestamp":"2026-03-08T06:58:54.756609299Z","issue_id":"bar-0b4","payload":{"description":"biscayne-prepare-agave.yml unconditionally imports ashburn-relay-biscayne.yml\nat the end. This couples filesystem preparation to relay setup. The relay\nplaybook fails if the kind node isn't running (ping to 172.20.0.2 fails).\nShould be a separate playbook invocation, not an import.","priority":"3","title":"biscayne-prepare-agave imports ashburn-relay-biscayne unconditionally","type":"bug"}} {"type":"close","timestamp":"2026-03-08T06:59:00.140156099Z","issue_id":"bar-02e","payload":{}} +{"type":"create","timestamp":"2026-03-10T08:05:07.190617713Z","issue_id":"bar-2c9","payload":{"description":"laconic-so build-containers --include filter does exact string match via\ninclude_exclude_check(). Container names use slash (laconicnetwork/agave),\nnot dash. Using --include laconicnetwork-agave silently skips the build\nand reports success.\n\nFixed in biscayne-sync-tools.yml (commit ceea8f0) but the underlying\nlaconic-so behavior of silently skipping with no warning is a bug.","priority":"2","title":"build-containers --include uses slash not dash in container names","type":"bug"}} +{"type":"create","timestamp":"2026-03-10T08:05:12.506655809Z","issue_id":"bar-6cb","payload":{"description":"When laconic-so deployment restart deletes the namespace, PVCs are\ncascade-deleted but PVs (cluster-scoped) survive in Released state with\nstale claimRefs pointing to the old PVC UIDs. New PVCs created by the\nrestarted deployment can't bind because the PVs still reference the\ndeleted PVCs.\n\nWorkaround: patch Released PVs to clear claimRef after restart.\nAdded to biscayne-restart.yml. Root cause is in laconic-so — it should\nclear stale claimRefs as part of the restart flow.\n\nRelated: so-933 (namespace termination race).","priority":"1","title":"PV claimRefs go stale after deployment restart","type":"bug"}} +{"type":"create","timestamp":"2026-03-10T08:05:15.941416301Z","issue_id":"bar-fec","payload":{"description":"monitoring-grafana-data volume is defined in spec.yml but laconic-so's\nget_pvcs() does not generate a PVC for it. The PV is created but no\nmatching PVC exists, so the grafana container can't mount its data volume.\n\nWorkaround: manually kubectl apply the PVC after each deployment restart.\nRoot cause is in stack-orchestrator deploy_k8s.py get_pvcs().","priority":"2","title":"grafana PVC not generated by get_pvcs()","type":"bug"}} +{"type":"create","timestamp":"2026-03-10T08:05:22.853965263Z","issue_id":"bar-822","payload":{"description":"Rebuilding a container image on the Docker host does NOT update the image\ninside the kind node. With imagePullPolicy: IfNotPresent (the default for\n:local tags), kind uses its cached copy. Must run:\n\n kind load docker-image laconicnetwork/agave:local \\\n --name laconic-70ce4c4b47e23b85\n\nafter every rebuild. This step is not in any playbook or laconic-so flow.\nShould be added to biscayne-sync-tools.yml build-container tag or to\nlaconic-so build-containers itself.","priority":"2","title":"kind load docker-image required after container rebuild","type":"bug"}} +{"type":"create","timestamp":"2026-03-10T08:05:28.585915055Z","issue_id":"bar-571","payload":{"description":"Full snapshot slots differ per validator depending on when each started.\nThe entrypoint's incremental download loop assumes it can find an\nincremental keyed to any full snapshot's base slot, but no other validator\nmay have produced a full at that exact slot.\n\nThis causes the incremental download to retry forever when the local\nfull snapshot has a base slot that no network peer has incrementals for.\n\nDocumented for awareness. The entrypoint's infinite retry is intentional\n(user decision) — eventually a matching incremental will appear or the\nentrypoint falls through to download a fresh full+incremental pair.","priority":"3","title":"snapshot base slots are not consensus-aligned across validators","type":"bug"}} +{"type":"create","timestamp":"2026-03-10T08:05:32.262889286Z","issue_id":"bar-2d9","payload":{"description":"When spec.yml has explicit values for env vars that also have defaults in\nthe compose file, the spec.yml values win. Changing compose file defaults\nhas no effect unless the spec.yml override is also removed.\n\nThis is by design (spec.yml is deployment-specific config) but the\ninteraction is non-obvious. Bit us when changing snapshot settings in\ncompose but spec.yml still had the old values.\n\nNot a code bug — more a documentation/workflow issue. Operators must\ncheck both compose defaults and spec.yml overrides.","priority":"3","title":"spec.yml overrides compose defaults silently","type":"bug"}} +{"type":"create","timestamp":"2026-03-10T08:05:36.212405156Z","issue_id":"bar-31a","payload":{"description":"laconic-so deployment restart sleeps only 5s between down and up. If the\nnamespace is still terminating when 'up' runs, k8s returns 403 Forbidden\ncreating configmaps in the new namespace.\n\nCross-ref: so-933 in the stack-orchestrator pebbles project.\n\nWorkaround: retry the restart or wait manually. The restart playbook\n(biscayne-restart.yml) handles this by scaling to 0 first, waiting for\npod termination, then calling laconic-so restart.","priority":"1","title":"deployment restart namespace termination race","type":"bug"}}