ip_echo_preflight.py: reimplements Solana ip_echo client protocol in
Python. Verifies UDP port reachability before snapshot download, called
from entrypoint.py. Prevents wasting hours on a snapshot only to
crash-loop on port reachability.
docs/postmortem-ashburn-relay-outbound.md: root cause analysis of the
firewalld nftables FORWARD chain blocking outbound relay traffic.
docs/ashburn-relay-checklist.md: 7-layer verification checklist for
relay path debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entrypoint changes:
- Always require full + incremental before starting (retry until found)
- Check incremental freshness against convergence threshold (500 slots)
- Gap monitor thread: if validator falls >5000 slots behind for 3
consecutive checks, graceful stop + restart with fresh incremental
- cmd_serve is now a loop: download → run → monitor → leapfrog → repeat
- --no-snapshot-fetch moved to common args (both RPC and validator modes)
- --maximum-full-snapshots-to-retain default 1 (validator deletes
downloaded full after generating its own)
- SNAPSHOT_MAX_AGE_SLOTS default 100000 (one full snapshot generation)
snapshot_download.py refactoring:
- Extract _discover_and_benchmark() and _rolling_incremental_download()
as shared helpers
- Restore download_incremental_for_slot() using shared helpers (downloads
only an incremental for an existing full snapshot)
- download_best_snapshot() uses shared helpers, downloads full then
incremental as separate operations
The leapfrog cycle: validator generates full snapshots at standard 100k
block height intervals (same slots as the rest of the network). When the
gap monitor triggers, the entrypoint loops back to maybe_download_snapshot
which finds the validator's local full, downloads a fresh network
incremental (generated every ~40s, converges within the ~11hr full
generation window), and restarts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Overwrite lines in place instead of clear+redraw (no flicker)
- Pad lines to terminal width to clear stale characters
- Blank leftover rows when output shrinks between frames
- Hide cursor during watch mode
- Remove section comment bars
- Replace unicode checkmarks with +/x
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of hardcoding the laconic cluster ID, namespace, deployment
name, and pod label, read cluster-id from deployment.yml on biscayne
and derive everything from it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The recovery playbook now exits after scaling to 1. The container
entrypoint handles snapshot download (60+ min) and validator startup
autonomously. Removed all polling/verification steps that would
time out waiting.
Added scripts/check-status.py for monitoring download progress,
validator slot, gap to mainnet, catch-up rate, and ramdisk usage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without a bound, the loop runs forever if sources never serve an
incremental close enough to head (e.g. full snapshot base slot is
too old). After 30 minutes, proceed with the best incremental
available or none.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After the full snapshot downloads, continuously re-probe all fast sources
for newer incrementals until the best available is within convergence_slots
(default 500) of head. Each iteration finds the highest-slot incremental
matching our full snapshot's base slot, downloads it (replacing any previous),
and checks the gap to mainnet head.
- Extract probe_incremental() from inline re-probe code
- Add convergence_slots param to download_best_snapshot() (default 500)
- Add --convergence-slots CLI arg
- Pass SNAPSHOT_CONVERGENCE_SLOTS env var from entrypoint.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The container entrypoint (entrypoint.py) handles snapshot download
internally via aria2c. Ansible no longer needs to scale-to-0, download,
scale-to-1 — it just deploys and lets the container manage startup.
- biscayne-redeploy.yml: remove snapshot download section, simplify to
teardown → wipe → deploy → verify
- biscayne-sync-tools.yml: new playbook to sync laconic-so and
agave-stack repos on biscayne, with separate branch controls
- snapshot_download.py: re-probe for fresh incremental after full
snapshot download completes (old incremental is stale by then)
- Switch laconic_so_branch to fix/kind-mount-propagation (has
hostNetwork translation code)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- playbooks/biscayne-iptables.yml: manages PREROUTING DNAT and DOCKER-USER
rules for both host IP (186.233.184.235) and relay loopback (137.239.194.65).
Idempotent, persists via netfilter-persistent.
- scripts/snapshot-download.py: replaced standalone copy with symlink to
agave-stack source of truth, eliminating duplication.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The /dev/ram0 + XFS + format-ramdisk.service approach was unnecessary
complexity from a migration confusion — there was no actual tmpfs bug
with io_uring. tmpfs is simpler (no format-on-boot), resizable on the
fly, and what every other Solana operator uses.
Changes:
- prepare-agave: remove format-ramdisk.service and ramdisk-accounts.service,
use tmpfs fstab entry with size=1024G (was 600G /dev/ram0, too small)
- recover: remove ramdisk_device var (no longer needed)
- redeploy: wipe accounts by rm -rf instead of umount+mkfs
- snapshot-download.py: extract download_best_snapshot() public API for
use by the new container entrypoint.py (in agave-stack)
- CLAUDE.md: update ramdisk docs, fix /srv/solana → /srv/kind/solana paths
- health-check: fix ramdisk path references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: Docker FORWARD chain policy DROP blocked all DNAT'd relay
traffic (UDP/TCP 8001, UDP 9000-9025) to the kind node. The DOCKER
chain only ACCEPTs specific TCP ports (6443, 443, 80). Added ACCEPT
rules in DOCKER-USER chain which runs before all Docker chains.
Changes:
- ashburn-relay-biscayne.yml: add DOCKER-USER ACCEPT rules (inbound
tag) and rollback cleanup
- ashburn-relay-setup.sh.j2: persist DOCKER-USER rules across reboot
- relay-inbound-udp-test.yml: controlled e2e test — listener in kind
netns, sender from kelce, assert arrival
- relay-link-test.yml: link-by-link tcpdump captures at each hop
- relay-test-udp-listen.py, relay-test-udp-send.py: test helpers
- relay-test-ip-echo.py: full ip_echo protocol test
- inventory/kelce.yml, inventory/panic.yml: test host inventories
- test-ashburn-relay.sh: add ip_echo UDP reachability test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three Python scripts send real packets from the kind node through the
full relay path (biscayne → tunnel → mia-sw01 → was-sw01 → internet)
and verify responses come back via the inbound path. No indirect
counter-checking — a response proves both directions work.
- relay-test-udp.py: DNS query with sport 8001
- relay-test-tcp-sport.py: HTTP request with sport 8001
- relay-test-tcp-dport.py: TCP connect to entrypoint dport 8001 (ip_echo)
- test-ashburn-relay.sh: orchestrates from ansible controller via nsenter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Playbook fixes from testing:
- ashburn-relay-biscayne: insert DNAT rules at position 1 before
Docker's ADDRTYPE LOCAL rule (was being swallowed at position 3+)
- ashburn-relay-mia-sw01: add inbound route for 137.239.194.65 via
egress-vrf vrf1 (nexthop only, no interface — EOS silently drops
cross-VRF routes that specify a tunnel interface)
- ashburn-relay-was-sw01: replace PBR with static route, remove
Loopback101
Bug doc (bug-ashburn-tunnel-port-filtering.md): root cause is the
DoubleZero agent on mia-sw01 overwrites SEC-USER-500-IN ACL, dropping
outbound gossip with src 137.239.194.65. The DZ agent controls
Tunnel500's lifecycle. Fix requires a separate GRE tunnel using
mia-sw01's free LAN IP (209.42.167.137) to bypass DZ infrastructure.
Also adds all repo docs, scripts, inventory, and remaining playbooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A. F. Dudley