type: create

description: Route all validator traffic (gossip, repair, TVU, TPU) through 137.239.194.65 on laconic-was-sw01 in Ashburn. Supersedes old TVU-only shred relay. See docs/ashburn-validator-relay.md for full design.

title: Ashburn Full Validator Traffic Relay

type: epic


description: Create Loopback101 (137.239.194.65/32), VALIDATOR-RELAY ACL + traffic-policy on Et1/1, replacing old SHRED-RELAY. Uses 5-min auto-revert config session. Playbook: playbooks/ashburn-relay-was-sw01.yml

title: was-sw01: Inbound validator relay config

type: task


description: Add 137.239.194.65/32 to lo, DNAT rules for ports 8001,9000-9025 to kind node 172.20.0.2. Playbook: playbooks/ashburn-relay-biscayne.yml -t inbound

title: biscayne: Inbound DNAT rules


description: Ping 137.239.194.65 from external host, check DNAT counters on biscayne, verify traffic-policy counters on was-sw01.

title: Verify inbound relay


description: Pre-flight to discover GRE tunnel interface, then apply VALIDATOR-OUTBOUND traffic-policy redirecting src 137.239.194.65 to was-sw01 via backbone. Playbook: playbooks/ashburn-relay-mia-sw01.yml

title: mia-sw01: Outbound validator redirect


description: fwmark 100 on validator source ports, SNAT to 137.239.194.65, policy route via doublezero0 table ashburn. Playbook: playbooks/ashburn-relay-biscayne.yml -t outbound

title: biscayne: Outbound SNAT + policy routing


description: Verify traffic-policy counters on both switches, iptables counters on biscayne, validator gossip ContactInfo shows 137.239.194.65, repair peer count increases, slot catchup rate improves. Write memory on both switches.

title: End-to-end verification


description: After stable: remove old SHRED-RELAY policy and ACL from was-sw01, remove old 64.92.84.81:20000 DNAT from biscayne.

title: Cleanup old SHRED-RELAY


type: update

description: Run ansible playbook (pane A) to apply config session with 5-min auto-revert. Review output. In pane B, SSH to install@137.239.200.198 and manually verify (show session-config diffs, show traffic-policy counters). Type 'configure session validator-relay commit' and 'write memory' when satisfied. Playbook: playbooks/ashburn-relay-was-sw01.yml (do NOT use -e commit=true; commit is manual via SSH).


description: Run ansible playbook pre-flight (pane A) to discover GRE tunnel interface. Then run with -e apply=true -e tunnel_interface=TunnelX for 5-min auto-revert. In pane B, SSH to install@209.42.167.133 and manually verify. Type 'configure session validator-outbound commit' and 'write memory' when satisfied. Playbook: playbooks/ashburn-relay-mia-sw01.yml (do NOT use -e commit=true; commit is manual via SSH).


description:
/srv/solana is a directory on the ZFS dataset biscayne/DATA/srv (mounted at /srv
with overlay=on). The fstab zvol mount at /srv/solana was shadowed by ZFS.

Fixed 2026-03-08: removed /srv/solana fstab entries, canonical data path is now
/srv/kind/solana. All playbooks updated. fstab clean. Mounts verified.

title: zvol mount: /srv/solana resolves to ZFS dataset, not zvol

type: bug


description:
laconic-so creates configmap resources for telegraf but does not generate
volumeMounts in the pod spec. The telegraf container crashes because
/etc/telegraf and /scripts are empty. Manual configmap creation works but
the volume mounts are still missing. Root cause is in laconic-so's stack
migration — configmap volume mount generation is incomplete.

title: telegraf volume mounts missing from pod spec


description:
Validator exits shortly after starting. Log shows UDP port reachability checks
and TCP port checks failing. Needs full log analysis from kind node path
(/mnt/validator-log/validator.log). May be related to networking/firewall
configuration or the shred relay setup.

title: agave-validator crash after ~57 seconds


description:
Once laconic-so deployment prepare lands, update biscayne-redeploy.yml to use
prepare instead of start+scale-to-0 workaround. The deploy tag section should
call deployment prepare, and scale-up should call deployment start
--skip-cluster-management.

title: update biscayne-redeploy to use deployment prepare


description:
Automate the leapfrog recovery strategy documented in CLAUDE.md. When the
validator is stuck in a repair-dependent gap, download a fresh snapshot past
the incomplete zone while preserving the existing ledger (which has turbine
shreds at the tip). Needs: shred completeness check, snapshot slot targeting,
selective wipe (accounts+snapshots only, keep ledger).

title: snapshot leapfrog recovery playbook

type: feature


description:
biscayne-prepare-agave.yml unconditionally imports ashburn-relay-biscayne.yml
at the end. This couples filesystem preparation to relay setup. The relay
playbook fails if the kind node isn't running (ping to 172.20.0.2 fails).
Should be a separate playbook invocation, not an import.

title: biscayne-prepare-agave imports ashburn-relay-biscayne unconditionally