kylechase · August 8, 2025 13:18 · Apr 29, 2025 · Apr 28, 2025 · Apr 28, 2025 · Apr 28, 2025
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -157,21 +157,34 @@ ipv6 forwarding
 interface en05
  ip router openfabric 1
  ipv6 router openfabric 1
+ openfabric hello-interval 1
+ openfabric hello-multiplier 3
+ openfabric csnp-interval 5
+ openfabric psnp-interval 2
 exit
 
 interface en06
  ip router openfabric 1
  ipv6 router openfabric 1
+ openfabric hello-interval 1
+ openfabric hello-multiplier 3
+ openfabric csnp-interval 5
+ openfabric psnp-interval 2
 exit
 
 interface lo
  ip router openfabric 1
  ipv6 router openfabric 1
+ openfabric hello-interval 1
+ openfabric hello-multiplier 3
+ openfabric csnp-interval 5
+ openfabric psnp-interval 2
  openfabric passive
 exit
 
 router openfabric 1
 net 49.0000.0000.000x.00
+lsp-gen-interval 5
 exit
 !
 exit

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -66,9 +66,9 @@ iface lo inet loopback
 
 ## FRR Setup
 
-### Install & enable FRR (not needed on proxmox 8.4)
-Install Free Range Routing (FRR) `apt install frr`
-Enable frr `systemctl enable frr`
+### Install & enable FRR (not needed on proxmox 8.4+ )
+1. Install Free Range Routing (FRR) `apt install frr`
+2. Enable frr `systemctl enable frr`
 
 ### Enable the fabricd daemon
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -17,17 +17,20 @@ Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 
-## NOTES on Dual Stack
-~I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....~
-
-With all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 for ceph as documented in that gists in the series. This is to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
-
+> ** NOTES on Dual Stack*
+> 
+> Having spent 3 days hammering my network and playing with various different routed toplogies i am of the current opinion
+> - i still prefer IPv6 for my mesh but if you setup for IPv4 it should now be fine but my gists will continue to assume you used IPv6 for ceph
+> - i have no opinion on squid and dual stack yet - should be doable... we will seee
+> - if you use ONLY IPv6 for the love-of-god(tm) make sure that `ms_bind_ipv4 = false` is set in ceph.conf or really bad things will eventuall happen
 
 
 ## Defining thunderbolt network
 
+> This was revised on 2025.04.27 to move loopback IP addressing back from frr.conf to here (along with some reliability changes recommended by chatgpt) having loopback IPs was a stupid idea as they should be up irrespective of the state of the mesh to allow ceph processes to start binding to it.
+
 Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
-There should no lober be any IP addresses in this file for lo and lo:6
+
 
 ```bash
 # Thunderbolt interfaces for pve1 (Node 81)

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -1,6 +1,6 @@
 #  Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing
 
-## Version 2.2 (2025.04.25) 
+## Version 2.5 (2025.04.27) 
 
 [this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
 
@@ -10,8 +10,8 @@ This changes the previous file design thanks to @NRGNet and @tisayama to make th
 
 Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 
-- move IP address configuration from `interfaces.d/thundebolt` to frr configuration
-- new approach to remove dependecy on post-up, new script in if-up.d that logs to systemlog
+- ~move IP address configuration from `interfaces.d/thundebolt` to frr configuration~ i reverted this on 2025.04.27 and improved settings  in interfaces.d/thunderbolt based on recommendations from chatGPT to solve issues i hit it my routed network setup (coming soon)
+- new approach to remove dependecy on post-up with new scripts in if-up.d that logs to systemlog
 - reminder to copy frr.conf > frr.conf.local to prevent breakage if you enable Proxmox SDN
 - dependent on the changes to the udev link scripts [here](/67fdc9a517faefa68f730f82d7fa3570#set-interfaces-to-up-on-reboots-and-cable-insertions)
 
@@ -29,16 +29,30 @@ With all the scripts and changes folks have contributed IPv4 should now be stabl
 Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
 There should no lober be any IP addresses in this file for lo and lo:6
 
-```     
-allow-hotplug en05
-iface en05 inet manual
-        mtu 65520
+```bash
+# Thunderbolt interfaces for pve1 (Node 81)
 
-allow-hotplug en06
-iface en06 inet manual
-        mtu 65520
+auto en05
+iface en05 inet6 static
+    pre-up ip link set $IFACE up
+    mtu 65520
+
+auto en06
+iface en06 inet6 static
+    pre-up ip link set $IFACE up
+    mtu 65520
+
+# Loopback for Ceph MON
+auto lo
+iface lo inet loopback
+    up ip -6 addr add fc00::81/128 dev lo
+    up ip addr add 10.0.0.81/32 dev lo
 ```
-Save file, repeat on each node.
+> **Notes:**
+> - doing loopback IP is more reliable in interfaces file than in frr.conf the ip address will always be available for the mon, mgr, and mds processes of ceph to bind to irrespective of frr service status
+> - mtus are super importantor BGP and openfabric seem to have node to node negotiation issues
+> - the `pre-up` and `up` directives were recommended by chatGPT to ensure the interfaces are up before applying the IP address and MTU - should make things more reliable
+
 
 ## Enable IPv4 and IPv6 forwarding
 1. use `nano /etc/sysctl.conf` to open the file
@@ -49,7 +63,7 @@ Save file, repeat on each node.
 
 ## FRR Setup
 
-### Install & enable FRR
+### Install & enable FRR (not needed on proxmox 8.4)
 Install Free Range Routing (FRR) `apt install frr`
 Enable frr `systemctl enable frr`
 
@@ -62,7 +76,11 @@ Enable frr `systemctl enable frr`
 ### Mitigate FRR Timing Issues (I need someone with an MS-101 to confirm if helps solve their IPv4 issues)
 
 #### create script that is automatically processed when en05/en06 are brougt up to restart frr
-this should make IPv4 more stable for all users (i ended up seeing IPv4 issues too, just less commonly than MS-101 users)
+
+> **notes**
+> - this should make IPv4 more stable for all users (i ended up seeing IPv4 issues too, just less commonly than MS-101 users)
+> - i found the chnages i introduced in 2.5 version of this gist make this less needed, occasionally ifreload / ifupdown2 may cause enough changes that frr gets restarted too often and the service will need to be unblocked with systemctl.
+
 
 1. create a new file with `nano /etc/network/if-up.d/en0x`
 2. add to file the following
@@ -132,25 +150,23 @@ make it executable with `chmod +x /etc/network/if-up.d/lo`
 ```
 ip forwarding
 ipv6 forwarding
-!
+
 interface en05
-ip router openfabric 1
-ipv6 router openfabric 1
+ ip router openfabric 1
+ ipv6 router openfabric 1
 exit
-!
+
 interface en06
-ip router openfabric 1
-ipv6 router openfabric 1
+ ip router openfabric 1
+ ipv6 router openfabric 1
 exit
-!
+
 interface lo
-ip address 10.0.0.8x/32
-ipv6 address fc00::8x/128
-ip router openfabric 1
-ipv6 router openfabric 1
-openfabric passive
+ ip router openfabric 1
+ ipv6 router openfabric 1
+ openfabric passive
 exit
-!
+
 router openfabric 1
 net 49.0000.0000.000x.00
 exit

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -20,7 +20,7 @@ This will result in an IPv4 and IPv6 routable mesh network that can survive any
 ## NOTES on Dual Stack
 ~I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....~
 
-I think now with all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 for ceph as documented in that gists in the series, this is to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
+With all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 for ceph as documented in that gists in the series. This is to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
 
 
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -163,7 +163,7 @@ exit
 7. show the configure applied correctly with `show running-config` - note the order of the items will be different to how you entered them and thats ok.  (If you made a mistake i found the easiest way was to edt `/etc/frr/frr.conf` - but be careful if you do that.)
 8. use the command `exit` to leave setup
 9. repeat steps 1 to 9 on the other 3 nodes
-10. once you have configured all 3 nodes issue the command `vtysh -c "show openfabric topology"` if you did everything right you will see:
+10. once you have configured all 3 nodes issue the command `vtysh -c "show openfabric topology"` if you did everything right you will see (note it may take 45 seconds for for all routes to show if you just restarted frr for any reason):
 ```
 Area 1:
 IS-IS paths to level-2 routers that speak IP

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -193,4 +193,6 @@ Now you should be in a place to ping each node from evey node across the thunder
 ### IMPORTAT - you need to do this to stop SDN breaking you in future 
 if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.
 
+**note: if you already have SDN configured do not do the step above as you will mess both your SDN and this openfabric topology (see note at start of frr instructions)
+
 based on this response https://forum.proxmox.com/threads/relationship-of-frr-conf-and-frr-conf-local.165465/ if you have SDN all local (non SDN) configuration changes should be made in .local, this should be read next time SDN apply is used. do not copy frr.conf > frr.conf.local after doing anything with SDN or when you tear down SDN the settings will not be removed from frr.conf
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -1,6 +1,6 @@
 #  Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing
-# !!WORK IN PROGRESS!!
-## Version 2.1 (2025.04.24) 
+
+## Version 2.2 (2025.04.25) 
 
 [this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -120,6 +120,9 @@ fi
 make it executable with `chmod +x /etc/network/if-up.d/lo`
 
 ### Configure OpenFabric (perforn on all nodes)
+
+**note: if (and only if) you have already configured SDN you should make these settings in /etc/frr/frr.conf.local and reapply your SDN configuration to have SDN propogate these into frr.conf (you can also make the edits to both files if you prefer) if you make these edits to only frr.conf with SDN active and then reapply the settings it will loose these settings.
+
 1. enter the FRR shell with `vtysh`
 2. optionally show the current config with `show running-config`
 3. enter the configure mode with `configure`
@@ -155,7 +158,7 @@ exit
 exit
 
 ```
-5. you may need to pres return after the last `exit` to get to a new line - if so do this
+5. you may need to press return after the last `exit` to get to a new line - if so do this
 6. save the configu with `write memory`
 7. show the configure applied correctly with `show running-config` - note the order of the items will be different to how you entered them and thats ok.  (If you made a mistake i found the easiest way was to edt `/etc/frr/frr.conf` - but be careful if you do that.)
 8. use the command `exit` to leave setup
@@ -188,6 +191,6 @@ Vertex               Type         Metric Next-Hop             Interface Parent
 Now you should be in a place to ping each node from evey node across the thunderbolt mesh using IPv4 or IPv6 as you see fit.
 
 ### IMPORTAT - you need to do this to stop SDN breaking you in future 
-if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.  
+if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.
 
 based on this response https://forum.proxmox.com/threads/relationship-of-frr-conf-and-frr-conf-local.165465/ if you have SDN all local (non SDN) configuration changes should be made in .local, this should be read next time SDN apply is used. do not copy frr.conf > frr.conf.local after doing anything with SDN or when you tear down SDN the settings will not be removed from frr.conf
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -188,8 +188,6 @@ Vertex               Type         Metric Next-Hop             Interface Parent
 Now you should be in a place to ping each node from evey node across the thunderbolt mesh using IPv4 or IPv6 as you see fit.
 
 ### IMPORTAT - you need to do this to stop SDN breaking you in future 
-if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.  So do this copy whenever you have edited the conf using vtysh or by hand.
+if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.  
 
-
-
-I haven't yet tested to see if we can have just the settings in frr.conf.local when SDN is not configured....
+based on this response https://forum.proxmox.com/threads/relationship-of-frr-conf-and-frr-conf-local.165465/ if you have SDN all local (non SDN) configuration changes should be made in .local, this should be read next time SDN apply is used. do not copy frr.conf > frr.conf.local after doing anything with SDN or when you tear down SDN the settings will not be removed from frr.conf
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -20,7 +20,7 @@ This will result in an IPv4 and IPv6 routable mesh network that can survive any
 ## NOTES on Dual Stack
 ~I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....~
 
-I think now with all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 ceph as docuemnted in that gists in the series to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
+I think now with all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 for ceph as documented in that gists in the series, this is to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
 
 
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -6,7 +6,7 @@
 
 This assumes you are running Proxmox 8.4 and that the line `source /etc/network/interfaces.d/*` is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).
 
-This changes the previous file design thanks to @NRGNet and @tisayama to make the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network ~~(i still recommend the use of the IPv6 FC00 network you will see in these docs)
+This changes the previous file design thanks to @NRGNet and @tisayama to make the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network ~(i still recommend the use of the IPv6 FC00 network you will see in these docs)~
 
 Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 
@@ -18,7 +18,7 @@ Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 
 ## NOTES on Dual Stack
-~~I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
+~I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....~
 
 I think now with all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 ceph as docuemnted in that gists in the series to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -6,7 +6,7 @@
 
 This assumes you are running Proxmox 8.4 and that the line `source /etc/network/interfaces.d/*` is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).
 
-This changes the previous file design thanks to @NRGNet and @tisayama to make the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)
+This changes the previous file design thanks to @NRGNet and @tisayama to make the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network ~~(i still recommend the use of the IPv6 FC00 network you will see in these docs)
 
 Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -1,6 +1,6 @@
 #  Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing
 # !!WORK IN PROGRESS!!
-## Version 2 (2025.04.20) 
+## Version 2.1 (2025.04.24) 
 
 [this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
 
@@ -18,7 +18,10 @@ Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 
 ## NOTES on Dual Stack
-I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
+~~I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
+
+I think now with all the scripts and changes folks have contributed IPv4 should now be stable.  I am recommending new folks use IPv4 ceph as docuemnted in that gists in the series to avoid ongoing issues with SDN and IPv6.  I have yet to decide if i will migrate my ceph back to IPv4 so i can play with SDN or just wait for the SDN issues to be solved. 
+
 
 
 ## Defining thunderbolt network

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -66,14 +66,26 @@ this should make IPv4 more stable for all users (i ended up seeing IPv4 issues t
 
 ```
 #!/bin/bash
-# note the looger entries log to the system journal in the pve UI etc
+# note the logger entries log to the system journal in the pve UI etc
 
 INTERFACE=$IFACE
 
 if [ "$INTERFACE" = "en05" ] || [ "$INTERFACE" = "en06" ]; then
+    logger "Checking if frr.service is running for $INTERFACE"
+    
+    if ! systemctl is-active --quiet frr.service; then
+        logger -t SCYTO "   [SCYTO SCRIPT ] frr.service not running. Starting service."
+        if systemctl start frr.service; then
+            logger -t SCYTO "   [SCYTO SCRIPT ] Successfully started frr.service"
+        else
+            logger -t SCYTO "   [SCYTO SCRIPT ] Failed to start frr.service"
+        fi
+        exit 0
+    fi
+
     logger "Attempting to reload frr.service for $INTERFACE"
     if systemctl reload frr.service; then
-        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully reload frr.service for $INTERFACE"
+        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully reloaded frr.service for $INTERFACE"
     else
         logger -t SCYTO "   [SCYTO SCRIPT ] Failed to reload frr.service for $INTERFACE"
     fi

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -46,8 +46,9 @@ Save file, repeat on each node.
 
 ## FRR Setup
 
-### Install FRR
+### Install & enable FRR
 Install Free Range Routing (FRR) `apt install frr`
+Enable frr `systemctl enable frr`
 
 ### Enable the fabricd daemon
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -79,7 +79,7 @@ if [ "$INTERFACE" = "en05" ] || [ "$INTERFACE" = "en06" ]; then
 fi
 ```
 
-3. make it executable with `chmod +x /etc/network/if-up.d/frr`
+3. make it executable with `chmod +x /etc/network/if-up.d/en0x`
 
 ### mitgigate issues cause by things that reset the loopback
 #### create script that is automatically processed when lo is reprocessed by ifreload, ifupdown2, pve set, etc
@@ -101,6 +101,7 @@ if [ "$INTERFACE" = "lo" ]  ; then
     fi
 fi
 ```
+make it executable with `chmod +x /etc/network/if-up.d/lo`
 
 ### Configure OpenFabric (perforn on all nodes)
 1. enter the FRR shell with `vtysh`

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -81,6 +81,27 @@ fi
 
 3. make it executable with `chmod +x /etc/network/if-up.d/frr`
 
+### mitgigate issues cause by things that reset the loopback
+#### create script that is automatically processed when lo is reprocessed by ifreload, ifupdown2, pve set, etc
+
+1. create a new file with `nano /etc/network/if-up.d/lo`
+2. add to file the following
+
+```
+#!/bin/bash
+
+INTERFACE=$IFACE
+
+if [ "$INTERFACE" = "lo" ]  ; then
+    logger "Attempting to restart frr.service for $INTERFACE"
+    if systemctl restart frr.service; then
+        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully restart frr.service for $INTERFACE"
+    else
+        logger -t SCYTO "   [SCYTO SCRIPT ] Failed to restart frr.service for $INTERFACE"
+    fi
+fi
+```
+
 ### Configure OpenFabric (perforn on all nodes)
 1. enter the FRR shell with `vtysh`
 2. optionally show the current config with `show running-config`
@@ -152,24 +173,6 @@ Now you should be in a place to ping each node from evey node across the thunder
 ### IMPORTAT - you need to do this to stop SDN breaking you in future 
 if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.  So do this copy whenever you have edited the conf using vtysh or by hand.
 
-#### create script that is automatically processed when lo is reprocessed by ifreload, ifupdown2, pve set, etc
-
-1. create a new file with `nano /etc/network/if-up.d/lo`
-2. add to file the following
-
-```
-#!/bin/bash
-
-INTERFACE=$IFACE
 
-if [ "$INTERFACE" = "lo" ]  ; then
-    logger "Attempting to restart frr.service for $INTERFACE"
-    if systemctl restart frr.service; then
-        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully restart frr.service for $INTERFACE"
-    else
-        logger -t SCYTO "   [SCYTO SCRIPT ] Failed to restart frr.service for $INTERFACE"
-    fi
-fi
-```
 
 I haven't yet tested to see if we can have just the settings in frr.conf.local when SDN is not configured....
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -60,7 +60,7 @@ Install Free Range Routing (FRR) `apt install frr`
 #### create script that is automatically processed when en05/en06 are brougt up to restart frr
 this should make IPv4 more stable for all users (i ended up seeing IPv4 issues too, just less commonly than MS-101 users)
 
-1. create a new file with `nano /etc/network/if-up.d/frr`
+1. create a new file with `nano /etc/network/if-up.d/en0x`
 2. add to file the following
 
 ```
@@ -152,4 +152,24 @@ Now you should be in a place to ping each node from evey node across the thunder
 ### IMPORTAT - you need to do this to stop SDN breaking you in future 
 if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.  So do this copy whenever you have edited the conf using vtysh or by hand.
 
+#### create script that is automatically processed when lo is reprocessed by ifreload, ifupdown2, pve set, etc
+
+1. create a new file with `nano /etc/network/if-up.d/lo`
+2. add to file the following
+
+```
+#!/bin/bash
+
+INTERFACE=$IFACE
+
+if [ "$INTERFACE" = "lo" ]  ; then
+    logger "Attempting to restart frr.service for $INTERFACE"
+    if systemctl restart frr.service; then
+        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully restart frr.service for $INTERFACE"
+    else
+        logger -t SCYTO "   [SCYTO SCRIPT ] Failed to restart frr.service for $INTERFACE"
+    fi
+fi
+```
+
 I haven't yet tested to see if we can have just the settings in frr.conf.local when SDN is not configured....
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -70,11 +70,11 @@ this should make IPv4 more stable for all users (i ended up seeing IPv4 issues t
 INTERFACE=$IFACE
 
 if [ "$INTERFACE" = "en05" ] || [ "$INTERFACE" = "en06" ]; then
-    logger "Attempting to restart frr.service for $INTERFACE"
-    if systemctl restart frr.service; then
-        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully restarted frr.service for $INTERFACE"
+    logger "Attempting to reload frr.service for $INTERFACE"
+    if systemctl reload frr.service; then
+        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully reload frr.service for $INTERFACE"
     else
-        logger -t SCYTO "   [SCYTO SCRIPT ] Failed to restart frr.service for $INTERFACE"
+        logger -t SCYTO "   [SCYTO SCRIPT ] Failed to reload frr.service for $INTERFACE"
     fi
 fi
 ```

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -149,6 +149,7 @@ Vertex               Type         Metric Next-Hop             Interface Parent
 ```
 Now you should be in a place to ping each node from evey node across the thunderbolt mesh using IPv4 or IPv6 as you see fit.
 
+### IMPORTAT - you need to do this to stop SDN breaking you in future 
 if all is working issue a `cp /etc/frr/frr.conf /etc/frr/frr.conf.local` this is because when enabling proxmox SDN proxmox will overwrite frr.conf - however it will read the .local file and apply that.  So do this copy whenever you have edited the conf using vtysh or by hand.
 
 I haven't yet tested to see if we can have just the settings in frr.conf.local when SDN is not configured....
diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -24,7 +24,7 @@ I have included this for completeness, i only run the FC00:: IPv6 network as cep
 ## Defining thunderbolt network
 
 Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
-Remember X should match you node number, so for example 1,2 or 3.  using this file ensures proxmox never overwites this configuration
+There should no lober be any IP addresses in this file for lo and lo:6
 
 ```     
 allow-hotplug en05

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -111,7 +111,7 @@ openfabric passive
 exit
 !
 router openfabric 1
-net 49.0000.0000.000X.00
+net 49.0000.0000.000x.00
 exit
 !
 exit

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -87,7 +87,7 @@ fi
 3. enter the configure mode with `configure`
 4. Apply the bellow configuration (it is possible to cut and paste this into the shell instead of typing it manually, you may need to press return to set the last !.  Also check there were no errors in repsonse to the paste text.).
 
-**Note: the X should be the number of the node you are working on** For example node 1 would use 1 in place of X**
+**Note: the X should be the number of the node you are working on** For example node 1 would use 1 in place of X
 ```
 ip forwarding
 ipv6 forwarding

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -87,7 +87,7 @@ fi
 3. enter the configure mode with `configure`
 4. Apply the bellow configuration (it is possible to cut and paste this into the shell instead of typing it manually, you may need to press return to set the last !.  Also check there were no errors in repsonse to the paste text.).
 
-**Note: the X should be the number of the node you are working on, as an example -  0.0.0.1, 0.0.0.2 or 0.0.0.3.**
+**Note: the X should be the number of the node you are working on** For example node 1 would use 1 in place of X**
 ```
 ip forwarding
 ipv6 forwarding
@@ -104,7 +104,7 @@ exit
 !
 interface lo
 ip address 10.0.0.8x/32
-ipv6 address fc00::81/128
+ipv6 address fc00::8x/128
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -13,6 +13,7 @@ Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 - move IP address configuration from `interfaces.d/thundebolt` to frr configuration
 - new approach to remove dependecy on post-up, new script in if-up.d that logs to systemlog
 - reminder to copy frr.conf > frr.conf.local to prevent breakage if you enable Proxmox SDN
+- dependent on the changes to the udev link scripts [here](/67fdc9a517faefa68f730f82d7fa3570#set-interfaces-to-up-on-reboots-and-cable-insertions)
 
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -11,7 +11,8 @@ This changes the previous file design thanks to @NRGNet and @tisayama to make th
 Notable changes from original version [here](/4c664734535da122f4ab2951b22b2085)
 
 - move IP address configuration from `interfaces.d/thundebolt` to frr configuration
-- new approach to remove dependecy on post-up
+- new approach to remove dependecy on post-up, new script in if-up.d that logs to systemlog
+- reminder to copy frr.conf > frr.conf.local to prevent breakage if you enable Proxmox SDN
 
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -63,15 +63,16 @@ this should make IPv4 more stable for all users (i ended up seeing IPv4 issues t
 
 ```
 #!/bin/bash
+# note the looger entries log to the system journal in the pve UI etc
 
 INTERFACE=$IFACE
 
 if [ "$INTERFACE" = "en05" ] || [ "$INTERFACE" = "en06" ]; then
     logger "Attempting to restart frr.service for $INTERFACE"
     if systemctl restart frr.service; then
-        logger "Successfully restarted frr.service for $INTERFACE"
+        logger -t SCYTO "   [SCYTO SCRIPT ] Successfully restarted frr.service for $INTERFACE"
     else
-        logger "Failed to restart frr.service for $INTERFACE"
+        logger -t SCYTO "   [SCYTO SCRIPT ] Failed to restart frr.service for $INTERFACE"
     fi
 fi
 ```

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -64,8 +64,15 @@ this should make IPv4 more stable for all users (i ended up seeing IPv4 issues t
 ```
 #!/bin/bash
 
-if [[ $IFACE == "en05" ]] || [[ $IFACE == "en06" ]]; then
- sleep 5 && /usr/bin/systemctl restart frr.service
+INTERFACE=$IFACE
+
+if [ "$INTERFACE" = "en05" ] || [ "$INTERFACE" = "en06" ]; then
+    logger "Attempting to restart frr.service for $INTERFACE"
+    if systemctl restart frr.service; then
+        logger "Successfully restarted frr.service for $INTERFACE"
+    else
+        logger "Failed to restart frr.service for $INTERFACE"
+    fi
 fi
 ```
 

diff --git a/dual-stack-openfabric-mesh-v2.md b/dual-stack-openfabric-mesh-v2.md
@@ -62,6 +62,8 @@ this should make IPv4 more stable for all users (i ended up seeing IPv4 issues t
 2. add to file the following
 
 ```
+#!/bin/bash
+
 if [[ $IFACE == "en05" ]] || [[ $IFACE == "en06" ]]; then
  sleep 5 && /usr/bin/systemctl restart frr.service
 fi
No results found