andersonrobot · December 27, 2024 10:12 · Sep 8, 2024 · Sep 8, 2024 · Sep 8, 2024 · Sep 8, 2024
diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -36,17 +36,6 @@ iface en06 inet manual
 ```
 Save file, repeat on each node.
 
-## Mitigate Timing Issues
-
-### Add post-up command to /etc/network/interfaces
-sudo nano `/etc/network/interfaces`
-
-Add `post-up /usr/bin/systemctl restart frr.service`as the _last_ line in the file
-
-#### NOTE for Minisforum MS-01 users
-make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` instead 
-this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
-
 ## Enable IPv4 and IPv6 forwarding
 1. use `nano /etc/sysctl.conf` to open the file
 2. uncomment `#net.ipv6.conf.all.forwarding=1` (remove the # symbol)
@@ -65,6 +54,18 @@ Install Free Range Routing (FRR) `apt install frr`
 2. save the file
 3. restart the service with `systemctl restart frr`
 
+### Mitigate FRR Timing Issues at Boot
+
+### Add post-up command to /etc/network/interfaces
+sudo nano `/etc/network/interfaces`
+
+Add `post-up /usr/bin/systemctl restart frr.service`as the _last_ line in the file (this should go after the line that starts `source`)
+
+#### NOTE for Minisforum MS-01 users
+make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` instead 
+this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
+
+
 
 ### Configure OpenFabric (perforn on all nodes)
 1. enter the FRR shell with `vtysh`

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -34,12 +34,19 @@ allow-hotplug en06
 iface en06 inet manual
         mtu 65520
 ```
+Save file, repeat on each node.
+
+## Mitigate Timing Issues
+
+### Add post-up command to /etc/network/interfaces
+sudo nano `/etc/network/interfaces`
+
+Add `post-up /usr/bin/systemctl restart frr.service`as the _last_ line in the file
+
 #### NOTE for Minisforum MS-01 users
 make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` instead 
 this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
 
-Save file, repeat on each node.
-
 ## Enable IPv4 and IPv6 forwarding
 1. use `nano /etc/sysctl.conf` to open the file
 2. uncomment `#net.ipv6.conf.all.forwarding=1` (remove the # symbol)

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -33,9 +33,6 @@ iface en05 inet manual
 allow-hotplug en06
 iface en06 inet manual
         mtu 65520
-
-#this must always be the last line in this file
-post-up /usr/bin/systemctl restart frr.service
 ```
 #### NOTE for Minisforum MS-01 users
 make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` instead 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -37,7 +37,7 @@ iface en06 inet manual
 #this must always be the last line in this file
 post-up /usr/bin/systemctl restart frr.service
 ```
-#### MOTE for Minisforum MS-01 users
+#### NOTE for Minisforum MS-01 users
 make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` instead 
 this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -37,8 +37,9 @@ iface en06 inet manual
 #this must always be the last line in this file
 post-up /usr/bin/systemctl restart frr.service
 ```
-#### note for MS-01 users
-make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
+#### MOTE for Minisforum MS-01 users
+make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` instead 
+this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
 
 Save file, repeat on each node.
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -37,6 +37,8 @@ iface en06 inet manual
 #this must always be the last line in this file
 post-up /usr/bin/systemctl restart frr.service
 ```
+#### note for MS-01 users
+make the post-up line above read `sleep 5 && post-up /usr/bin/systemctl restart frr.service` this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
 
 Save file, repeat on each node.
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -45,7 +45,7 @@ Save file, repeat on each node.
 2. uncomment `#net.ipv6.conf.all.forwarding=1` (remove the # symbol)
 3. uncomment `#net.ipv4.ip_forward=1` (remove the # symbol)
 4. save the file
-5. reboot?
+5. issue `reboot now` for a complete reboot
 
 ## FRR Setup
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -25,6 +25,14 @@ iface lo:0 inet static
 auto lo:6
 iface lo:6 inet static
         address fc00::8X/128
+        
+allow-hotplug en05
+iface en05 inet manual
+        mtu 65520
+
+allow-hotplug en06
+iface en06 inet manual
+        mtu 65520
 
 #this must always be the last line in this file
 post-up /usr/bin/systemctl restart frr.service

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -12,7 +12,7 @@ This will result in an IPv4 and IPv6 routable mesh network that can survive any
 I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
 
 
-### Defining thunderbolt network
+## Defining thunderbolt network
 
 Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
 Remember X should match you node number, so for example 1,2 or 3.

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -15,6 +15,7 @@ I have included this for completeness, i only run the FC00:: IPv6 network as cep
 ### Defining thunderbolt network
 
 Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
+Remember X should match you node number, so for example 1,2 or 3.
 
 ```
 auto lo:0
@@ -25,20 +26,6 @@ auto lo:6
 iface lo:6 inet static
         address fc00::8X/128
 
-post-up /usr/bin/systemctl restart frr.service
-```
-
-so on the first node it would look comething like this:
-
-```
-auto lo:0
-iface lo:0 inet static
-        address 10.0.0.81/32
-
-auto lo:6
-iface lo:6 inet static
-        address fc00::81/128
-
 #this must always be the last line in this file
 post-up /usr/bin/systemctl restart frr.service
 ```

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -1,5 +1,7 @@
 # Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing
 
+[this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
+
 This assumes you are running Proxmox 8.2 and that the line `source /etc/network/interfaces.d/*` is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).
 
 This changes the previous file design thanks to @NRGNet for the suggestions to move thunderbolt settings to a file in /etc/network/interfaces.d  it makes the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)
@@ -9,7 +11,8 @@ This will result in an IPv4 and IPv6 routable mesh network that can survive any
 ## NOTES on Dual Stack
 I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
 
-[this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
+
+### Defining thunderbolt network
 
 Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -6,7 +6,8 @@ This changes the previous file design thanks to @NRGNet for the suggestions to m
 
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 
-Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
+## NOTES on Dual Stack
+I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6.  For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
 
 [this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -10,23 +10,7 @@ Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the
 
 [this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
 
-## Create Loopback interfaces
-Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 addrees and that these addresses stay constant no matter what.
-
-Add the following to each node using `nano /etc/network/interfaces` this to remind you not to edit en05 and en06 in the GUI
-
-This  fragment should go between the existing `auto lo` section and adapater sections.
-Remeber for each node the X should be 1, 2 or 3 depending on the node. 
-
-```
-iface en05 inet manual
-#do not edit it GUI
-
-iface en06 inet manual
-#do not edit in GUI
-```
-
-Now create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
+Create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
 
 ```
 auto lo:0

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -15,7 +15,8 @@ Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 ad
 
 Add the following to each node using `nano /etc/network/interfaces` this to remind you not to edit en05 and en06 in the GUI
 
-This should go uder the `auto lo` section and for each node the X should be 1, 2 or depending on the node 
+This  fragment should go between the existing `auto lo` section and adapater sections.
+Remeber for each node the X should be 1, 2 or 3 depending on the node. 
 
 ```
 iface en05 inet manual
@@ -24,7 +25,8 @@ iface en05 inet manual
 iface en06 inet manual
 #do not edit in GUI
 ```
-Now create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the follwing
+
+Now create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the following
 
 ```
 auto lo:0
@@ -34,6 +36,8 @@ iface lo:0 inet static
 auto lo:6
 iface lo:6 inet static
         address fc00::8X/128
+
+post-up /usr/bin/systemctl restart frr.service
 ```
 
 so on the first node it would look comething like this:

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -1,5 +1,9 @@
 # Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing
 
+This assumes you are running Proxmox 8.2 and that the line `source /etc/network/interfaces.d/*` is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).
+
+This changes the previous file design thanks to @NRGNet for the suggestions to move thunderbolt settings to a file in /etc/network/interfaces.d  it makes the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)
+
 This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 
 Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
@@ -8,10 +12,20 @@ Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the
 
 ## Create Loopback interfaces
 Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 addrees and that these addresses stay constant no matter what.
-Add the following to each node using `nano /etc/network/interfaces`
+
+Add the following to each node using `nano /etc/network/interfaces` this to remind you not to edit en05 and en06 in the GUI
 
 This should go uder the `auto lo` section and for each node the X should be 1, 2 or depending on the node 
 
+```
+iface en05 inet manual
+#do not edit it GUI
+
+iface en06 inet manual
+#do not edit in GUI
+```
+Now create a new file using `nano /etc/network/interfaces.d/thunderbolt` and populate with the follwing
+
 ```
 auto lo:0
 iface lo:0 inet static
@@ -25,24 +39,15 @@ iface lo:6 inet static
 so on the first node it would look comething like this:
 
 ```
-...
-auto lo
-iface lo inet loopback
- 
 auto lo:0
 iface lo:0 inet static
         address 10.0.0.81/32
 
 auto lo:6
 iface lo:6 inet static
         address fc00::81/128
-...
-```
 
-also add this is as the last line to the interfaces file unless there is a sources file in which case put it immeditately before the sources lines (or delete the sources line)
-
-```
-# This must be the last line in the file unless there is a sources line in which case put this immediately above the sources line (or delete the sources line)
+#this must always be the last line in this file
 post-up /usr/bin/systemctl restart frr.service
 ```
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -39,10 +39,10 @@ iface lo:6 inet static
 ...
 ```
 
-also add this is as the last line to the interfaces file
+also add this is as the last line to the interfaces file unless there is a sources file in which case put it immeditately before the sources lines (or delete the sources line)
 
 ```
-# This must be the last line in the file
+# This must be the last line in the file unless there is a sources line in which case put this immediately above the sources line (or delete the sources line)
 post-up /usr/bin/systemctl restart frr.service
 ```
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -78,11 +78,6 @@ Install Free Range Routing (FRR) `apt install frr`
 ip forwarding
 ipv6 forwarding
 !
-frr version 8.5.2
-frr defaults traditional
-hostname pve1
-service integrated-vtysh-config
-!
 interface en05
 ip router openfabric 1
 ipv6 router openfabric 1

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -39,6 +39,13 @@ iface lo:6 inet static
 ...
 ```
 
+also add this is as the last line to the interfaces file
+
+```
+# This must be the last line in the file
+post-up /usr/bin/systemctl restart frr.service
+```
+
 Save file, repeat on each node.
 
 ## Enable IPv4 and IPv6 forwarding

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -77,23 +77,23 @@ hostname pve1
 service integrated-vtysh-config
 !
 interface en05
- ip router openfabric 1
- ipv6 router openfabric 1
+ip router openfabric 1
+ipv6 router openfabric 1
 exit
 !
 interface en06
- ip router openfabric 1
- ipv6 router openfabric 1
+ip router openfabric 1
+ipv6 router openfabric 1
 exit
 !
 interface lo
- ip router openfabric 1
- ipv6 router openfabric 1
- openfabric passive
+ip router openfabric 1
+ipv6 router openfabric 1
+openfabric passive
 exit
 !
 router openfabric 1
- net 49.0000.0000.000X.00
+net 49.0000.0000.000X.00
 exit
 !
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -96,6 +96,7 @@ router openfabric 1
  net 49.0000.0000.000X.00
 exit
 !
+
 ```
 5. you may need to pres return after the last `!` to get to a new line - if so do this
 6. exit the configure mode with the command `end`

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -57,7 +57,7 @@ Install Free Range Routing (FRR) `apt install frr`
 
 1. edit the frr daemons file (`nano /etc/frr/daemons`) to change `fabricd=no` to `fabricd=yes` 
 2. save the file
-3. restart the service with systemctl restart frr
+3. restart the service with `systemctl restart frr`
 
 
 ### Configure OpenFabric (perforn on all nodes)

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -50,9 +50,14 @@ Save file, repeat on each node.
 
 ## FRR Setup
 
+### Install FRR
+Install Free Range Routing (FRR) `apt install frr`
+
 ### Enable the fabricd daemon
 
-edit the frr daemons file (`nano /etc/frr/daemons`) to change `fabricd=no` to `fabricd=yes` 
+1. edit the frr daemons file (`nano /etc/frr/daemons`) to change `fabricd=no` to `fabricd=yes` 
+2. save the file
+3. restart the service with systemctl restart frr
 
 
 ### Configure OpenFabric (perforn on all nodes)

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -60,7 +60,8 @@ edit the frr daemons file (`nano /etc/frr/daemons`) to change `fabricd=no` to `f
 2. optionally show the current config with `show running-config`
 3. enter the configure mode with `configure`
 4. Apply the bellow configuration (it is possible to cut and paste this into the shell instead of typing it manually, you may need to press return to set the last !.  Also check there were no errors in repsonse to the paste text.).
-Note: the X should be the number of the node you are working on, so for my stetup this would 0.0.0.1, 0.0.0.2 or 0.0.0.3.
+
+**Note: the X should be the number of the node you are working on, as an example -  0.0.0.1, 0.0.0.2 or 0.0.0.3.**
 ```
 ip forwarding
 ipv6 forwarding

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -49,9 +49,10 @@ Save file, repeat on each node.
 5. reboot?
 
 ## FRR Setup
-This is the content for FRR - remember to increment the openfabric net id on each node you use this on where you see X
 
-edit the frr daemons file to change `fabricd=no` to `fabricd=yes` 
+### Enable the fabricd daemon
+
+edit the frr daemons file (`nano /etc/frr/daemons`) to change `fabricd=no` to `fabricd=yes` 
 
 
 ### Configure OpenFabric (perforn on all nodes)

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -1,6 +1,6 @@
 # Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing
 
-This will result in an IPv4 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
+This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
 
 Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -49,7 +49,7 @@ Save file, repeat on each node.
 5. reboot?
 
 ## FRR Setup
-This is the content for FRR - rememvber to increment the router-ids on each node you use this on where you see X
+This is the content for FRR - remember to increment the openfabric net id on each node you use this on where you see X
 
 edit the frr daemons file to change `fabricd=no` to `fabricd=yes` 
 

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -46,6 +46,7 @@ Save file, repeat on each node.
 2. uncomment `#net.ipv6.conf.all.forwarding=1` (remove the # symbol)
 3. uncomment `#net.ipv4.ip_forward=1` (remove the # symbol)
 4. save the file
+5. reboot?
 
 ## FRR Setup
 This is the content for FRR - rememvber to increment the router-ids on each node you use this on where you see X

diff --git a/dual-stack-openfabric-mesh.md b/dual-stack-openfabric-mesh.md
@@ -4,6 +4,8 @@ This will result in an IPv4 routable mesh network that can survive any one node
 
 Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky.... 
 
+[this gist is part of this series](/76e94832927a89d977ea989da157e9dc)
+
 ## Create Loopback interfaces
 Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 addrees and that these addresses stay constant no matter what.
 Add the following to each node using `nano /etc/network/interfaces`