Skip to content

Instantly share code, notes, and snippets.

@scyto

scyto/lan-access-to-mesh.md

Last active November 1, 2025 13:30
Show Gist options
  • Save scyto/c0df83c269c5f5c192cb8a08a0d4a559 to your computer and use it in GitHub Desktop.
Save scyto/c0df83c269c5f5c192cb8a08a0d4a559 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. scyto revised this gist Apr 29, 2025. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -110,9 +110,9 @@ This has only be tested on an EFG running network app 9.1.92
    > - the BGP Router ID can be anything really, convention makes it the router IPv4 address
    > - The LAN addresses are you normal LAN addresses of you routers LAN port
    ### Prepare a BGP conf file for upload to Unifie OS
    ### Prepare a BGP conf file for upload to Unifi OS

    Remeber to use something that uses linux text formatting.
    Remember to use something that uses linux text formatting.

    ```bash
    router bgp 65001
  2. scyto revised this gist Apr 29, 2025. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -49,7 +49,7 @@ REMEMBER ceph clients want to access the MONSs / OSDs / MGRs and MDSs on the `lo
    1. nano `/etc/frr/daemons` change `bgpd=no` to `bgpd=yes` and save the file
    2. then `systemctl reload frr`

    ### Add BGP confing to frr.conf
    ### Add BGP confing to frr.conf (node `pve1` example)

    > use the settings from the table above and remember to change them as needed for each node
  3. scyto revised this gist Apr 29, 2025. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -57,7 +57,7 @@ REMEMBER ceph clients want to access the MONSs / OSDs / MGRs and MDSs on the `lo

    ```
    router bgp 65001
    bgp router-id 192.0.2.81
    bgp router-id 192.168.1.81
    no bgp ebgp-requires-policy
    neighbor 2001:db8:1000:1::82 remote-as 65001
    neighbor 2001:db8:1000:1::83 remote-as 65001
    @@ -276,6 +276,7 @@ If you seen nothing or something missing the mostly like issues are:
    Example frr.conf after these changes (this is from my pve1)

    <details>
    <summary>Click me</summary>

    ```
    root@pve1 16:29:49 ~ # cat /etc/frr/frr.conf
    @@ -345,4 +346,5 @@ router bgp 65001
    neighbor 192.168.1.83 activate
    neighbor 192.168.1.1 activate
    exit-address-family
    ```
    ```
    </details>
  4. scyto revised this gist Apr 29, 2025. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -25,7 +25,7 @@ REMEMBER ceph clients want to access the MONSs / OSDs / MGRs and MDSs on the `lo
    ## FRR BGP pve node Settings
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | BGP Router-ID | `192.168.1.81` | `192.168.1.82` | `192.168.1.83` |
    | BGP ASN | `65001` | `65001` | `65001` |
    | IPv6 Neighbors<br>(Node LAN IP) | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors<br>(Node LAN IP) | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
  5. scyto revised this gist Apr 29, 2025. 1 changed file with 74 additions and 0 deletions.
    74 changes: 74 additions & 0 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -272,3 +272,77 @@ If you seen nothing or something missing the mostly like issues are:
    1. MTU mismatch between br0 on the router and vmbr0 on the proxmox nodes
    2. a firewall blocking BGP packets somehwere

    ---
    Example frr.conf after these changes (this is from my pve1)

    <details>

    ```
    root@pve1 16:29:49 ~ # cat /etc/frr/frr.conf
    frr version 8.5.2
    frr defaults datacenter
    hostname pve1
    log syslog informational
    service integrated-vtysh-config
    interface en05
    ip router openfabric 1
    ipv6 router openfabric 1
    openfabric hello-interval 1
    openfabric hello-multiplier 3
    openfabric csnp-interval 5
    openfabric psnp-interval 2
    exit
    interface en06
    ip router openfabric 1
    ipv6 router openfabric 1
    openfabric hello-interval 1
    openfabric hello-multiplier 3
    openfabric csnp-interval 5
    openfabric psnp-interval 2
    exit
    interface lo
    ip router openfabric 1
    ipv6 router openfabric 1
    openfabric passive
    exit
    interface vmbr100
    ip router openfabric 1
    ipv6 router openfabric 1
    openfabric passive
    exit
    router openfabric 1
    net 49.0000.0000.0081.00
    lsp-gen-interval 5
    exit
    router bgp 65001
    bgp router-id 192.168.1.81
    no bgp ebgp-requires-policy
    neighbor 2600:a801:830:1::82 remote-as 65001
    neighbor 2600:a801:830:1::83 remote-as 65001
    neighbor 2600:a801:830:1::1 remote-as 65001
    neighbor 192.168.1.82 remote-as 65001
    neighbor 192.168.1.83 remote-as 65001
    neighbor 192.168.1.1 remote-as 65001
    address-family ipv6 unicast
    network fc00::81/128
    network fc00:81::/64
    neighbor 2600:a801:830:1::82 activate
    neighbor 2600:a801:830:1::83 activate
    neighbor 2600:a801:830:1::1 activate
    exit-address-family
    address-family ipv4 unicast
    network 10.0.0.81/32
    network 10.0.81.0/24
    neighbor 192.168.1.82 activate
    neighbor 192.168.1.83 activate
    neighbor 192.168.1.1 activate
    exit-address-family
    ```
  6. scyto revised this gist Apr 29, 2025. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,7 +6,9 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    ## Goals
    - let any client on LAN access the mesh
    - avoid setting static routes on my router
    - enable support for routing topology changes without having to reconfigure routee
    - enable support for routing topology changes without having to reconfigure router

    REMEMBER ceph clients want to access the MONSs / OSDs / MGRs and MDSs on the `lo` interface loopback addresses - thats the goal!

    ## Overview
    - BGP is used to advertise routes for both LAN based clients and VM based clients
  7. scyto revised this gist Apr 29, 2025. 1 changed file with 35 additions and 1 deletion.
    36 changes: 35 additions & 1 deletion lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -233,6 +233,40 @@ Displayed 6 routes and 12 total paths
    ---
    # Bonus Tip: Monitor Convergence

    Go to a mesh node and run `ip -6 route show && ip route show`
    If you don't see the right things above see double check the router can see its neighbors wth `vtysh -c "show bgp sum"` you should get something like this:

    ```
    IPv4 Unicast Summary (VRF default):
    BGP router identifier 192.168.1.1, local AS number 65001 vrf-id 0
    BGP table version 6
    RIB entries 11, using 2024 bytes of memory
    Peers 6, using 4338 KiB of memory
    Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
    pve1(192.168.1.81) 4 65001 86 85 0 0 0 00:04:09 2 0 N/A
    pve2(192.168.1.82) 4 65001 86 85 0 0 0 00:04:08 2 0 N/A
    pve3(192.168.1.83) 4 65001 86 85 0 0 0 00:04:08 2 0 N/A
    pve1(2001:db8:1000:1::81) 4 65001 88 86 0 0 0 00:04:09 2 0 N/A
    pve2(2001:db8:1000:1::82) 4 65001 88 86 0 0 0 00:04:08 2 0 N/A
    pve3(2001:db8:1000:1::83) 4 65001 88 86 0 0 0 00:04:08 2 0 N/A
    Total number of neighbors 6
    IPv6 Unicast Summary (VRF default):
    BGP router identifier 192.168.1.1, local AS number 65001 vrf-id 0
    BGP table version 6
    RIB entries 11, using 2024 bytes of memory
    Peers 3, using 2169 KiB of memory
    Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
    pve1(2001:db8:1000:1::81) 4 65001 88 86 0 0 0 00:04:09 2 0 N/A
    pve2(2001:db8:1000:1::82) 4 65001 88 86 0 0 0 00:04:08 2 0 N/A
    pve3(2001:db8:1000:1::83) 4 65001 88 86 0 0 0 00:04:08 2 0 N/A
    Total number of neighbors 3
    ```

    If you seen nothing or something missing the mostly like issues are:
    1. MTU mismatch between br0 on the router and vmbr0 on the proxmox nodes
    2. a firewall blocking BGP packets somehwere

  8. scyto revised this gist Apr 29, 2025. 1 changed file with 6 additions and 40 deletions.
    46 changes: 6 additions & 40 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,7 +13,7 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    - BGP P2P links have to be used as other BGP mechnanisms didn't seem to work
    - BGP routes are explictly defined for dvertisement, no broadcast is used

    ## Asummptions
    ## Asumptions
    - Ubiquiti unifi OS router with BGP feature (EFG and maybe others)
    - all previous gists have been followed and are working perfectly, this will only detail changes to the overall setup
    - you have a true dual stack setup on your LAN (if you only have IPv4 including for ceph you drop the IPv6 sections)
    @@ -80,7 +80,7 @@ router bgp 65001
    neighbor 192.168.1.1 activate
    exit-address-family
    ```
    > Remeber to change the IP addreses to match the table above the easy way to think about this is:
    > Remember to change the IP addreses to match the table above the easy way to think about this is:
    > - network = subnets on this node i want to advertise
    > - neighbor = other routers/nodes i have to talk to
    >
    @@ -173,8 +173,8 @@ router bgp 65001
    > - I found some times times frr.service can crash on UI whem you upload, if it does just restart it
    > - The router learns `fc00::8x/128`, `fc00:8x::/64`, `10.0.0.8x/32`, and `10.0.8x.0/24` routes from your the nodes.
    > - you can use the following command to check everything looks good:
    > - `vtysh -c "show bgp ipv4 unicast bestpath"`
    > - `vtysh -c "show bgp ipv6 unicast bestpath"`
    > - `vtysh -c "show bgp ipv6 unicast summary"`
    > - `vtysh -c "show bgp ipv4 unicast summary"`
    it will look something like this:

    @@ -229,44 +229,10 @@ RPKI validation codes: V valid, I invalid, N Not found
    * i 192.168.1.83 0 100 0 i
    Displayed 6 routes and 12 total paths
    ```
    ---
    # Quick Checklist
    - [x] Configure `/etc/network/interfaces` correctly on each node.
    - [x] Assign unique IPv6 and IPv4 loopbacks.
    - [x] Setup `openfabric` on Thunderbolt interfaces.
    - [x] Advertise loopback and VM networks into BGP.
    - [x] Peering between Proxmox nodes and LAN router.
    - [x] Configure VMs to route through `vmbr100`.
    ---
    # Bonus Tip: Monitor Convergence

    You can monitor bgp mesh convergence using on any node or the router:
    ```bash
    vtysh -c "show bgp ipv6 unicast summary"
    vtysh -c "show bgp ipv4 unicast summary"
    ```

    You can monitor openfabric mesh convergence on any node (but not router) using:

    ```bash
    vtysh -c "show openfabfric toplology"
    ```

    Super tip you can shorten word like above "sh open topo" will work instead

    ---



    ---
    Go to a mesh node and run `ip -6 route show && ip route show`

    # Conclusion

    thanks for listening (there was some chipper and annoying chatgpt output here before)
    i will be posting how to join ceph from the docker client over in my swarm gist later - i have instructions written up that just need to be tested....
  9. scyto revised this gist Apr 29, 2025. 1 changed file with 28 additions and 15 deletions.
    43 changes: 28 additions & 15 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -96,17 +96,21 @@ router bgp 65001
    This has only be tested on an EFG running network app 9.1.92

    ### Router Settings
    BGP ASN :65001
    BGP Router ID :192.168.1.1

    1. create a file called bgp.conf
    2. populate it with the contents below
    3. in unifi network application go to `settings > routing > bgp`
    4. name = ceph-mesh
    5. device = name of router (should be on the drop down)
    6. click upload and upload file
    7. do NOT select the check box `override wan monitors` (uncheck it it is checked)
    8. you can shell into the router and use `xxxxx` and `yyyyyy` to check the routes formed correctly
    | Field | Value |
    |:-----------------|:-------------:|
    | BGP ASN | `65001` |
    | BGP Router ID | `192.168.1.1` |
    | IPv4 LAN Address | `192.168.1.1` |
    | IPv6 LAN Address | 2001:db8:1000:1::1/64 |

    > **notes
    > - the BGP AS must be in the range of `64512-65534` or your ISP will get very cross at you ;-)
    > - the BGP Router ID can be anything really, convention makes it the router IPv4 address
    > - The LAN addresses are you normal LAN addresses of you routers LAN port
    ### Prepare a BGP conf file for upload to Unifie OS

    Remeber to use something that uses linux text formatting.

    ```bash
    router bgp 65001
    @@ -153,12 +157,21 @@ router bgp 65001
    exit-address-family
    ```

    > **note
    > - the format above is ordered for easy reading `vtysh -c "show running-config"` will show a different layout
    > - the settings above wont be written to `/etc/frr/frr.conf` - so don't worry if thats empty
    ### Upload to the Unifi OS router

    1. in unifi network application go to `settings > routing > bgp`
    2. name = ceph-mesh
    3. device = name of router (should be on the drop down)
    4. click upload and upload file
    5. do NOT select the check box `override wan monitors` (uncheck it it is checked)

    > **Notes:**
    > - The router learns `fc00::8x/128`, `fc00:8x::/64`, `10.0.0.8x/32`, and `10.0.8x.0/24` routes from your the nodes.
    > - The LAN can access VMs without NAT.
    > - I found some times times frr.service can crash on UI whem you upload, if it does just restart it
    > - the format above is ordered for east reading `vtysh -c "show running-config"` will show a different layout
    > - the settings above wont be written to `/etc/frr/frr.conf` - so don't worry if thats empty
    > - The router learns `fc00::8x/128`, `fc00:8x::/64`, `10.0.0.8x/32`, and `10.0.8x.0/24` routes from your the nodes.
    > - you can use the following command to check everything looks good:
    > - `vtysh -c "show bgp ipv4 unicast bestpath"`
    > - `vtysh -c "show bgp ipv6 unicast bestpath"`
  10. scyto revised this gist Apr 29, 2025. 1 changed file with 6 additions and 1 deletion.
    7 changes: 6 additions & 1 deletion lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -20,10 +20,11 @@ I have other devices that need to access the ceph mesh that are on my LAN. This

    ---

    ## FRR BGP Settings
    ## FRR BGP pve node Settings
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | BGP ASN | `65001` | `65001` | `65001` |
    | IPv6 Neighbors<br>(Node LAN IP) | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors<br>(Node LAN IP) | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Routes<br>(mesh network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    @@ -94,6 +95,10 @@ router bgp 65001

    This has only be tested on an EFG running network app 9.1.92

    ### Router Settings
    BGP ASN :65001
    BGP Router ID :192.168.1.1

    1. create a file called bgp.conf
    2. populate it with the contents below
    3. in unifi network application go to `settings > routing > bgp`
  11. scyto revised this gist Apr 29, 2025. 1 changed file with 13 additions and 13 deletions.
    26 changes: 13 additions & 13 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -30,7 +30,7 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    | IPv4 Routes<br>(mesh network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

    > **notes
    > - `2001:db8:1000:1::` is not my real subnet, this is a subnet resevered for documentation
    > - `2001:db8:1000:1::` is not my real subnet, `2001:db8::` is a subnet resevered for documentation
    > - you should use your subnet addresses as appropriate

    @@ -120,22 +120,22 @@ router bgp 65001
    neighbor 192.168.1.83 update-source 192.168.1.1
    neighbor 192.168.1.83 next-hop-self

    neighbor 2600:a801:830:1::81 remote-as 65001
    neighbor 2600:a801:830:1::81 update-source 2600:a801:830:1::1
    neighbor 2001:db8:1000:1::81 remote-as 65001
    neighbor 22001:db8:1000:1::81 update-source 2001:db8:1000:1::1

    neighbor 2600:a801:830:1::82 remote-as 65001
    neighbor 2600:a801:830:1::82 update-source 2600:a801:830:1::1
    neighbor 2001:db8:1000:1::82 remote-as 65001
    neighbor 22001:db8:1000:1::82 update-source 2001:db8:1000:1::1

    neighbor 2600:a801:830:1::83 remote-as 65001
    neighbor 2600:a801:830:1::83 update-source 2600:a801:830:1::1
    neighbor 2001:db8:1000:1::83 remote-as 65001
    neighbor2001:db8:1000:1::83 update-source 2001:db8:1000:1::1

    address-family ipv6 unicast
    neighbor 2600:a801:830:1::81 activate
    neighbor 2600:a801:830:1::81 next-hop-self
    neighbor 2600:a801:830:1::82 activate
    neighbor 2600:a801:830:1::82 next-hop-self
    neighbor 2600:a801:830:1::83 activate
    neighbor 2600:a801:830:1::83 next-hop-self
    neighbor 2001:db8:1000::81 activate
    neighbor 2001:db8:1000:1::81 next-hop-self
    neighbor 22001:db8:1000:1::82 activate
    neighbor 2001:db8:1000:1::82 next-hop-self
    neighbor 2001:db8:1000:1::83 activate
    neighbor 2001:db8:1000:1::83 next-hop-self
    exit-address-family

    address-family ipv4 unicast
  12. scyto revised this gist Apr 29, 2025. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -83,10 +83,10 @@ router bgp 65001
    > - network = subnets on this node i want to advertise
    > - neighbor = other routers/nodes i have to talk to
    >
    > - Instead of explicitly defining the networks you could use one line that says `redistribute connected` in place of the two `network` values in each familly.
    > - This would advetise all learnt routes (not kernel / static routes) to your router.
    > - I think this would be a better way as there is less reconfiguration.
    > - But given the unpredictably of this in different peoples environments I elected to explicitly define the routes for predictability.
    > Instead of explicitly defining the networks you could use one line that says `redistribute connected` in place of the two `network` values in each familly.
    > This would advetise all learnt routes (not kernel / static routes) to your router.
    > I think this would be a better way as there is less reconfiguration.
    > But given the unpredictably of this in different peoples environments I elected to explicitly define the routes for predictability.


  13. scyto revised this gist Apr 29, 2025. 1 changed file with 9 additions and 3 deletions.
    12 changes: 9 additions & 3 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -32,8 +32,7 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    > **notes
    > - `2001:db8:1000:1::` is not my real subnet, this is a subnet resevered for documentation
    > - you should use your subnet addresses as appropriate
    > - instead of explicitly defining the networks you could use `redistribute connected` in place of the `network` values as this would advetise all learnt routes (not kernel / static routes) to your router.
    > I think this would be a better way as there is less reconfiguration but given the unpredictably of this in different peoples environments I elected to explicitly define the routes for predictability.

    ---
    ## Network Prep
    @@ -80,7 +79,14 @@ router bgp 65001
    neighbor 192.168.1.1 activate
    exit-address-family
    ```

    > Remeber to change the IP addreses to match the table above the easy way to think about this is:
    > - network = subnets on this node i want to advertise
    > - neighbor = other routers/nodes i have to talk to
    >
    > - Instead of explicitly defining the networks you could use one line that says `redistribute connected` in place of the two `network` values in each familly.
    > - This would advetise all learnt routes (not kernel / static routes) to your router.
    > - I think this would be a better way as there is less reconfiguration.
    > - But given the unpredictably of this in different peoples environments I elected to explicitly define the routes for predictability.


  14. scyto revised this gist Apr 29, 2025. 1 changed file with 6 additions and 6 deletions.
    12 changes: 6 additions & 6 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -57,19 +57,19 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    router bgp 65001
    bgp router-id 192.0.2.81
    no bgp ebgp-requires-policy
    neighbor dead:beef:dead:1::82 remote-as 65001
    neighbor dead:beef:dead:1::83 remote-as 65001
    neighbor dead:beef:dead:1::1 remote-as 65001
    neighbor 2001:db8:1000:1::82 remote-as 65001
    neighbor 2001:db8:1000:1::83 remote-as 65001
    neighbor 2001:db8:1000:1::1 remote-as 65001
    neighbor 192.168.1.82 remote-as 65001
    neighbor 192.168.1.83 remote-as 65001
    neighbor 192.168.1.1 remote-as 65001
    address-family ipv6 unicast
    network fc00::81/128
    network fc00:81::/64
    neighbor dead:beef:dead:1::82 activate
    neighbor dead:beef:dead:1::83 activate
    neighbor dead:beef:dead:1:1 activate
    neighbor 2001:db8:1000:1::82 activate
    neighbor 2001:db8:1000:1::83 activate
    neighbor 2001:db8:1000:1::1 activate
    exit-address-family
    address-family ipv4 unicast
  15. scyto revised this gist Apr 29, 2025. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,8 +26,8 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | IPv6 Neighbors<br>(Node LAN IP) | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors<br>(Node LAN IP) | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Routes<br>(network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |
    | IPv6 Routes<br>(mesh network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(mesh network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

    > **notes
    > - `2001:db8:1000:1::` is not my real subnet, this is a subnet resevered for documentation
  16. scyto revised this gist Apr 29, 2025. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -24,8 +24,8 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | IPv6 Neighbors<br>Node LAN IP | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors<br>Node LAN IP | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Neighbors<br>(Node LAN IP) | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors<br>(Node LAN IP) | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Routes<br>(network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

  17. scyto revised this gist Apr 29, 2025. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -24,8 +24,8 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | IPv6 Neighbors | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Neighbors<br>Node LAN IP | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors<br>Node LAN IP | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Routes<br>(network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

  18. scyto revised this gist Apr 29, 2025. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,6 +16,7 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    ## Asummptions
    - Ubiquiti unifi OS router with BGP feature (EFG and maybe others)
    - all previous gists have been followed and are working perfectly, this will only detail changes to the overall setup
    - you have a true dual stack setup on your LAN (if you only have IPv4 including for ceph you drop the IPv6 sections)

    ---

  19. scyto revised this gist Apr 29, 2025. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -28,7 +28,11 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    | IPv6 Routes<br>(network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

    > **note instead of explicitly defining the networks you could use `redistribute connected` in place of the bgp `network` lines this will advetise all learnt routes (but not kernel / static routes). I think this is a better way of doing it, but given the unpredictably of this in different peoples environments elected to explicitly define the routes for predictability.
    > **notes
    > - `2001:db8:1000:1::` is not my real subnet, this is a subnet resevered for documentation
    > - you should use your subnet addresses as appropriate
    > - instead of explicitly defining the networks you could use `redistribute connected` in place of the `network` values as this would advetise all learnt routes (not kernel / static routes) to your router.
    > I think this would be a better way as there is less reconfiguration but given the unpredictably of this in different peoples environments I elected to explicitly define the routes for predictability.
    ---
    ## Network Prep
  20. scyto revised this gist Apr 29, 2025. 1 changed file with 17 additions and 12 deletions.
    29 changes: 17 additions & 12 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,5 @@
    # enable any LAN client to access mesh
    # Enable any LAN client to access mesh
    ## Version 0.5 (2025.04.29)

    I have other devices that need to access the ceph mesh that are on my LAN. This gist is only needed if you want LAN clients to access the ceph mesh.

    @@ -12,34 +13,38 @@ I have other devices that need to access the ceph mesh that are on my LAN. This
    - BGP P2P links have to be used as other BGP mechnanisms didn't seem to work
    - BGP routes are explictly defined for dvertisement, no broadcast is used

    Asummptions
    ## Asummptions
    - Ubiquiti unifi OS router with BGP feature (EFG and maybe others)
    - all previous gists have been followed and are working perfectly, this will only detail changes to the overall setup

    ---

    ## FRR BGP Settings
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | IPv6 Neighbors | `dead:beef:dead:1::82`<br> `dead:beef:dead:1::83`<br>`dead:beef:dead:1::1` |`dead:beef:dead:1::81`<br> `dead:beef:dead:1::83`<br>`dead:beef:dead:1::1` | `dead:beef:dead:1::81`<br> `dead:beef:dead:1::82`<br>`dead:beef:dead:1::1` |
    | IPv6 Neighbors | `2001:db8:1000:1::82`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` |`2001:db8:1000:1::81`<br> `2001:db8:1000:1::83`<br>`2001:db8:1000:1::1` | `2001:db8:1000d:1::81`<br> `2001:db8:1000:1::82`<br>`d2001:db8:1000:1::1` |
    | IPv4 Neighbors | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Routes<br>(network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

    > **note instead of explicitly defining the networks you could use `redistribute connected` in place of the bgp `network` lines this will advetise all learnt routes (but not kernel / static routes). I think this is a better way of doing it, but given the unpredictably of this in different peoples environments elected to explicitly define the routes for predictability.
    interface file changes
    ---
    ## Network Prep

    > **Notes:**
    > - `vmbr0` MTU settinsg must be the same as:
    > - the seetings on your routers br0
    > - must be set the same on you vmbr1000 members (in my case enp86s0)
    > - if not BGP routing will fail
    > - `vmbr100` is for routed VMs hosted on the proxmox nodes
    > - the IPv6 /65 subnet `dead:beef:dead:1::/64` is to hide my real IPv6 addresses
    1. Find out the MTU your router uses - in my case it is 9182
    2. Ensure same MTU is set on vmbr0 - this can be done in the proxmox gui or by setting `mtu 9182` in the `/etc/network/interfaces` in the vmbr0 stanza.

    ## Enable each node to advertise routes to the LAN.

    ### Enable BGP daemon
    1. nano `/etc/frr/daemons` change `bgpd=no` to `bgpd=yes` and save the file
    2. then `systemctl reload frr`

    ### Add BGP confing to frr.conf

    ## Advertise to the LAN with BGP
    > use the settings from the table above and remember to change them as needed for each node
    1. add this to `frr.conf` and restart frr

  21. scyto revised this gist Apr 29, 2025. 1 changed file with 12 additions and 6 deletions.
    18 changes: 12 additions & 6 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,16 +1,22 @@
    # enable any LAN client to access mesh

    ---
    I have other devices that need to access the ceph mesh that are on my LAN. This gist is only needed if you want LAN clients to access the ceph mesh.

    # Phase 2 - Extending the routable mesh to your LAN
    only needed if you want LAN clients to access the ceph mesh
    ## Goals
    - let any client on LAN access the mesh
    - avoid setting static routes on my router
    - enable support for routing topology changes without having to reconfigure routee

    - - Your LAN interface is: `vmbr0`
    ## Overview
    - BGP is used to advertise routes for both LAN based clients and VM based clients
    - BGP P2P links have to be used as other BGP mechnanisms didn't seem to work
    - BGP routes are explictly defined for dvertisement, no broadcast is used
    - Assumes a Ubiquiti unifi OS router with BGP feature (EFG and maybe others)

    ## Phase 2 - FRR BGP Settings
    Asummptions
    - Ubiquiti unifi OS router with BGP feature (EFG and maybe others)
    - all previous gists have been followed and are working perfectly, this will only detail changes to the overall setup

    ## FRR BGP Settings
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
  22. scyto revised this gist Apr 29, 2025. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,7 @@
    # Phase 2 - Extending the routable mesh to your LAN
    only needed if you want LAN clients to access the ceph mesh

    - - Your LAN interface is: `vmbr0`
    - BGP is used to advertise routes for both LAN based clients and VM based clients
    - BGP P2P links have to be used as other BGP mechnanisms didn't seem to work
    - BGP routes are explictly defined for dvertisement, no broadcast is used
  23. scyto revised this gist Apr 29, 2025. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions lan-access-to-mesh.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,11 @@
    # Phase 2 - Extending the routable mesh to your LAN
    only needed if you want LAN clients to access the ceph mesh

    - BGP is used to advertise routes for both LAN based clients and VM based clients
    - BGP P2P links have to be used as other BGP mechnanisms didn't seem to work
    - BGP routes are explictly defined for dvertisement, no broadcast is used
    - Assumes a Ubiquiti unifi OS router with BGP feature (EFG and maybe others)

    ## Phase 2 - FRR BGP Settings
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
  24. scyto renamed this gist Apr 29, 2025. 1 changed file with 0 additions and 0 deletions.
    0 lan-access-to-mesh → lan-access-to-mesh.md
    View file
    File renamed without changes.
  25. scyto created this gist Apr 29, 2025.
    226 changes: 226 additions & 0 deletions lan-access-to-mesh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,226 @@

    ---

    # Phase 2 - Extending the routable mesh to your LAN
    only needed if you want LAN clients to access the ceph mesh

    ## Phase 2 - FRR BGP Settings
    | Field | pve1 | pve2 | pve3 |
    |:-----------------:|:-----------------------:|:-----------------------:|:-----------------------:|
    | BGP Router-ID | `192.0.2.81` | `192.0.2.82` | `192.0.2.83` |
    | IPv6 Neighbors | `dead:beef:dead:1::82`<br> `dead:beef:dead:1::83`<br>`dead:beef:dead:1::1` |`dead:beef:dead:1::81`<br> `dead:beef:dead:1::83`<br>`dead:beef:dead:1::1` | `dead:beef:dead:1::81`<br> `dead:beef:dead:1::82`<br>`dead:beef:dead:1::1` |
    | IPv4 Neighbors | `192.168.1.82`<br>`192.168.1.83`<br>`192.168.1.1`|`192.168.1.81`<br>`192.168.1.83`<br> `192.168.1.1`|`192.168.1.81`<br>`192.168.1.82`<br>`192.168.1.1`|
    | IPv6 Routes<br>(network) |`fc00::81/128`<br>`fc00:81::/64`|`fc00::82/128`<br>`fc00:82::/64`|`fc00::83/128`<br>`fc00:83::/64` |
    | IPv4 Routes<br>(network)| `10.0.0.81/32`<br>`10.0.81.0/24`| `10.0.0.82/32`<br>`10.0.82.0/24`| `10.0.0.83/32`<br>`10.0.83.0/24` |

    > **note instead of explicitly defining the networks you could use `redistribute connected` in place of the bgp `network` lines this will advetise all learnt routes (but not kernel / static routes). I think this is a better way of doing it, but given the unpredictably of this in different peoples environments elected to explicitly define the routes for predictability.

    interface file changes

    > **Notes:**
    > - `vmbr0` MTU settinsg must be the same as:
    > - the seetings on your routers br0
    > - must be set the same on you vmbr1000 members (in my case enp86s0)
    > - if not BGP routing will fail
    > - `vmbr100` is for routed VMs hosted on the proxmox nodes
    > - the IPv6 /65 subnet `dead:beef:dead:1::/64` is to hide my real IPv6 addresses



    ## Advertise to the LAN with BGP

    1. add this to `frr.conf` and restart frr

    ```
    router bgp 65001
    bgp router-id 192.0.2.81
    no bgp ebgp-requires-policy
    neighbor dead:beef:dead:1::82 remote-as 65001
    neighbor dead:beef:dead:1::83 remote-as 65001
    neighbor dead:beef:dead:1::1 remote-as 65001
    neighbor 192.168.1.82 remote-as 65001
    neighbor 192.168.1.83 remote-as 65001
    neighbor 192.168.1.1 remote-as 65001

    address-family ipv6 unicast
    network fc00::81/128
    network fc00:81::/64
    neighbor dead:beef:dead:1::82 activate
    neighbor dead:beef:dead:1::83 activate
    neighbor dead:beef:dead:1:1 activate
    exit-address-family

    address-family ipv4 unicast
    network 10.0.0.81/32
    network 10.0.81.0/24
    neighbor 192.168.1.82 activate
    neighbor 192.168.1.83 activate
    neighbor 192.168.1.1 activate
    exit-address-family
    ```




    ## Ubiuiti UnifiOS Router Integration

    This has only be tested on an EFG running network app 9.1.92

    1. create a file called bgp.conf
    2. populate it with the contents below
    3. in unifi network application go to `settings > routing > bgp`
    4. name = ceph-mesh
    5. device = name of router (should be on the drop down)
    6. click upload and upload file
    7. do NOT select the check box `override wan monitors` (uncheck it it is checked)
    8. you can shell into the router and use `xxxxx` and `yyyyyy` to check the routes formed correctly

    ```bash
    router bgp 65001
    bgp router-id 192.168.1.1
    no bgp ebgp-requires-policy

    neighbor 192.168.1.81 remote-as 65001
    neighbor 192.168.1.81 update-source 192.168.1.1
    neighbor 192.168.1.81 next-hop-self

    neighbor 192.168.1.82 remote-as 65001
    neighbor 192.168.1.82 update-source 192.168.1.1
    neighbor 192.168.1.82 next-hop-self

    neighbor 192.168.1.83 remote-as 65001
    neighbor 192.168.1.83 update-source 192.168.1.1
    neighbor 192.168.1.83 next-hop-self

    neighbor 2600:a801:830:1::81 remote-as 65001
    neighbor 2600:a801:830:1::81 update-source 2600:a801:830:1::1

    neighbor 2600:a801:830:1::82 remote-as 65001
    neighbor 2600:a801:830:1::82 update-source 2600:a801:830:1::1

    neighbor 2600:a801:830:1::83 remote-as 65001
    neighbor 2600:a801:830:1::83 update-source 2600:a801:830:1::1

    address-family ipv6 unicast
    neighbor 2600:a801:830:1::81 activate
    neighbor 2600:a801:830:1::81 next-hop-self
    neighbor 2600:a801:830:1::82 activate
    neighbor 2600:a801:830:1::82 next-hop-self
    neighbor 2600:a801:830:1::83 activate
    neighbor 2600:a801:830:1::83 next-hop-self
    exit-address-family

    address-family ipv4 unicast
    neighbor 192.168.1.81 activate
    neighbor 192.168.1.81 next-hop-self
    neighbor 192.168.1.82 activate
    neighbor 192.168.1.82 next-hop-self
    neighbor 192.168.1.83 activate
    neighbor 192.168.1.83 next-hop-self
    exit-address-family
    ```

    > **Notes:**
    > - The router learns `fc00::8x/128`, `fc00:8x::/64`, `10.0.0.8x/32`, and `10.0.8x.0/24` routes from your the nodes.
    > - The LAN can access VMs without NAT.
    > - I found some times times frr.service can crash on UI whem you upload, if it does just restart it
    > - the format above is ordered for east reading `vtysh -c "show running-config"` will show a different layout
    > - the settings above wont be written to `/etc/frr/frr.conf` - so don't worry if thats empty
    > - you can use the following command to check everything looks good:
    > - `vtysh -c "show bgp ipv4 unicast bestpath"`
    > - `vtysh -c "show bgp ipv6 unicast bestpath"`

    it will look something like this:

    > note the i infront of the IPv6 addresses is just a known display bug in frr 8.1 that my router is running

    ```
    root@EFG:/etc/frr# vtysh -c "show bgp ipv6 unicast"
    BGP table version is 6, local router ID is 192.168.1.1, vrf id 0
    Default local pref 100, local AS 65001
    Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
    i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes: i - IGP, e - EGP, ? - incomplete
    RPKI validation codes: V valid, I invalid, N Not found

    Network Next Hop Metric LocPrf Weight Path
    *>ifc00::81/128 fe80::4a21:bff:fe58:9c45
    0 100 0 i
    *>ifc00::82/128 fe80::4a21:bff:fe56:a5d8
    0 100 0 i
    *>ifc00::83/128 fe80::4a21:bff:fe56:a650
    0 100 0 i
    *>ifc00:81::/64 fe80::4a21:bff:fe58:9c45
    0 100 0 i
    *>ifc00:82::/64 fe80::4a21:bff:fe56:a5d8
    0 100 0 i
    *>ifc00:83::/64 fe80::4a21:bff:fe56:a650
    0 100 0 i

    Displayed 6 routes and 6 total paths
    root@EFG:/etc/frr# vtysh -c "show bgp ipv4 unicast"
    BGP table version is 6, local router ID is 192.168.1.1, vrf id 0
    Default local pref 100, local AS 65001
    Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
    i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes: i - IGP, e - EGP, ? - incomplete
    RPKI validation codes: V valid, I invalid, N Not found

    Network Next Hop Metric LocPrf Weight Path
    *>i10.0.0.81/32 192.168.1.81 0 100 0 i
    * i 192.168.1.81 0 100 0 i
    *>i10.0.0.82/32 192.168.1.82 0 100 0 i
    * i 192.168.1.82 0 100 0 i
    *>i10.0.0.83/32 192.168.1.83 0 100 0 i
    * i 192.168.1.83 0 100 0 i
    *>i10.0.81.0/24 192.168.1.81 0 100 0 i
    * i 192.168.1.81 0 100 0 i
    *>i10.0.82.0/24 192.168.1.82 0 100 0 i
    * i 192.168.1.82 0 100 0 i
    *>i10.0.83.0/24 192.168.1.83 0 100 0 i
    * i 192.168.1.83 0 100 0 i

    Displayed 6 routes and 12 total paths

    ---

    # Quick Checklist

    - [x] Configure `/etc/network/interfaces` correctly on each node.
    - [x] Assign unique IPv6 and IPv4 loopbacks.
    - [x] Setup `openfabric` on Thunderbolt interfaces.
    - [x] Advertise loopback and VM networks into BGP.
    - [x] Peering between Proxmox nodes and LAN router.
    - [x] Configure VMs to route through `vmbr100`.

    ---

    # Bonus Tip: Monitor Convergence

    You can monitor bgp mesh convergence using on any node or the router:

    ```bash
    vtysh -c "show bgp ipv6 unicast summary"
    vtysh -c "show bgp ipv4 unicast summary"
    ```

    You can monitor openfabric mesh convergence on any node (but not router) using:

    ```bash
    vtysh -c "show openfabfric toplology"
    ```

    Super tip you can shorten word like above "sh open topo" will work instead

    ---



    ---

    # Conclusion

    thanks for listening (there was some chipper and annoying chatgpt output here before)
    i will be posting how to join ceph from the docker client over in my swarm gist later - i have instructions written up that just need to be tested....