Skip to content

Instantly share code, notes, and snippets.

@hgomez

hgomez/howto-openstack-zed-rocky-linux-9.1.md

Last active March 10, 2023 16:59
Show Gist options
  • Save hgomez/1ca02730fe56e5fbe047a8a68afb6778 to your computer and use it in GitHub Desktop.
Save hgomez/1ca02730fe56e5fbe047a8a68afb6778 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. hgomez revised this gist Mar 10, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    # How to install OpenStack Zed on Rocky Linux 9.1 - Using DevStack
    # OpenStack Zed on Rocky Linux 9.1 - Using DevStack

    Let see how to deploy OpenStack Zen on Rocky Linux 9.1

  2. hgomez revised this gist Mar 10, 2023. 1 changed file with 30 additions and 0 deletions.
    30 changes: 30 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -557,6 +557,36 @@ Mar 10 17:30:13 rocky2 [email protected][3870]: --- no python application f
    Mar 10 17:30:13 rocky2 [email protected][3870]: [pid: 3870|app: -1|req: -1/12] 10.0.0.42 () {64 vars in 1164 bytes} [Fri Mar 10 17:30:13 2023] GET /volume/ => generated 21 bytes in 0 msecs (HTTP/1.1 500) 3 headers in 102 bytes (0 switches on core 0)
    ```

    or block storage damaged

    ```
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): Metadata corruption detected at xfs_dir3_data_reada_verify+0x3c/0x70 [xfs], xfs_dir3_data_reada block 0x2831068
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): Unmount and run xfs_repair
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): First 128 bytes of corrupted metadata buffer:
    Mar 10 17:32:37 rocky2 kernel: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): Metadata CRC error detected at xfs_dir3_block_read_verify+0xe0/0x120 [xfs], xfs_dir3_block block 0x2831068
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): Unmount and run xfs_repair
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): First 128 bytes of corrupted metadata buffer:
    Mar 10 17:32:37 rocky2 kernel: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Mar 10 17:32:37 rocky2 kernel: XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0xe1/0x140 [xfs]" at daddr 0x2831068 len 8 error 74
    Mar 10 17:32:37 rocky2 [email protected][3884]: #033[00;32mDEBUG keystone.server.flask.request_processing.req_logging [#033[01;36mNone req-9b9bc8a1-6b10-4820-bfac-c90bfddcd51a #033[00;36mNone None#033[00;32m] #033[01;35m#033[00;32mREQUEST_METHOD: `GET`#033[00m #033[00;33m{{(pid=3884) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}#033[00m#033[00m
    ```


    # Conclusion

    You can install OpenStack (latest) on Rocky Linux 9.1 [commit ab8e51eb49](https://opendev.org/openstack/devstack/commit/ab8e51eb49068a8c5004007c18fdfb9b1fcc0954)
  3. hgomez revised this gist Mar 10, 2023. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -550,6 +550,12 @@ Call to Cinder API failed

    > Internal Server Error (HTTP 500)

    Someone is listening to cinder ? (/var/log/messages)

    ```
    Mar 10 17:30:13 rocky2 [email protected][3870]: --- no python application found, check your startup logs for errors ---
    Mar 10 17:30:13 rocky2 [email protected][3870]: [pid: 3870|app: -1|req: -1/12] 10.0.0.42 () {64 vars in 1164 bytes} [Fri Mar 10 17:30:13 2023] GET /volume/ => generated 21 bytes in 0 msecs (HTTP/1.1 500) 3 headers in 102 bytes (0 switches on core 0)
    ```

    # Conclusion

  4. hgomez revised this gist Mar 10, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -544,7 +544,7 @@ export OS_IMAGE_API_VERSION=2
    Call to Cinder API failed

    cinder list
    ERROR: Internal Server Error (HTTP 500)
    > ERROR: Internal Server Error (HTTP 500)

    openstack volume list

  5. hgomez revised this gist Mar 10, 2023. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -541,7 +541,10 @@ export OS_IMAGE_API_VERSION=2
    source admin.rc
    openstack volume list

    Cinder API failed
    Call to Cinder API failed

    cinder list
    ERROR: Internal Server Error (HTTP 500)

    openstack volume list

  6. hgomez revised this gist Mar 10, 2023. 1 changed file with 23 additions and 0 deletions.
    23 changes: 23 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -524,6 +524,29 @@ for http://10.0.0.42/dashboard/project/instances/
    2023-03-10 16:18:16.952448 return view_func(request, *args, **kwargs)
    ```

    cinder status failed

    Define admin.rc

    ```
    export OS_USER_DOMAIN_NAME=Default
    export OS_PROJECT_NAME=admin
    export OS_USERNAME=admin
    export OS_PASSWORD=nomoresecret
    export OS_AUTH_URL=http://10.0.0.42/identity/
    export OS_IDENTITY_API_VERSION=3
    export OS_IMAGE_API_VERSION=2
    ```

    source admin.rc
    openstack volume list

    Cinder API failed

    openstack volume list

    > Internal Server Error (HTTP 500)


    # Conclusion

  7. hgomez revised this gist Mar 10, 2023. 1 changed file with 22 additions and 0 deletions.
    22 changes: 22 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -504,7 +504,29 @@ httpd and memcached services wasn't activated, to make it survive a reboot, acti
    sudo systemctl start memcached.service
    sudo systemctl enable memcached.service

    # Horizon

    While browsing around UI, I noticed errors

    for http://10.0.0.42/dashboard/project/instances/

    ```
    2023-03-10 16:18:16.908037 DEBUG novaclient.v2.client GET call to compute for http://10.0.0.42/compute/v2.1/flavors/detail used request id req-09556b52-adb7-47f3-aef5-b7cf21bec613
    2023-03-10 16:18:16.952418 ERROR django.request Internal Server Error: /dashboard/project/instances/
    2023-03-10 16:18:16.952440 Traceback (most recent call last):
    2023-03-10 16:18:16.952441 File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    2023-03-10 16:18:16.952442 response = get_response(request)
    2023-03-10 16:18:16.952443 File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    2023-03-10 16:18:16.952444 response = wrapped_callback(request, *callback_args, **callback_kwargs)
    2023-03-10 16:18:16.952445 File "/opt/stack/horizon/horizon/decorators.py", line 51, in dec
    2023-03-10 16:18:16.952447 return view_func(request, *args, **kwargs)
    2023-03-10 16:18:16.952447 File "/opt/stack/horizon/horizon/decorators.py", line 35, in dec
    2023-03-10 16:18:16.952448 return view_func(request, *args, **kwargs)
    ```


    # Conclusion

    You can install OpenStack (latest) on Rocky Linux 9.1 [commit ab8e51eb49](https://opendev.org/openstack/devstack/commit/ab8e51eb49068a8c5004007c18fdfb9b1fcc0954)

    But there is some issues and not being a big fan of using master/trunk contents, I should find another way to have OpenStack inside my homelab :)
  8. hgomez revised this gist Mar 10, 2023. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -496,10 +496,13 @@ To be reached from outside, disable firewall
    sudo systemctl stop iptables
    sudo systemctl disable iptables

    httpd service wasn't activated, to make it survive a reboot, activate and enable
    httpd and memcached services wasn't activated, to make it survive a reboot, activate and enable

    sudo systemctl start httpd.service
    sudo systemctl enable httpd.service

    sudo systemctl start memcached.service
    sudo systemctl enable memcached.service

    # Conclusion

  9. hgomez revised this gist Mar 10, 2023. 1 changed file with 10 additions and 0 deletions.
    10 changes: 10 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -491,6 +491,16 @@ Change: ab8e51eb49068a8c5004007c18fdfb9b1fcc0954 Merge "Disable memory_tracker a
    OS Version: Rocky 9.1
    ```

    To be reached from outside, disable firewall

    sudo systemctl stop iptables
    sudo systemctl disable iptables

    httpd service wasn't activated, to make it survive a reboot, activate and enable

    sudo systemctl start httpd.service
    sudo systemctl enable httpd.service

    # Conclusion

    You can install OpenStack (latest) on Rocky Linux 9.1 [commit ab8e51eb49](https://opendev.org/openstack/devstack/commit/ab8e51eb49068a8c5004007c18fdfb9b1fcc0954)
  10. hgomez revised this gist Mar 10, 2023. 1 changed file with 26 additions and 3 deletions.
    29 changes: 26 additions & 3 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -66,14 +66,14 @@ You should see
    # Configure DevStack

    Create a configuration file on devstack folder.
    You should specify VM IP, example 10.0.0.123
    You should specify VM IP, example 10.0.0.41

    [[local|localrc]]
    ADMIN_PASSWORD=secret
    ADMIN_PASSWORD=nomoresecret
    DATABASE_PASSWORD=$ADMIN_PASSWORD
    RABBIT_PASSWORD=$ADMIN_PASSWORD
    SERVICE_PASSWORD=$ADMIN_PASSWORD
    HOST_IP=10.0.0.123
    HOST_IP=10.0.0.41

    # Patch Zed DevStack

    @@ -370,6 +370,29 @@ Problem is not related to Rocky Linux 9.1 as it was quickly patched from master

    I recreated a new VM, and redo operations but using latest devstack (master)

    local.conf used

    ```
    [[local|localrc]]
    ADMIN_PASSWORD=nomoresecret
    DATABASE_PASSWORD=stackdb
    RABBIT_PASSWORD=stackqueue
    SERVICE_PASSWORD=$ADMIN_PASSWORD
    HOST_IP=10.0.0.42
    HOST_IPV6=fe80::a00:27ff:fe45:dc5f/64
    LOGFILE=$DEST/logs/stack.sh.log
    LOGDAYS=2
    SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
    SWIFT_REPLICAS=1
    SWIFT_DATA_DIR=$DEST/data
    ```

    Installation started

    ./stack.sh

    A few minutes later, Victory

    ```
    =========================
    DevStack Component Timing
  11. hgomez revised this gist Mar 10, 2023. 1 changed file with 7 additions and 2 deletions.
    9 changes: 7 additions & 2 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -25,7 +25,7 @@ Make sure that root account allow remote login
    Be sure to be in en_US and UTF-8 (ie: /etc/environment)

    LANG=en_US.utf-8
    LC_ALL=en_US.utf-8
    LC_ALL=en_US.utf-8

    Disable selinux

    @@ -368,7 +368,7 @@ Problem is not related to Rocky Linux 9.1 as it was quickly patched from master

    # Next try, latest OpenStack on Rocky Linux 9.1

    I recreated a new VM, and redo operations but using latest devstack (master)
    I recreated a new VM, and redo operations but using latest devstack (master)

    ```
    =========================
    @@ -467,3 +467,8 @@ DevStack Version: 2023.1
    Change: ab8e51eb49068a8c5004007c18fdfb9b1fcc0954 Merge "Disable memory_tracker and file_tracker in unstask.sh properly" 2023-02-28 06:13:08 +0000
    OS Version: Rocky 9.1
    ```

    # Conclusion

    You can install OpenStack (latest) on Rocky Linux 9.1 [commit ab8e51eb49](https://opendev.org/openstack/devstack/commit/ab8e51eb49068a8c5004007c18fdfb9b1fcc0954)

  12. hgomez revised this gist Mar 10, 2023. 1 changed file with 102 additions and 1 deletion.
    103 changes: 102 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -360,9 +360,110 @@ Hint: Some lines were ellipsized, use -l to show in full.

    Quand ça ne veut pas, ça ne veut pas

    # Conclusion
    # Zed + Zed => Failed

    I cannot deploy OpenStack Zed, latest stable release, using DevStack (using Zed branch) on Rocky Linux 9.1.

    Problem is not related to Rocky Linux 9.1 as it was quickly patched from master but from Zed and DevStack Zed who didn't seem really aligned.

    # Next try, latest OpenStack on Rocky Linux 9.1

    I recreated a new VM, and redo operations but using latest devstack (master)

    ```
    =========================
    DevStack Component Timing
    (times are in seconds)
    =========================
    wait_for_service 9
    async_wait 136
    osc 147
    test_with_retry 4
    yum_install 173
    dbsync 11
    pip_install 138
    run_process 19
    git_timed 177
    -------------------------
    Unaccounted time 147
    =========================
    Total runtime 961
    =================
    Async summary
    =================
    Time spent in the background minus waits: 355 sec
    Elapsed time: 961 sec
    Time if we did everything serially: 1316 sec
    Speedup: 1.36941
    Post-stack database query stats:
    +------------+-----------+-------+
    | db | op | count |
    +------------+-----------+-------+
    | keystone | SELECT | 41625 |
    | keystone | INSERT | 98 |
    | glance | SELECT | 999 |
    | glance | CREATE | 65 |
    | glance | INSERT | 256 |
    | glance | SHOW | 8 |
    | glance | UPDATE | 17 |
    | glance | ALTER | 9 |
    | glance | DROP | 1 |
    | neutron | SELECT | 4790 |
    | neutron | CREATE | 319 |
    | neutron | SHOW | 39 |
    | cinder | SELECT | 165 |
    | cinder | SHOW | 1 |
    | cinder | CREATE | 74 |
    | cinder | SET | 1 |
    | cinder | ALTER | 18 |
    | neutron | INSERT | 1128 |
    | neutron | UPDATE | 232 |
    | neutron | ALTER | 153 |
    | neutron | DROP | 53 |
    | neutron | DELETE | 25 |
    | nova_cell1 | SELECT | 145 |
    | nova_cell1 | SHOW | 35 |
    | nova_cell1 | CREATE | 157 |
    | nova_cell0 | SELECT | 132 |
    | nova_cell0 | SHOW | 35 |
    | nova_cell0 | CREATE | 157 |
    | nova_cell0 | ALTER | 2 |
    | nova_cell1 | ALTER | 2 |
    | placement | SELECT | 38 |
    | placement | INSERT | 54 |
    | placement | SET | 1 |
    | nova_api | SELECT | 114 |
    | nova_cell0 | INSERT | 4 |
    | placement | UPDATE | 3 |
    | nova_cell1 | INSERT | 4 |
    | nova_cell1 | UPDATE | 19 |
    | cinder | INSERT | 5 |
    | nova_cell0 | UPDATE | 19 |
    | cinder | UPDATE | 14 |
    | nova_api | INSERT | 20 |
    | nova_api | SAVEPOINT | 10 |
    | nova_api | RELEASE | 10 |
    | cinder | DELETE | 1 |
    | keystone | DELETE | 5 |
    +------------+-----------+-------+
    This is your host IP address: 10.0.0.42
    This is your host IPv6 address: fe80::a00:27ff:fe45:dc5f/64
    Horizon is now available at http://10.0.0.42/dashboard
    Keystone is serving at http://10.0.0.42/identity/
    The default users are: admin and demo
    The password: nomoresecret
    Services are running under systemd unit files.
    For more information see:
    https://docs.openstack.org/devstack/latest/systemd.html
    DevStack Version: 2023.1
    Change: ab8e51eb49068a8c5004007c18fdfb9b1fcc0954 Merge "Disable memory_tracker and file_tracker in unstask.sh properly" 2023-02-28 06:13:08 +0000
    OS Version: Rocky 9.1
    ```
  13. hgomez revised this gist Mar 10, 2023. 1 changed file with 19 additions and 1 deletion.
    20 changes: 19 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -347,4 +347,22 @@ There is still folders /var/run/ovn and /var/run/openswitch, let's do more house
    sudo rm -rf /var/run/openswitch
    sudo rm /var/run/openvswitch.useropts


    ./stack.sh

    ```
    Mar 10 14:22:11 localhost.localdomain neutron-ovn-metadata-agent[381109]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=381109) lock /usr/local/lib/pyt…ils.py:333}}
    Mar 10 14:22:11 localhost.localdomain systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
    Mar 10 14:22:11 localhost.localdomain systemd[1]: [email protected]: Failed with result 'exit-code'.
    Mar 10 14:22:11 localhost.localdomain systemd[1]: [email protected]: Unit process 381109 (neutron-ovn-met) remains running after unit stopped.
    Mar 10 14:22:11 localhost.localdomain systemd[1]: [email protected]: Consumed 1.035s CPU time.
    Hint: Some lines were ellipsized, use -l to show in full.
    ```

    Quand ça ne veut pas, ça ne veut pas

    # Conclusion

    I cannot deploy OpenStack Zed, latest stable release, using DevStack (using Zed branch) on Rocky Linux 9.1.

    Problem is not related to Rocky Linux 9.1 as it was quickly patched from master but from Zed and DevStack Zed who didn't seem really aligned.

  14. hgomez revised this gist Mar 10, 2023. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -338,6 +338,10 @@ Mar 10 14:01:00 localhost.localdomain systemd[1]: devstack@q-ovn-metadata-agent.

    There is still folders /var/run/ovn and /var/run/openswitch, let's do more housekeeping

    ./unstack.sh
    ./clean.sh
    sudo dnf remove -y ovn-common ovn-controller-vtep ovn-host ovn-central

    sudo dnf remove openstack-network-scripts
    sudo rm -rf /var/run/ovn
    sudo rm -rf /var/run/openswitch
  15. hgomez revised this gist Mar 10, 2023. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -334,4 +334,13 @@ Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274711]: DEBUG
    Mar 10 14:01:00 localhost.localdomain systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
    Mar 10 14:01:00 localhost.localdomain systemd[1]: [email protected]: Failed with result 'exit-code'.
    Mar 10 14:01:00 localhost.localdomain systemd[1]: [email protected]: Consumed 1.050s CPU time.
    ```
    ```

    There is still folders /var/run/ovn and /var/run/openswitch, let's do more housekeeping

    sudo dnf remove openstack-network-scripts
    sudo rm -rf /var/run/ovn
    sudo rm -rf /var/run/openswitch
    sudo rm /var/run/openvswitch.useropts


  16. hgomez revised this gist Mar 10, 2023. 1 changed file with 15 additions and 0 deletions.
    15 changes: 15 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -319,4 +319,19 @@ Same problem, let's grab ovn_agent at that time
    else
    _start_process "$OVN_CONTROLLER_VTEP_SERVICE"
    fi
    ```

    Sadly, it didn't fix problem

    ```
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274711]: INFO eventlet.wsgi.server [-] (274711) wsgi exited, is_accepting=True
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274711]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "singleton_lock" {{(pid=274711) lock /usr/local/lib/pyt…ils.py:312}}
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274710]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "singleton_lock" {{(pid=274710) lock /usr/local/lib/pyt…ils.py:312}}
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274710]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=274710) lock /usr/local/lib/pyth…ils.py:315}}
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274711]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=274711) lock /usr/local/lib/pyth…ils.py:315}}
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274710]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=274710) lock /usr/local/lib/pyt…ils.py:333}}
    Mar 10 14:01:00 localhost.localdomain neutron-ovn-metadata-agent[274711]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=274711) lock /usr/local/lib/pyt…ils.py:333}}
    Mar 10 14:01:00 localhost.localdomain systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
    Mar 10 14:01:00 localhost.localdomain systemd[1]: [email protected]: Failed with result 'exit-code'.
    Mar 10 14:01:00 localhost.localdomain systemd[1]: [email protected]: Consumed 1.050s CPU time.
    ```
  17. hgomez revised this gist Mar 10, 2023. 1 changed file with 140 additions and 2 deletions.
    142 changes: 140 additions & 2 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -165,7 +165,10 @@ ln: failed to create symbolic link '/var/run/ovn/openvswitch': File exists
    +lib/neutron_plugins/ovn_agent:install_ovn:1 exit_trap
    ```

    Same symlink issue, in master they removed [this symlink](https://opendev.org/openstack/devstack/commit/71c99655479174750bcedfe458328328a1596766), let's do the same

    Same symlink issue...

    In master they removed [this symlink](https://opendev.org/openstack/devstack/commit/71c99655479174750bcedfe458328328a1596766) to fix [bug](https://bugs.launchpad.net/devstack/+bug/1980421), let's remove this symlink too (may be not be enough)

    ```
    --- lib/neutron_plugins/ovn_agent.orig 2023-03-10 13:40:48.218521413 +0100
    @@ -181,4 +184,139 @@ ln: failed to create symbolic link '/var/run/ovn/openvswitch': File exists
    # If OVS is already installed, remove it, because we're about to
    ```

    There is a lot of changes in lib/neutron_plugins/ovn_agent since [commit](https://opendev.org/openstack/devstack/commit/71c99655479174750bcedfe458328328a1596766), removing symlink may not be enough
    Same problem, let's grab ovn_agent at that time

    ```
    --- lib/neutron_plugins/ovn_agent.orig 2023-03-10 13:40:48.218521413 +0100
    +++ lib/neutron_plugins/ovn_agent 2023-03-10 13:51:56.539450583 +0100
    @@ -244,11 +244,12 @@
    local cmd="$2"
    local stop_cmd="$3"
    local group=$4
    - local user=${5:-$STACK_USER}
    + local user=$5
    + local rundir=${6:-$OVS_RUNDIR}
    local systemd_service="devstack@$service.service"
    local unit_file="$SYSTEMD_DIR/$systemd_service"
    - local environment="OVN_RUNDIR=$OVS_RUNDIR OVN_DBDIR=$OVN_DATADIR OVN_LOGDIR=$LOGDIR OVS_RUNDIR=$OVS_RUNDIR OVS_DBDIR=$OVS_DATADIR OVS_LOGDIR=$LOGDIR"
    + local environment="OVN_RUNDIR=$OVN_RUNDIR OVN_DBDIR=$OVN_DATADIR OVN_LOGDIR=$LOGDIR OVS_RUNDIR=$OVS_RUNDIR OVS_DBDIR=$OVS_DATADIR OVS_LOGDIR=$LOGDIR"
    echo "Starting $service executed command": $cmd
    @@ -264,14 +265,14 @@
    _start_process $systemd_service
    - local testcmd="test -e $OVS_RUNDIR/$service.pid"
    + local testcmd="test -e $rundir/$service.pid"
    test_with_retry "$testcmd" "$service did not start" $SERVICE_TIMEOUT 1
    local service_ctl_file
    - service_ctl_file=$(ls $OVS_RUNDIR | grep $service | grep ctl)
    + service_ctl_file=$(ls $rundir | grep $service | grep ctl)
    if [ -z "$service_ctl_file" ]; then
    die $LINENO "ctl file for service $service is not present."
    fi
    - sudo ovs-appctl -t $OVS_RUNDIR/$service_ctl_file vlog/set console:off syslog:info file:info
    + sudo ovs-appctl -t $rundir/$service_ctl_file vlog/set console:off syslog:info file:info
    }
    function clone_repository {
    @@ -370,10 +371,6 @@
    sudo mkdir -p $OVS_RUNDIR
    sudo chown $(whoami) $OVS_RUNDIR
    - # NOTE(lucasagomes): To keep things simpler, let's reuse the same
    - # RUNDIR for both OVS and OVN. This way we avoid having to specify the
    - # --db option in the ovn-{n,s}bctl commands while playing with DevStack
    - sudo ln -s $OVS_RUNDIR $OVN_RUNDIR
    if [[ "$OVN_BUILD_FROM_SOURCE" == "True" ]]; then
    # If OVS is already installed, remove it, because we're about to
    @@ -590,7 +587,6 @@
    rm -f $OVS_DATADIR/.*.db.~lock~
    sudo rm -f $OVN_DATADIR/*.db
    sudo rm -f $OVN_DATADIR/.*.db.~lock~
    - sudo rm -f $OVN_RUNDIR/*.sock
    }
    function _start_ovs {
    @@ -617,12 +613,12 @@
    dbcmd+=" --remote=db:hardware_vtep,Global,managers $OVS_DATADIR/vtep.db"
    fi
    dbcmd+=" $OVS_DATADIR/conf.db"
    - _run_process ovsdb-server "$dbcmd" "" "$STACK_GROUP" "root"
    + _run_process ovsdb-server "$dbcmd" "" "$STACK_GROUP" "root" "$OVS_RUNDIR"
    # Note: ovn-controller will create and configure br-int once it is started.
    # So, no need to create it now because nothing depends on that bridge here.
    local ovscmd="$OVS_SBINDIR/ovs-vswitchd --log-file --pidfile --detach"
    - _run_process ovs-vswitchd "$ovscmd" "" "$STACK_GROUP" "root"
    + _run_process ovs-vswitchd "$ovscmd" "" "$STACK_GROUP" "root" "$OVS_RUNDIR"
    else
    _start_process "$OVSDB_SERVER_SERVICE"
    _start_process "$OVS_VSWITCHD_SERVICE"
    @@ -661,7 +657,7 @@
    enable_service ovs-vtep
    local vtepcmd="$OVS_SCRIPTDIR/ovs-vtep --log-file --pidfile --detach br-v"
    - _run_process ovs-vtep "$vtepcmd" "" "$STACK_GROUP" "root"
    + _run_process ovs-vtep "$vtepcmd" "" "$STACK_GROUP" "root" "$OVS_RUNDIR"
    vtep-ctl set-manager tcp:$HOST_IP:6640
    fi
    @@ -705,26 +701,26 @@
    local cmd="/bin/bash $SCRIPTDIR/ovn-ctl --no-monitor start_northd"
    local stop_cmd="/bin/bash $SCRIPTDIR/ovn-ctl stop_northd"
    - _run_process ovn-northd "$cmd" "$stop_cmd" "$STACK_GROUP" "root"
    + _run_process ovn-northd "$cmd" "$stop_cmd" "$STACK_GROUP" "root" "$OVN_RUNDIR"
    else
    _start_process "$OVN_NORTHD_SERVICE"
    fi
    # Wait for the service to be ready
    # Check for socket and db files for both OVN NB and SB
    - wait_for_sock_file $OVS_RUNDIR/ovnnb_db.sock
    - wait_for_sock_file $OVS_RUNDIR/ovnsb_db.sock
    + wait_for_sock_file $OVN_RUNDIR/ovnnb_db.sock
    + wait_for_sock_file $OVN_RUNDIR/ovnsb_db.sock
    wait_for_db_file $OVN_DATADIR/ovnnb_db.db
    wait_for_db_file $OVN_DATADIR/ovnsb_db.db
    if is_service_enabled tls-proxy; then
    - sudo ovn-nbctl --db=unix:$OVS_RUNDIR/ovnnb_db.sock set-ssl $INT_CA_DIR/private/$DEVSTACK_CERT_NAME.key $INT_CA_DIR/$DEVSTACK_CERT_NAME.crt $INT_CA_DIR/ca-chain.pem
    - sudo ovn-sbctl --db=unix:$OVS_RUNDIR/ovnsb_db.sock set-ssl $INT_CA_DIR/private/$DEVSTACK_CERT_NAME.key $INT_CA_DIR/$DEVSTACK_CERT_NAME.crt $INT_CA_DIR/ca-chain.pem
    + sudo ovn-nbctl --db=unix:$OVN_RUNDIR/ovnnb_db.sock set-ssl $INT_CA_DIR/private/$DEVSTACK_CERT_NAME.key $INT_CA_DIR/$DEVSTACK_CERT_NAME.crt $INT_CA_DIR/ca-chain.pem
    + sudo ovn-sbctl --db=unix:$OVN_RUNDIR/ovnsb_db.sock set-ssl $INT_CA_DIR/private/$DEVSTACK_CERT_NAME.key $INT_CA_DIR/$DEVSTACK_CERT_NAME.crt $INT_CA_DIR/ca-chain.pem
    fi
    - sudo ovn-nbctl --db=unix:$OVS_RUNDIR/ovnnb_db.sock set-connection p${OVN_PROTO}:6641:$SERVICE_LISTEN_ADDRESS -- set connection . inactivity_probe=60000
    - sudo ovn-sbctl --db=unix:$OVS_RUNDIR/ovnsb_db.sock set-connection p${OVN_PROTO}:6642:$SERVICE_LISTEN_ADDRESS -- set connection . inactivity_probe=60000
    - sudo ovs-appctl -t $OVS_RUNDIR/ovnnb_db.ctl vlog/set console:off syslog:$OVN_DBS_LOG_LEVEL file:$OVN_DBS_LOG_LEVEL
    - sudo ovs-appctl -t $OVS_RUNDIR/ovnsb_db.ctl vlog/set console:off syslog:$OVN_DBS_LOG_LEVEL file:$OVN_DBS_LOG_LEVEL
    + sudo ovn-nbctl --db=unix:$OVN_RUNDIR/ovnnb_db.sock set-connection p${OVN_PROTO}:6641:$SERVICE_LISTEN_ADDRESS -- set connection . inactivity_probe=60000
    + sudo ovn-sbctl --db=unix:$OVN_RUNDIR/ovnsb_db.sock set-connection p${OVN_PROTO}:6642:$SERVICE_LISTEN_ADDRESS -- set connection . inactivity_probe=60000
    + sudo ovs-appctl -t $OVN_RUNDIR/ovnnb_db.ctl vlog/set console:off syslog:$OVN_DBS_LOG_LEVEL file:$OVN_DBS_LOG_LEVEL
    + sudo ovs-appctl -t $OVN_RUNDIR/ovnsb_db.ctl vlog/set console:off syslog:$OVN_DBS_LOG_LEVEL file:$OVN_DBS_LOG_LEVEL
    fi
    if is_service_enabled ovn-controller ; then
    @@ -732,7 +728,7 @@
    local cmd="/bin/bash $SCRIPTDIR/ovn-ctl --no-monitor start_controller"
    local stop_cmd="/bin/bash $SCRIPTDIR/ovn-ctl stop_controller"
    - _run_process ovn-controller "$cmd" "$stop_cmd" "$STACK_GROUP" "root"
    + _run_process ovn-controller "$cmd" "$stop_cmd" "$STACK_GROUP" "root" "$OVN_RUNDIR"
    else
    _start_process "$OVN_CONTROLLER_SERVICE"
    fi
    @@ -741,7 +737,7 @@
    if is_service_enabled ovn-controller-vtep ; then
    if [[ "$OVN_BUILD_FROM_SOURCE" == "True" ]]; then
    local cmd="$OVS_BINDIR/ovn-controller-vtep --log-file --pidfile --detach --ovnsb-db=$OVN_SB_REMOTE"
    - _run_process ovn-controller-vtep "$cmd" "" "$STACK_GROUP" "root"
    + _run_process ovn-controller-vtep "$cmd" "" "$STACK_GROUP" "root" "$OVN_RUNDIR"
    else
    _start_process "$OVN_CONTROLLER_VTEP_SERVICE"
    fi
    ```
  18. hgomez revised this gist Mar 10, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -181,4 +181,4 @@ ln: failed to create symbolic link '/var/run/ovn/openvswitch': File exists
    # If OVS is already installed, remove it, because we're about to
    ```


    There is a lot of changes in lib/neutron_plugins/ovn_agent since [commit](https://opendev.org/openstack/devstack/commit/71c99655479174750bcedfe458328328a1596766), removing symlink may not be enough
  19. hgomez revised this gist Mar 10, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -165,7 +165,7 @@ ln: failed to create symbolic link '/var/run/ovn/openvswitch': File exists
    +lib/neutron_plugins/ovn_agent:install_ovn:1 exit_trap
    ```

    Same symlink issue, in master they removed this symlink, let's do the same
    Same symlink issue, in master they removed [this symlink](https://opendev.org/openstack/devstack/commit/71c99655479174750bcedfe458328328a1596766), let's do the same

    ```
    --- lib/neutron_plugins/ovn_agent.orig 2023-03-10 13:40:48.218521413 +0100
  20. hgomez revised this gist Mar 10, 2023. 1 changed file with 17 additions and 1 deletion.
    18 changes: 17 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -165,4 +165,20 @@ ln: failed to create symbolic link '/var/run/ovn/openvswitch': File exists
    +lib/neutron_plugins/ovn_agent:install_ovn:1 exit_trap
    ```

    Same symlink issue
    Same symlink issue, in master they removed this symlink, let's do the same

    ```
    --- lib/neutron_plugins/ovn_agent.orig 2023-03-10 13:40:48.218521413 +0100
    +++ lib/neutron_plugins/ovn_agent 2023-03-10 13:40:58.115464335 +0100
    @@ -373,7 +373,7 @@
    # NOTE(lucasagomes): To keep things simpler, let's reuse the same
    # RUNDIR for both OVS and OVN. This way we avoid having to specify the
    # --db option in the ovn-{n,s}bctl commands while playing with DevStack
    - sudo ln -s $OVS_RUNDIR $OVN_RUNDIR
    + # sudo ln -s $OVS_RUNDIR $OVN_RUNDIR
    if [[ "$OVN_BUILD_FROM_SOURCE" == "True" ]]; then
    # If OVS is already installed, remove it, because we're about to
    ```


  21. hgomez revised this gist Mar 10, 2023. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -157,3 +157,12 @@ Try reinstall

    ./stack.sh


    ```
    +lib/neutron_plugins/ovn_agent:install_ovn:372 sudo chown stack /var/run/openvswitch
    +lib/neutron_plugins/ovn_agent:install_ovn:376 sudo ln -s /var/run/openvswitch /var/run/ovn
    ln: failed to create symbolic link '/var/run/ovn/openvswitch': File exists
    +lib/neutron_plugins/ovn_agent:install_ovn:1 exit_trap
    ```

    Same symlink issue
  22. hgomez revised this gist Mar 10, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -147,9 +147,9 @@ Some hope here ? => https://stackoverflow.com/questions/68001501/error-opt-stack

    # Dirty way ?

    Uninstall all ovn packages after ./unstak.sh and ./clean.sh
    Uninstall all ovn packages after ./unstack.sh and ./clean.sh

    ./unstak.sh
    ./unstack.sh
    ./clean.sh
    sudo dnf remove -y ovn-common ovn-controller-vtep ovn-host ovn-central

  23. hgomez revised this gist Mar 10, 2023. 1 changed file with 14 additions and 0 deletions.
    14 changes: 14 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -143,3 +143,17 @@ For now stucked at


    Some hope here ? => https://stackoverflow.com/questions/68001501/error-opt-stack-devstack-lib-neutron-plugins-ovn-agent174-socket


    # Dirty way ?

    Uninstall all ovn packages after ./unstak.sh and ./clean.sh

    ./unstak.sh
    ./clean.sh
    sudo dnf remove -y ovn-common ovn-controller-vtep ovn-host ovn-central

    Try reinstall

    ./stack.sh

  24. hgomez revised this gist Mar 10, 2023. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -140,3 +140,6 @@ For now stucked at
    /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191:die
    [ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191 Socket /var/run/openvswitch/ovnnb_db.sock not found
    ```


    Some hope here ? => https://stackoverflow.com/questions/68001501/error-opt-stack-devstack-lib-neutron-plugins-ovn-agent174-socket
  25. hgomez revised this gist Mar 10, 2023. 1 changed file with 14 additions and 1 deletion.
    15 changes: 14 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -126,4 +126,17 @@ Logged as stack user and under /opt/stack/devstack directory
    cd /opt/stack/devstack
    ./stack.sh



    For now stucked at

    ```
    +lib/neutron_plugins/ovn_agent:wait_for_sock_file:191 die 191 'Socket /var/run/openvswitch/ovnnb_db.sock not found'
    +functions-common:die:264 local exitcode=0
    +functions-common:die:265 set +o xtrace
    [Call Trace]
    ./stack.sh:1273:start_ovn_services
    /opt/stack/devstack/lib/neutron-legacy:521:start_ovn
    /opt/stack/devstack/lib/neutron_plugins/ovn_agent:715:wait_for_sock_file
    /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191:die
    [ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:191 Socket /var/run/openvswitch/ovnnb_db.sock not found
    ```
  26. hgomez revised this gist Mar 10, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -121,9 +121,9 @@ Back port change found in master

    # Install OpenStack

    Logged as stack user and under /opt/stack directory
    Logged as stack user and under /opt/stack/devstack directory

    cd /opt/stack
    cd /opt/stack/devstack
    ./stack.sh


  27. hgomez revised this gist Mar 10, 2023. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -38,7 +38,15 @@ Make it permanent by editing /etc/sysconfig/selinux
    Reboot

    sudo reboot


    Verify after reboot

    sestatus

    You should see

    SELinux status: disabled


    # Connect to VM as stack user

  28. hgomez revised this gist Mar 10, 2023. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -35,6 +35,10 @@ Make it permanent by editing /etc/sysconfig/selinux

    SELINUX=disabled

    Reboot

    sudo reboot


    # Connect to VM as stack user

  29. hgomez revised this gist Mar 10, 2023. 1 changed file with 16 additions and 0 deletions.
    16 changes: 16 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -20,6 +20,22 @@ Make sure that root account allow remote login
    echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
    sudo -u stack -i

    # Prepare Environment

    Be sure to be in en_US and UTF-8 (ie: /etc/environment)

    LANG=en_US.utf-8
    LC_ALL=en_US.utf-8

    Disable selinux

    sudo setenforce 0

    Make it permanent by editing /etc/sysconfig/selinux

    SELINUX=disabled


    # Connect to VM as stack user

    You should be connected as non root user, let's use stack account
  30. hgomez revised this gist Mar 10, 2023. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions howto-openstack-zed-rocky-linux-9.1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -91,5 +91,11 @@ Back port change found in master
    [ "$os_VENDOR" = "OracleServer" ] || [ "$os_VENDOR" = "Virtuozzo" ]
    ```

    # Install OpenStack

    Logged as stack user and under /opt/stack directory

    cd /opt/stack
    ./stack.sh