Skip to content

Instantly share code, notes, and snippets.

@DeckerSU

DeckerSU/openssh-latest-build-install.md

Forked from jtmoon79/openssh-latest-build-install.md
Created July 1, 2024 17:15
Show Gist options
  • Save DeckerSU/2d1f6a8f1c0dc8f20583dee4ab3a69b1 to your computer and use it in GitHub Desktop.
Save DeckerSU/2d1f6a8f1c0dc8f20583dee4ab3a69b1 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @jtmoon79 jtmoon79 revised this gist Dec 10, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -320,7 +320,7 @@ Optionally, the `/opt/openssh-latest/etc/sshd_config` can refer to the key files

    The file `/opt/openssh-latest/etc/sshd_config` should have lines

    $ grep -Fe 'HostKey' /opt/openssh-latest/etc/sshd_config
    $ grep -Fe 'HostKey' /opt/openssh-${VER}/etc/sshd_config
    HostKey /etc/ssh/ssh_host_rsa_key
    HostKey /etc/ssh/ssh_host_ecdsa_key
    HostKey /etc/ssh/ssh_host_ed25519_key
  2. @jtmoon79 jtmoon79 revised this gist Dec 10, 2023. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -79,7 +79,7 @@ I _suspect_ you also need these

    These guesses have not been thoroughly tested.

    Other helpful tools for the build and install process:
    Other helpful tools for debugging server issues are:

    apt install netcat lsof wget diffutils

    @@ -320,6 +320,7 @@ Optionally, the `/opt/openssh-latest/etc/sshd_config` can refer to the key files

    The file `/opt/openssh-latest/etc/sshd_config` should have lines

    $ grep -Fe 'HostKey' /opt/openssh-latest/etc/sshd_config
    HostKey /etc/ssh/ssh_host_rsa_key
    HostKey /etc/ssh/ssh_host_ecdsa_key
    HostKey /etc/ssh/ssh_host_ed25519_key
  3. @jtmoon79 jtmoon79 revised this gist Dec 10, 2023. 1 changed file with 4 additions and 3 deletions.
    7 changes: 4 additions & 3 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -91,7 +91,8 @@ Other helpful tools for the build and install process:
    2. Download the latest archive.<br/>
    This will example will use [openssh-9.0p1.tar.gz](https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/openssh-9.0p1.tar.gz).

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz
    VER=9.0p1
    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${VER}.tar.gz

    ### Verify the download using `gpg`

    @@ -106,8 +107,8 @@ Other helpful tools for the build and install process:

    **_Do this each update._**

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz.asc
    gpg --verbose --verify openssh-9.0p1.tar.gz.asc
    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${VER}.tar.gz.asc
    gpg --verbose --verify openssh-${VER}.tar.gz.asc

    ## Build

  4. @jtmoon79 jtmoon79 revised this gist Dec 2, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -342,11 +342,11 @@ The directory `/opt` should look like

    ## Restart Service

    Watch the service logs
    First, watch the service logs

    journalctl -f -x -u ssh-latest

    Restart
    Then restart

    systemctl restart ssh-latest

  5. @jtmoon79 jtmoon79 revised this gist Nov 12, 2023. 1 changed file with 43 additions and 4 deletions.
    47 changes: 43 additions & 4 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -37,6 +37,8 @@ This is for a Debian-derived Linux System using systemd. This document will use

    OpenSSH will be built on the system that will also run the ssh service.

    Commands are presumed to run as user `root`.

    ## Prepare `/opt`

    **_Do this once._**
    @@ -145,6 +147,7 @@ This should change the statement to

    **_Do this once._**

    rm -vf /opt/openssh-latest /etc/ssh-latest
    ln -fvs /opt/openssh-${VER} /opt/openssh-latest
    ln -fvs /opt/openssh-latest /etc/ssh-latest

    @@ -178,7 +181,7 @@ Be sure not to use the generated systemd files typically found at `/etc/systemd`

    Find all `*ssh*` files

    find / -xdev -name '*ssh*' | sort
    find / -xdev -name '*ssh*' 2>/dev/null | sort

    Copy the systemd service files. In my case on Debian 11

    @@ -188,6 +191,8 @@ Copy the systemd service files. In my case on Debian 11
    cp -av ssh.socket ssh-latest.socket
    cp -av rescue-ssh.target rescue-ssh-latest.target

    MMV: the path to systemd files tends to change often per distribution and and per release.

    In the newly copied `ssh-latest*` systemd files, manually update references from `ssh` to `ssh-latest`.

    In my case on Debian 11, the changes looked like:
    @@ -279,28 +284,62 @@ Repeat section [_Download_](#download), and [_Build_](#build).

    ## Copy the Previous `-latest` to the New `-latest`

    Set `PREV` to the appropriate value. In my case it was `8.6p1`.

    cd /opt/openssh-${VER}/etc/
    # set aside default config files
    mkdir -vp _original
    mv -v ssh_host_* sshd_config ssh_config _original/
    cp -av /opt/openssh-latest/etc/ssh_host_* /opt/openssh-latest/etc/{sshd_config,ssh_config} .
    PREV=8.6p1
    cp -av /opt/openssh-${PREV}/etc/ssh_host_* /opt/openssh-${PREV}/etc/{sshd_config,ssh_config} .

    The files in `/opt/openssh-latest/etc/` should be similar to:

    moduli
    _original/
    ssh_config
    sshd_config
    ssh_host_dsa_key
    ssh_host_dsa_key.pub
    ssh_host_ecdsa_key
    ssh_host_ecdsa_key.pub
    ssh_host_ed25519_key
    ssh_host_ed25519_key.pub
    ssh_host_rsa_key
    ssh_host_rsa_key.pub

    ### Use default key files (optional)

    Optionally, the `/opt/openssh-latest/etc/sshd_config` can refer to the key files at `/etc/ssh`. Then there is no need to copy the `/opt/openssh-latest/etc/ssh_host_*` files.
    Optionally, the `/opt/openssh-latest/etc/sshd_config` can refer to the key files at `/etc/ssh`. Then there is no need to copy the `/opt/openssh-latest/etc/ssh_host_*` files. In that case, the prior file listing of `/opt/openssh-latest/etc/` should be similar to:

    The file `/opt/openssh-latest/etc/sshd_config` would have lines
    moduli
    _original/
    ssh_config
    sshd_config

    The file `/opt/openssh-latest/etc/sshd_config` should have lines

    HostKey /etc/ssh/ssh_host_rsa_key
    HostKey /etc/ssh/ssh_host_ecdsa_key
    HostKey /etc/ssh/ssh_host_ed25519_key

    Review the copied `sshd_config` with the original default `sshd_config`.

    cd /opt/openssh-${VER}/etc/
    diff -y -W $COLUMNS sshd_config _original/sshd_config | less -SR

    ## Update Symlinks for `-latest`

    systemctl stop ssh-latest
    cd /opt
    rm -v openssh-latest && ln -fvs openssh-${VER} openssh-latest

    The directory `/opt` should look like

    openssh-8.6p1/
    openssh-9.0p1/
    openssh-latest -> openssh-9.0p1

    ## Restart Service

    Watch the service logs
  6. @jtmoon79 jtmoon79 revised this gist Nov 12, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -118,7 +118,7 @@ Preview the next subsection before running these commands.
    cd openssh-${VER}
    ./configure --prefix=/opt/openssh-${VER}
    make
    sudo make install
    make install

    If `./configure` is missing then the non-portable version was downloaded.

  7. @jtmoon79 jtmoon79 revised this gist Nov 12, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -118,7 +118,7 @@ Preview the next subsection before running these commands.
    cd openssh-${VER}
    ./configure --prefix=/opt/openssh-${VER}
    make
    make install
    sudo make install

    If `./configure` is missing then the non-portable version was downloaded.

  8. @jtmoon79 jtmoon79 revised this gist Oct 26, 2022. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -146,6 +146,7 @@ This should change the statement to
    **_Do this once._**

    ln -fvs /opt/openssh-${VER} /opt/openssh-latest
    ln -fvs /opt/openssh-latest /etc/ssh-latest

    ## Set aside built-in OpenSSH

  9. @jtmoon79 jtmoon79 revised this gist Oct 26, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,7 +5,7 @@ Build, install, and run the latest OpenSSH Server as a systemd service.
    ---

    - [openssh latest](#openssh-latest)
    - [About d](#about-d)
    - [About](#about)
    - [Prepare `/opt`](#prepare-opt)
    - [apt install](#apt-install)
    - [Download](#download)
    @@ -29,7 +29,7 @@ Build, install, and run the latest OpenSSH Server as a systemd service.

    ---

    ## About d
    ## About

    Running the latest OpenSSH Server is easy and [security-wise](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=OpenSSH). This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).

  10. @jtmoon79 jtmoon79 revised this gist Oct 26, 2022. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,15 +5,15 @@ Build, install, and run the latest OpenSSH Server as a systemd service.
    ---

    - [openssh latest](#openssh-latest)
    - [About](#about)
    - [About d](#about-d)
    - [Prepare `/opt`](#prepare-opt)
    - [apt install](#apt-install)
    - [Download](#download)
    - [Verify the download using `gpg`](#verify-the-download-using-gpg)
    - [Import the public signing key](#import-the-public-signing-key)
    - [Verify](#verify)
    - [Build](#build)
    - [Change the banner (optional)](#change-the-banner-optional)
    - [Change the banner (optional) (INCOMPLETE)](#change-the-banner-optional-incomplete)
    - [Create Symlink for `-latest`](#create-symlink-for--latest)
    - [Set aside built-in OpenSSH](#set-aside-built-in-openssh)
    - [Create systemd service `-latest`](#create-systemd-service--latest)
    @@ -29,7 +29,7 @@ Build, install, and run the latest OpenSSH Server as a systemd service.

    ---

    ## About
    ## About d

    Running the latest OpenSSH Server is easy and [security-wise](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=OpenSSH). This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).

  11. @jtmoon79 jtmoon79 revised this gist Oct 26, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -122,7 +122,7 @@ Preview the next subsection before running these commands.

    If `./configure` is missing then the non-portable version was downloaded.

    ### Change the banner (optional)
    ### Change the banner (optional) (INCOMPLETE)

    _This section has not yet successfully tested 😔. Come back later._

  12. @jtmoon79 jtmoon79 revised this gist Oct 18, 2022. 1 changed file with 10 additions and 10 deletions.
    20 changes: 10 additions & 10 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -39,13 +39,13 @@ OpenSSH will be built on the system that will also run the ssh service.

    ## Prepare `/opt`

    _Do this once._
    **_Do this once._**

    mkdir -vp /opt/

    ## apt install

    _Do this once._
    **_Do this once._**

    List the dpkg requirements for OpenSSH server

    @@ -83,7 +83,7 @@ Other helpful tools for the build and install process:

    ## Download

    _Do this each update._
    **_Do this each update._**

    1. [Pick a mirror](https://www.openssh.com/portable.html)
    2. Download the latest archive.<br/>
    @@ -95,21 +95,21 @@ _Do this each update._

    #### Import the public signing key

    _Do this once._
    **_Do this once._**

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc
    gpg --import RELEASE_KEY.asc

    #### Verify

    _Do this each update._
    **_Do this each update._**

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz.asc
    gpg --verbose --verify openssh-9.0p1.tar.gz.asc

    ## Build

    _Do this each update._
    **_Do this each update._**

    Preview the next subsection before running these commands.

    @@ -143,13 +143,13 @@ This should change the statement to

    ## Create Symlink for `-latest`

    _Do this once._
    **_Do this once._**

    ln -fvs /opt/openssh-${VER} /opt/openssh-latest

    ## Set aside built-in OpenSSH

    _Do this once._
    **_Do this once._**

    **Very Important!** _Keep the functioning built-in sshd service in-place or running at a different port_.

    @@ -168,7 +168,7 @@ Now there is a reliable fallback SSH Service.

    ## Create systemd service `-latest`

    _Do this once._
    **_Do this once._**

    Copy the systemd template service files for the ssh service.

    @@ -270,7 +270,7 @@ Check the ports in-use

    # openssh updates

    _Do this each update._
    **_Do this each update._**

    After a new version of OpenSSH Server is released, it's relatively easy to update things.

  13. @jtmoon79 jtmoon79 revised this gist Oct 11, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -31,7 +31,7 @@ Build, install, and run the latest OpenSSH Server as a systemd service.

    ## About

    Running the latest OpenSSH Server is easy and [security-wise](https://www.openssh.com/releasenotes.html). This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).
    Running the latest OpenSSH Server is easy and [security-wise](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=OpenSSH). This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).

    This is for a Debian-derived Linux System using systemd. This document will use version _9.0p1_.

  14. @jtmoon79 jtmoon79 revised this gist Aug 21, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -86,8 +86,8 @@ Other helpful tools for the build and install process:
    _Do this each update._

    1. [Pick a mirror](https://www.openssh.com/portable.html)
    2. Download the latest archive.
    This will example will use [https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/openssh-9.0p1.tar.gz](https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/openssh-9.0p1.tar.gz).
    2. Download the latest archive.<br/>
    This will example will use [openssh-9.0p1.tar.gz](https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/openssh-9.0p1.tar.gz).

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz

  15. @jtmoon79 jtmoon79 revised this gist Aug 21, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -33,7 +33,7 @@ Build, install, and run the latest OpenSSH Server as a systemd service.

    Running the latest OpenSSH Server is easy and [security-wise](https://www.openssh.com/releasenotes.html). This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).

    This is for a Debian-derived Linux System using systemd.
    This is for a Debian-derived Linux System using systemd. This document will use version _9.0p1_.

    OpenSSH will be built on the system that will also run the ssh service.

  16. @jtmoon79 jtmoon79 revised this gist Aug 3, 2022. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -212,6 +212,7 @@ In my case on Debian 11, the changes looked like:
    Before=ssh-latest.service | Before=ssh.service
    Conflicts=ssh-latest.service | Conflicts=ssh.service
    ConditionPathExists=!/opt/openssh-latest/etc/sshd_not_to_be_r | ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
    ListenStream=2222 | ListenStream=22

    $ diff -y --suppress-common-lines rescue-ssh.target rescue-ssh-latest.target
    Requires=network-online.target ssh.service | Requires=network-online.target ssh-latest.service
  17. @jtmoon79 jtmoon79 revised this gist Jul 20, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -31,11 +31,11 @@ Build, install, and run the latest OpenSSH Server as a systemd service.

    ## About

    Running the latest OpenSSH Server is easy and [security-wise](https://www.openssh.com/releasenotes.html).
    Running the latest OpenSSH Server is easy and [security-wise](https://www.openssh.com/releasenotes.html). This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).

    This is for a Debian-derived Linux System using systemd.

    OpenSSH is built on the system that will also run the ssh service.
    OpenSSH will be built on the system that will also run the ssh service.

    ## Prepare `/opt`

  18. @jtmoon79 jtmoon79 created this gist Jul 20, 2022.
    317 changes: 317 additions & 0 deletions openssh-latest-build-install.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,317 @@
    # openssh latest

    Build, install, and run the latest OpenSSH Server as a systemd service.

    ---

    - [openssh latest](#openssh-latest)
    - [About](#about)
    - [Prepare `/opt`](#prepare-opt)
    - [apt install](#apt-install)
    - [Download](#download)
    - [Verify the download using `gpg`](#verify-the-download-using-gpg)
    - [Import the public signing key](#import-the-public-signing-key)
    - [Verify](#verify)
    - [Build](#build)
    - [Change the banner (optional)](#change-the-banner-optional)
    - [Create Symlink for `-latest`](#create-symlink-for--latest)
    - [Set aside built-in OpenSSH](#set-aside-built-in-openssh)
    - [Create systemd service `-latest`](#create-systemd-service--latest)
    - [Copy the default environment file](#copy-the-default-environment-file)
    - [Enable and restart](#enable-and-restart)
    - [Verify new ssh service and old ssh service](#verify-new-ssh-service-and-old-ssh-service)
    - [openssh updates](#openssh-updates)
    - [Copy the Previous `-latest` to the New `-latest`](#copy-the-previous--latest-to-the-new--latest)
    - [Use default key files (optional)](#use-default-key-files-optional)
    - [Update Symlinks for `-latest`](#update-symlinks-for--latest)
    - [Restart Service](#restart-service)
    - [Test](#test)

    ---

    ## About

    Running the latest OpenSSH Server is easy and [security-wise](https://www.openssh.com/releasenotes.html).

    This is for a Debian-derived Linux System using systemd.

    OpenSSH is built on the system that will also run the ssh service.

    ## Prepare `/opt`

    _Do this once._

    mkdir -vp /opt/

    ## apt install

    _Do this once._

    List the dpkg requirements for OpenSSH server

    apt show openssh-server

    Most likely, the packages listed under `Depends:` will be needed to build OpenSSH.

    My typical Debian system addtionally needed these

    apt install \
    libssl-dev \
    gcc g++ gdb cpp \
    make cmake \
    libtool \
    libc6 \
    autoconf automake pkg-config \
    build-essential \
    gettext \

    (I am not sure if _all_ of these packages are needed, but it did the trick)

    I _suspect_ you also need these

    apt install \
    libzstd1 zlib1g \
    libssh-4 libssh-dev libssl3 \
    libc6-dev libc6 \
    libcrypt-dev

    These guesses have not been thoroughly tested.

    Other helpful tools for the build and install process:

    apt install netcat lsof wget diffutils

    ## Download

    _Do this each update._

    1. [Pick a mirror](https://www.openssh.com/portable.html)
    2. Download the latest archive.
    This will example will use [https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/openssh-9.0p1.tar.gz](https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/openssh-9.0p1.tar.gz).

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz

    ### Verify the download using `gpg`

    #### Import the public signing key

    _Do this once._

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc
    gpg --import RELEASE_KEY.asc

    #### Verify

    _Do this each update._

    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz.asc
    gpg --verbose --verify openssh-9.0p1.tar.gz.asc

    ## Build

    _Do this each update._

    Preview the next subsection before running these commands.

    VER=9.0p1
    tar -xvf openssh-${VER}.tar.gz
    cd openssh-${VER}
    ./configure --prefix=/opt/openssh-${VER}
    make
    make install

    If `./configure` is missing then the non-portable version was downloaded.

    ### Change the banner (optional)

    _This section has not yet successfully tested 😔. Come back later._

    Change string that _all_ connecting SSH clients receive (this occurs _before authentication_).
    This requires a change before running `make` from the previous _Build_ section.

    The statement that writes the "banner" string to the connected SSH channel is in file `./ssh_api.c`.

    if ((r = sshbuf_putf(banner, "SSH-2.0-%.100s\r\n", SSH_VERSION)) != 0)

    Before running `make`, change the file `./ssh_api.c`

    sed -i -Ee 's|(sshbuf_putf\(banner, )("SSH-.*", SSH_VERSION)(\))|\1"please_go_away"\3|' -- ./ssh_api.c

    This should change the statement to

    if ((r = sshbuf_putf(banner, "please_go_away")) != 0)

    ## Create Symlink for `-latest`

    _Do this once._

    ln -fvs /opt/openssh-${VER} /opt/openssh-latest

    ## Set aside built-in OpenSSH

    _Do this once._

    **Very Important!** _Keep the functioning built-in sshd service in-place or running at a different port_.

    1. Change the system built-in SSH Service `/etc/ssh/sshd_config`, add a non-typical port to the `Ports` declaration, for example `2222`.
    2. Restart the system built-in SSH Service.
    3. Login to the SSH service listening using the non-typical port `2222`.
    `ssh -p 2222 user@server`
    4. Remove from the `Ports` declaration the typical port `22`.
    5. Restart the system built-in SSH Service.

    _OR_

    1. Have the latest SSH service use a non-standard port(s), like `2222`.

    Now there is a reliable fallback SSH Service.

    ## Create systemd service `-latest`

    _Do this once._

    Copy the systemd template service files for the ssh service.

    This path may vary among `/lib/systemd/system`, or `/usr/lib/systemd/system`, or something else.
    Be sure not to use the generated systemd files typically found at `/etc/systemd`.

    Find all `*ssh*` files

    find / -xdev -name '*ssh*' | sort

    Copy the systemd service files. In my case on Debian 11

    cd /usr/lib/systemd/system/
    cp -av ssh.service ssh-latest.service
    cp -av [email protected] [email protected]
    cp -av ssh.socket ssh-latest.socket
    cp -av rescue-ssh.target rescue-ssh-latest.target

    In the newly copied `ssh-latest*` systemd files, manually update references from `ssh` to `ssh-latest`.

    In my case on Debian 11, the changes looked like:

    $ cd /usr/lib/systemd/system

    $ diff -y --suppress-common-lines [email protected] [email protected]
    EnvironmentFile=-/opt/openssh-latest/default/ssh | EnvironmentFile=-/etc/default/ssh
    ExecStart=/opt/openssh-latest/sbin/sshd -i $SSHD_OPTS | ExecStart=-/usr/sbin/sshd -i $SSHD_OPTS
    RuntimeDirectory=sshd-latest | RuntimeDirectory=sshd

    $ diff -y --suppress-common-lines ssh-latest.service ssh.service
    ConditionPathExists=!/opt/openssh-latest/etc/sshd_not_to_be_r | ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
    EnvironmentFile=-/opt/openssh-latest/default/ssh | EnvironmentFile=-/etc/default/ssh
    ExecStartPre=/opt/openssh-latest/sbin/sshd -t | ExecStartPre=/usr/sbin/sshd -t
    ExecStart=/opt/openssh-latest/sbin/sshd -D $SSHD_OPTS | ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
    ExecReload=/opt/openssh-latest/sbin/sshd -t | ExecReload=/usr/sbin/sshd -t
    Type=exec | Type=notify
    RuntimeDirectory=sshd-latest | RuntimeDirectory=sshd
    Alias=sshd-latest.service | Alias=sshd.service

    $ diff -y --suppress-common-lines ssh-latest.socket ssh.socket
    Before=ssh-latest.service | Before=ssh.service
    Conflicts=ssh-latest.service | Conflicts=ssh.service
    ConditionPathExists=!/opt/openssh-latest/etc/sshd_not_to_be_r | ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

    $ diff -y --suppress-common-lines rescue-ssh.target rescue-ssh-latest.target
    Requires=network-online.target ssh.service | Requires=network-online.target ssh-latest.service
    After=network-online.target ssh.service | After=network-online.target ssh-latest.service

    Notice the change of [`Type` from `notify` to `exec`](https://manpages.debian.org/testing/systemd/systemd.service.5.en.html).

    ### Copy the default environment file

    mkdir -vp /opt/openssh-latest/default/
    cp -av /etc/default/ssh /opt/openssh-latest/default/

    The `/opt/openssh-latest/default/ssh` file sets the `$SSHD_OPTS` which is used by the service file.

    ### Enable and restart

    Tell the systemd daemon to enable and start the new services

    systemctl enable ssh-latest.service
    systemctl enable ssh-latest.socket
    systemctl daemon-reload
    systemctl start ssh-latest

    The command `systemctl daemon-reload` will generate new files under `/etc/systemd`.

    Watch the service logs

    journalctl -f -x -u ssh-latest

    Check services status

    systemctl status ssh-latest

    ## Verify new ssh service and old ssh service

    Check the service replies with the expected latest openssh version.
    For each port with a listening ssh service

    echo | nc localhost 22
    echo | nc localhost 2222

    This should look like:

    $ echo | nc localhost 2222
    SSH-2.0-OpenSSH_9.0
    Invalid SSH identification string.

    Check there are two different `sshd -D` daemon processes

    ps -ef --forest | grep "sshd -D"

    Check the ports in-use

    lsof -PVn -iTCP | grep sshd | grep LISTEN

    # openssh updates

    _Do this each update._

    After a new version of OpenSSH Server is released, it's relatively easy to update things.

    Repeat section [_Download_](#download), and [_Build_](#build).

    ## Copy the Previous `-latest` to the New `-latest`

    cd /opt/openssh-${VER}/etc/
    # set aside default config files
    mkdir -vp _original
    mv -v ssh_host_* sshd_config ssh_config _original/
    cp -av /opt/openssh-latest/etc/ssh_host_* /opt/openssh-latest/etc/{sshd_config,ssh_config} .

    ### Use default key files (optional)

    Optionally, the `/opt/openssh-latest/etc/sshd_config` can refer to the key files at `/etc/ssh`. Then there is no need to copy the `/opt/openssh-latest/etc/ssh_host_*` files.

    The file `/opt/openssh-latest/etc/sshd_config` would have lines

    HostKey /etc/ssh/ssh_host_rsa_key
    HostKey /etc/ssh/ssh_host_ecdsa_key
    HostKey /etc/ssh/ssh_host_ed25519_key

    ## Update Symlinks for `-latest`

    systemctl stop ssh-latest
    cd /opt
    rm -v openssh-latest && ln -fvs openssh-${VER} openssh-latest

    ## Restart Service

    Watch the service logs

    journalctl -f -x -u ssh-latest

    Restart

    systemctl restart ssh-latest

    ## Test

    For each port with a listening ssh service

    echo | nc localhost 22
    echo | nc localhost 2222