# Interactive Debugging on Github Actions

Did you ever run into some issue where a job would behave slightly different in you CI environment than on your
local machine? Did you ever wish you could run just a few commands in a shell on your build machine?

These are, of course rhetorical questions. And if you're using Github Actions to run your CI jobs, you'll
have noticed that this use case is not supported at all. There are some workarounds (e.g. https://github.com/nektos/act), but since they're not officially supported they can be a bit unstable. Also, even they usually don't reproduce the 
**exact** environment found on github's servers.

### Minimal Reverse Shell

Anyways, here's a cool technique to investigate your CI failures interactively. It's creating a reverse shell from the
build machine, with strict TLS certificate pinning to prevent any random internet person to just look around your build.

First, run this on any server connected to the internet (or at least connected to the build machine):

```
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
openssl s_server -quiet -key key.pem -cert cert.pem -port 2222
```

The `cert.pem` created from the first command here should be made available to the CI job.

In your workflow definition, add the following step:
```
      - name: Do regular CI stuff
        [...]

      - name: Spawn Reverse Shell on Failure
        if: failure()
        run: |
          sudo apt-get -qqy install openssl
          mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -no-CApath -strict -verify 1 -verify_return_error -CAfile cert.pem -connect $SERVER_IP:2222 > /tmp/s; rm /tmp/s
```
(credit to int0x33 for the [original inspiration](https://medium.com/@int0x33/day-43-reverse-shell-with-openssl-1ee2574aa998))

Of course, this technique is not unique to github, it can be used on any CI runner with network connectivity
to the target host. Which these days is almost always the case. If the target port is changed from `2222` to `443`,
it will even go through almost any firewalls.


## Deluxe Reverse Shell

The above solution works great, but after using  it a few times it the limitations become more and more
noticable: There is no auto-completion, no line editing and, worst of all, hitting CTRL-C immediately and
irrevocably terminates the connection.

To remedy all of this, we can set up a ssh reverse shell. The basic idea is to start up a local ssh server
on the CI machine, and then create a reverse ssh tunnel so remote users can connect to the local server.

First, generate a new SSH key pair:

    ssh-keygen -t rsa -f rshell.id_rsa
    
The public and secret key generated by this should be made available to the CI job. We're using the keys
in both directions below, i.e. both to log in to the middle-man machine and to log in one the CI machine.

We also need a middle-man host, which needs to be accessible from the internet, or at least from the CI
machine. On this host, we create an `rshell` user that can be used for establishing the reverse tunnel.
It is set up to be able to login only with the private key generated above. To prevent this user from
doing anything else, its shell is set to `/bin/true`:
    
    adduser --disabled-password --shell /bin/true rshell
    mkdir -p /home/rshell/.ssh
    cp ${PUBLIC_KEY} /home/rshell/.ssh/authorized_keys
    chown -R rshell:rshell /home/rshell/.ssh

The final step is to set up the local ssh server and to create the tunnel: (it should also be possible to use `sshd` as the ssh server, but it has to be started as root and one needs to be careful with the configuration to stop it from interfering with the system.)

    - name: Spawn Deluxe Reverse Shell on Failure
       if: failure()
       env:
         MIDDLEMAN: ${{ secrets.RSHELL_REMOTE_HOST }}
       run: |
         sudo apt-get -qqy install dropbear-bin openssh-client
         mkdir -p ~/.ssh
         echo "${{ secrets.RSHELL_USER_PUBLIC_KEY }}" >> ~/.ssh/authorized_keys
         echo "${{ secrets.RSHELL_USER_SECRET_KEY }}" >> ~/.ssh/rshell.id_rsa
         echo "${{ secrets.RSHELL_REMOTE_HOSTKEY }}" >> ~/.ssh/known_hosts
         chmod 0400 ~/.ssh/rshell.id_rsa ~/.ssh/authorized_keys
         chmod go-w ~
         dropbearkey -t rsa -f ~/.ssh/dropbear.rsa
         dropbearkey -t ecdsa -f ~/.ssh/dropbear.ecdsa
         dropbear -E -R -w -g -a -p 2222 -P ./dropbear.pid -r ~/.ssh/dropbear.rsa -r ~/.ssh/dropbear.ecdsa
         ssh -N -i ~/.ssh/rshell.id_rsa -R 2222:127.0.0.1:2222 ${MIDDLEMAN}

Now, it is possible to log in from the middle-man machine. (if this is a special-purpose machine, one might consider setting `GatewayPorts yes` in the local sshd config to allow 

    $ ssh -i ${PRIVATE_KEY_FILE} -p 2222  runner@127.0.0.1
    
Voila. The connection will persist until the CI job is shut down, and even supports multiple users
investigating the CI job in parallel. As above, this can also be obfuscated by using port 443 for
environments with a restrictive firewall.

This also works on Mac, except that the `apt-get` in the first line has to be replaced by

    brew install dropbear openssh