Skip to content

Instantly share code, notes, and snippets.

@bikcrum

bikcrum/Connect Google Colab+Drive with SSH.ipynb

Last active October 16, 2024 03:23
Show Gist options
  • Save bikcrum/f9d4cfb0c40dff23329b89cd664d4964 to your computer and use it in GitHub Desktop.
Save bikcrum/f9d4cfb0c40dff23329b89cd664d4964 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. bikcrum revised this gist May 24, 2020. 1 changed file with 0 additions and 2 deletions.
    2 changes: 0 additions & 2 deletions Connect Google Colab+Drive with SSH.ipynb
    View file
    Original file line number Diff line number Diff line change
    @@ -586,8 +586,6 @@
    "from google.colab import drive\n",
    "drive.mount('/content/drive', force_remount=True)\n",
    "\n",
    "!mkdir -p /drive\n",
    "!mount --bind /content/drive/My\\ Drive /drive\n",
    "# To unmount using following\n",
    "# drive.flush_and_unmount()"
    ],
  2. bikcrum created this gist May 18, 2020.
    598 changes: 598 additions & 0 deletions Connect Google Colab+Drive with SSH.ipynb
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,598 @@
    {
    "nbformat": 4,
    "nbformat_minor": 0,
    "metadata": {
    "colab": {
    "name": "Connect Google Colab+Drive with SSH.ipynb",
    "provenance": [],
    "collapsed_sections": [
    "DDsNUSe-lWMk",
    "-framHcYlZZl",
    "hnP5d6SBmPOi",
    "azVypUY2nQiT",
    "woSIgj7TnImq",
    "syBlFACknx-D",
    "WV2KqCkQoVWn"
    ]
    },
    "kernelspec": {
    "name": "python3",
    "display_name": "Python 3"
    }
    },
    "cells": [
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "WCXdafPUkW62",
    "colab_type": "text"
    },
    "source": [
    "### Configure NGROK"
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "DDsNUSe-lWMk",
    "colab_type": "text"
    },
    "source": [
    "#### Installation"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "z5At1w0bk1O5",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip -O ngrok-stable-linux-amd64.zip\n",
    "!unzip -u ngrok-stable-linux-amd64.zip"
    ],
    "execution_count": 0,
    "outputs": []
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "-framHcYlZZl",
    "colab_type": "text"
    },
    "source": [
    "#### Copy Ngrok to root and change permission"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "fTOJQv7mlqDY",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "!cp /content/ngrok /ngrok\n",
    "!chmod +x /ngrok"
    ],
    "execution_count": 0,
    "outputs": []
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "zx8FRSbglwhm",
    "colab_type": "text"
    },
    "source": [
    "#### Authorize ngrok\n",
    "\n",
    "- Sign up for a free ngrok account: https://ngrok.com\n",
    "- Copy your authtoken from https://dashboard.ngrok.com/auth and paste it into the following cell\n",
    "- Run the cell"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "sv8y5ofImKwX",
    "colab_type": "code",
    "colab": {
    "base_uri": "https://localhost:8080/",
    "height": 34
    },
    "outputId": "8961de9c-d2ff-4e20-821c-1ed5690f9991"
    },
    "source": [
    "!/ngrok authtoken YOUR_AUTH_TOKEN_HERE"
    ],
    "execution_count": 9,
    "outputs": [
    {
    "output_type": "stream",
    "text": [
    "Authtoken saved to configuration file: /root/.ngrok2/ngrok.yml\n"
    ],
    "name": "stdout"
    }
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "hnP5d6SBmPOi",
    "colab_type": "text"
    },
    "source": [
    "#### Make config file for ngrok"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "C2s9ifkWmt4X",
    "colab_type": "code",
    "colab": {
    "base_uri": "https://localhost:8080/",
    "height": 34
    },
    "outputId": "dbfbca0a-13f5-4dd5-eefa-83b130ed9b4d"
    },
    "source": [
    "%%writefile /content/ssh.yml\n",
    "tunnels:\n",
    " ssh:\n",
    " proto: tcp\n",
    " addr: 22"
    ],
    "execution_count": 10,
    "outputs": [
    {
    "output_type": "stream",
    "text": [
    "Writing /content/ssh.yml\n"
    ],
    "name": "stdout"
    }
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "MXJ5FhF3m3A7",
    "colab_type": "code",
    "colab": {
    "base_uri": "https://localhost:8080/",
    "height": 34
    },
    "outputId": "f116118a-e0ac-4292-9d45-a2851cd37082"
    },
    "source": [
    "%%writefile /content/run_ngrok.sh\n",
    "#!/bin/sh\n",
    "set -x\n",
    "/ngrok start --config ~/.ngrok2/ngrok.yml --config /content/ssh.yml ssh"
    ],
    "execution_count": 11,
    "outputs": [
    {
    "output_type": "stream",
    "text": [
    "Writing /content/run_ngrok.sh\n"
    ],
    "name": "stdout"
    }
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "T3VC0sfCm8nR",
    "colab_type": "text"
    },
    "source": [
    "### Configura SSH"
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "azVypUY2nQiT",
    "colab_type": "text"
    },
    "source": [
    "#### Install ssh"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "uq2RkeOwnCa4",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "!apt-get install ssh"
    ],
    "execution_count": 0,
    "outputs": []
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "woSIgj7TnImq",
    "colab_type": "text"
    },
    "source": [
    "#### Overwrite sshd_config file "
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "D-DMY1iEnXie",
    "colab_type": "code",
    "colab": {
    "base_uri": "https://localhost:8080/",
    "height": 34
    },
    "outputId": "7dc6c2b4-9d49-4966-efbf-ebb3b066e21a"
    },
    "source": [
    "%%writefile /etc/ssh/sshd_config\n",
    "#\t$OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $\n",
    "\n",
    "# This is the sshd server system-wide configuration file. See\n",
    "# sshd_config(5) for more information.\n",
    "\n",
    "# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin\n",
    "\n",
    "# The strategy used for options in the default sshd_config shipped with\n",
    "# OpenSSH is to specify options with their default value where\n",
    "# possible, but leave them commented. Uncommented options override the\n",
    "# default value.\n",
    "\n",
    "#Port 22\n",
    "#AddressFamily any\n",
    "#ListenAddress 0.0.0.0\n",
    "#ListenAddress ::\n",
    "\n",
    "#HostKey /etc/ssh/ssh_host_rsa_key\n",
    "#HostKey /etc/ssh/ssh_host_ecdsa_key\n",
    "#HostKey /etc/ssh/ssh_host_ed25519_key\n",
    "\n",
    "# Ciphers and keying\n",
    "#RekeyLimit default none\n",
    "\n",
    "# Logging\n",
    "#SyslogFacility AUTH\n",
    "#LogLevel INFO\n",
    "\n",
    "# Authentication:\n",
    "\n",
    "#LoginGraceTime 2m\n",
    "#PermitRootLogin prohibit-password\n",
    "#StrictModes yes\n",
    "#MaxAuthTries 6\n",
    "#MaxSessions 10\n",
    "\n",
    "#PubkeyAuthentication yes\n",
    "\n",
    "# Expect .ssh/authorized_keys2 to be disregarded by default in future.\n",
    "#AuthorizedKeysFile\t.ssh/authorized_keys .ssh/authorized_keys2\n",
    "\n",
    "#AuthorizedPrincipalsFile none\n",
    "\n",
    "#AuthorizedKeysCommand none\n",
    "#AuthorizedKeysCommandUser nobody\n",
    "\n",
    "# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts\n",
    "#HostbasedAuthentication no\n",
    "# Change to yes if you don't trust ~/.ssh/known_hosts for\n",
    "# HostbasedAuthentication\n",
    "#IgnoreUserKnownHosts no\n",
    "# Don't read the user's ~/.rhosts and ~/.shosts files\n",
    "#IgnoreRhosts yes\n",
    "\n",
    "# To disable tunneled clear text passwords, change to no here!\n",
    "#PasswordAuthentication yes\n",
    "#PermitEmptyPasswords no\n",
    "\n",
    "# Change to yes to enable challenge-response passwords (beware issues with\n",
    "# some PAM modules and threads)\n",
    "ChallengeResponseAuthentication no\n",
    "\n",
    "# Kerberos options\n",
    "#KerberosAuthentication no\n",
    "#KerberosOrLocalPasswd yes\n",
    "#KerberosTicketCleanup yes\n",
    "#KerberosGetAFSToken no\n",
    "\n",
    "# GSSAPI options\n",
    "#GSSAPIAuthentication no\n",
    "#GSSAPICleanupCredentials yes\n",
    "#GSSAPIStrictAcceptorCheck yes\n",
    "#GSSAPIKeyExchange no\n",
    "\n",
    "# Set this to 'yes' to enable PAM authentication, account processing,\n",
    "# and session processing. If this is enabled, PAM authentication will\n",
    "# be allowed through the ChallengeResponseAuthentication and\n",
    "# PasswordAuthentication. Depending on your PAM configuration,\n",
    "# PAM authentication via ChallengeResponseAuthentication may bypass\n",
    "# the setting of \"PermitRootLogin without-password\".\n",
    "# If you just want the PAM account and session checks to run without\n",
    "# PAM authentication, then enable this but set PasswordAuthentication\n",
    "# and ChallengeResponseAuthentication to 'no'.\n",
    "UsePAM yes\n",
    "\n",
    "#AllowAgentForwarding yes\n",
    "AllowTcpForwarding yes\n",
    "#GatewayPorts no\n",
    "X11Forwarding yes\n",
    "#X11DisplayOffset 10\n",
    "#X11UseLocalhost yes\n",
    "#PermitTTY yes\n",
    "PrintMotd no\n",
    "#PrintLastLog yes\n",
    "#TCPKeepAlive yes\n",
    "#UseLogin no\n",
    "#PermitUserEnvironment no\n",
    "#Compression delayed\n",
    "#ClientAliveInterval 0\n",
    "#ClientAliveCountMax 3\n",
    "#UseDNS no\n",
    "#PidFile /var/run/sshd.pid\n",
    "#MaxStartups 10:30:100\n",
    "#PermitTunnel no\n",
    "#ChrootDirectory none\n",
    "#VersionAddendum none\n",
    "\n",
    "# no default banner path\n",
    "#Banner none\n",
    "\n",
    "# Allow client to pass locale environment variables\n",
    "AcceptEnv LANG LC_*\n",
    "\n",
    "# override default of no subsystems\n",
    "Subsystem\tsftp\t/usr/lib/openssh/sftp-server\n",
    "\n",
    "# Example of overriding settings on a per-user basis\n",
    "#Match User anoncvs\n",
    "#\tX11Forwarding no\n",
    "#\tAllowTcpForwarding no\n",
    "#\tPermitTTY no\n",
    "#\tForceCommand cvs server"
    ],
    "execution_count": 13,
    "outputs": [
    {
    "output_type": "stream",
    "text": [
    "Overwriting /etc/ssh/sshd_config\n"
    ],
    "name": "stdout"
    }
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "syBlFACknx-D",
    "colab_type": "text"
    },
    "source": [
    "#### Make .ssh directory in root"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "WQaNRQ_wn41F",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "!mkdir -p ~/.ssh"
    ],
    "execution_count": 0,
    "outputs": []
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "zbQdx_Pgn52L",
    "colab_type": "text"
    },
    "source": [
    "#### Copy content of id_rsa.pub from your system and paste below"
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "mi9NAas-y7eq",
    "colab_type": "text"
    },
    "source": [
    "- Goto `~/.ssh` on your system\n",
    "- Find id_rsa.pub then open it\n",
    "- In case if you don't have it create one using `ssh-keygen -t rsa`. Enter nothing in file name and password.\n",
    "- Copy content from id_rsa.pub and paste in the following cell"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "GNJw9O1yoJ57",
    "colab_type": "code",
    "colab": {
    "base_uri": "https://localhost:8080/",
    "height": 34
    },
    "outputId": "6a11f781-e063-40b5-8127-843a11ad04ac"
    },
    "source": [
    "%%writefile ~/.ssh/authorized_keys\n",
    "CONTENT_OF_ID_RSA_PUB_FILE"
    ],
    "execution_count": 15,
    "outputs": [
    {
    "output_type": "stream",
    "text": [
    "Writing /root/.ssh/authorized_keys\n"
    ],
    "name": "stdout"
    }
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "WV2KqCkQoVWn",
    "colab_type": "text"
    },
    "source": [
    "#### Restart ssh service"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "XidUAxZ6ocp9",
    "colab_type": "code",
    "colab": {
    "base_uri": "https://localhost:8080/",
    "height": 51
    },
    "outputId": "35aacbde-b16e-49e6-cd7e-68ad755609f5"
    },
    "source": [
    "!service ssh restart"
    ],
    "execution_count": 16,
    "outputs": [
    {
    "output_type": "stream",
    "text": [
    " * Restarting OpenBSD Secure Shell server sshd\n",
    " ...done.\n"
    ],
    "name": "stdout"
    }
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "Bqpvztxnodj9",
    "colab_type": "text"
    },
    "source": [
    "\n",
    "### START SSH TUNNELING"
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "jA6uBL8lrmOy",
    "colab_type": "text"
    },
    "source": [
    "#### Run .sh file created above"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "LeQKIcQlonGJ",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "get_ipython().system_raw('bash /content/run_ngrok.sh ssh &')"
    ],
    "execution_count": 0,
    "outputs": []
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "o2jHPggnrvEb",
    "colab_type": "text"
    },
    "source": [
    "#### Show ssh connect command"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "MIfMw7xHr0qU",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "import requests\n",
    "import urllib.parse\n",
    "\n",
    "def get_ngrok_info():\n",
    " return requests.get('http://localhost:4040/api/tunnels').json()\n",
    "\n",
    "def get_ngrok_tunnels():\n",
    " for tunnel in get_ngrok_info()['tunnels']:\n",
    " name = tunnel['name']\n",
    " yield name, tunnel\n",
    "\n",
    "def get_ngrok_tunnel(name):\n",
    " for name1, tunnel in get_ngrok_tunnels():\n",
    " if name == name1:\n",
    " return tunnel\n",
    "\n",
    "def get_ngrok_url(name, local=False):\n",
    " if local:\n",
    " return get_ngrok_tunnel(name)['config']['addr']\n",
    " else:\n",
    " return get_ngrok_tunnel(name)['public_url']\n",
    "\n",
    "\n",
    "public_url, public_port = urllib.parse.urlparse(get_ngrok_url('ssh')).netloc.split(':')\n",
    "print('To SSH into this colab instance, run the following command on your local machine:')\n",
    "print('ssh root@{} -p {}'.format(public_url, public_port))"
    ],
    "execution_count": 0,
    "outputs": []
    },
    {
    "cell_type": "markdown",
    "metadata": {
    "id": "tN9F_W45spkF",
    "colab_type": "text"
    },
    "source": [
    "#### Mount Google Drive to access it's file using SSH"
    ]
    },
    {
    "cell_type": "code",
    "metadata": {
    "id": "ZqAf7BrBswXz",
    "colab_type": "code",
    "colab": {}
    },
    "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive', force_remount=True)\n",
    "\n",
    "!mkdir -p /drive\n",
    "!mount --bind /content/drive/My\\ Drive /drive\n",
    "# To unmount using following\n",
    "# drive.flush_and_unmount()"
    ],
    "execution_count": 0,
    "outputs": []
    }
    ]
    }