Skip to content

Instantly share code, notes, and snippets.

@njemanzedavid
Last active July 31, 2025 13:50
Show Gist options
  • Save njemanzedavid/b1fc6fa3a7697947a757ccc973622004 to your computer and use it in GitHub Desktop.
Save njemanzedavid/b1fc6fa3a7697947a757ccc973622004 to your computer and use it in GitHub Desktop.
"Unified Neon + R2" Architecture

The "Unified Neon + R2" Architecture

This model consolidates all structured and vector data into Neon, while dedicating Cloudflare R2 to what it does best: storing large, unstructured files.

Component Service Used Role in the Stack Cost
1. Core n8n App Oracle Cloud Free Tier Your backend eg. the always-on n8n engine. $0
2. Networking Cloudflare Tunnel, DNS, WAF The secure front door to your app. $0
3. Unified Database Neon Serverless Postgres Stores application data (logs, credentials) AND your vector store (embeddings + metadata). $0
4. Raw File Storage Cloudflare R2 Stores all large source files (videos, audio, PDFs, etc.). $0
5. AI Models Cloudflare Workers AI On-demand embeddings and LLM calls. $0
TOTAL $0

Cloudflare R2's Free Tier Maximums

Before we get to the workflow, here are the generous limits for R2 that make this possible:

  • Storage: 10 GiB
  • Class A Operations (Writes/Lists): 10,000,000 requests per month
  • Class B Operations (Reads): 1,000,000 requests per month
  • Egress Bandwidth: Unlimited and Free

The 10GB of free storage is the key number. It gives you a massive runway for your multimodal source files before you ever have to pay a cent.


How This System Design Works (Step-by-Step)

Let's use your multimodal example: an AI that answers questions about a collection of videos and PowerPoint presentations.

Phase 1: Ingestion (Getting data into the system)

  1. Upload: You upload a 50MB video file to your n8n workflow.
  2. Store in R2: The very first step in your n8n workflow is to take that video file and upload it directly to your Cloudflare R2 bucket. This is critical. R2 gives you back a permanent, public or private URL (e.g., https://<bucket>.<account>.r2.dev/my-video.mp4).
  3. Chunk & Transcribe: Your n8n workflow (running on the powerful Oracle VM) can now process the video. It might use a library like ffmpeg to extract the audio, then a transcription model to get the text. It breaks this text into meaningful chunks.
  4. Embed: For each text chunk, the n8n workflow makes an HTTP request to Cloudflare Workers AI to generate its vector embedding.
  5. Store in Neon: The workflow now connects to your single Neon database. For each chunk, it runs a SQL INSERT command into a table you created called knowledge_base. This command saves:
    • The transcribed text chunk.
    • The vector embedding (using the pgvector data type).
    • The URL to the source file in R2 (https://.../my-video.mp4).
    • Metadata, like the timestamp in the video where this text appears (e.g., 5.32 for 5 minutes, 32 seconds).

Phase 2: Retrieval (Answering a question)

  1. User Asks: A user asks, "What was said about the Q3 budget forecast?"
  2. Query Embedding: The n8n workflow takes the user's question and first sends it to Workers AI to get its vector embedding.
  3. Vector Search in Neon: The workflow connects to Neon and performs a vector similarity search on your knowledge_base table. The query looks something like: SELECT text_chunk, source_url, timestamp FROM knowledge_base ORDER BY embedding <=> '<query_vector>' LIMIT 5;
  4. Retrieve Context: The database returns the top 5 most relevant text chunks, along with the R2 URLs and timestamps for each chunk.
  5. Generate Answer: The n8n workflow sends a prompt to the Workers AI LLM. The prompt includes the user's original question and the retrieved text chunks as context.
  6. Present to User: The LLM generates a text answer. Your n8n application can display this answer and provide clickable links that point directly to the specific moments in the videos stored in R2, allowing the user to verify the source.

Advantages of This Unified Architecture

  • Simplicity: You only have one database to manage, connect to, and back up. This dramatically simplifies your credentials and workflow design.
  • Data Integrity: Keeping your vectors and their associated metadata in the same relational database makes it easier to maintain consistency.
  • Powerful & Free: You are still combining the best-in-class free tiers: Oracle for always-on compute, R2 for bulk storage, Neon for a powerful serverless database, and Cloudflare for AI and networking. This design is robust, scalable, and costs nothing to start.

The Complete A-to-Z Deployment Guide

This guide details how to deploy three separate instances/accounts of the bot (Alpha, Beta, Gamma) on Oracle Cloud Free Tier, all connected to a single Neon database, with backups going to Cloudflare R2.

Phase 1: Infrastructure Setup (The Cloud Services)

Step 1.1: Set Up Neon (Unified Database)

  1. Go to Neon.tech and sign up for a free account.
  2. Create a new Project (e.g., telegram-forwarder-db).
  3. Navigate to your project's Dashboard. In the "Connection Details" card, find the connection string that looks like postgresql://user:[email protected]/dbname. Copy and save this string. This is your NEON_CONNECTION_STRING.
  4. Go to the SQL Editor in your Neon project. Run the following command to enable the vector extension, which is good practice for future AI features:
    CREATE EXTENSION IF NOT EXISTS vector;

Step 1.2: Set Up Cloudflare R2 (Backup Storage)

  1. Log in to your Cloudflare Dashboard.
  2. On the left sidebar, go to R2.
  3. Click Create bucket. Name it something unique (e.g., tg-forwarder-backups).
  4. Once created, go to the R2 page and click Manage R2 API Tokens on the right.
  5. Click Create API token. Give it a name (e.g., r2-backup-token) and grant it Admin Read & Write permissions.
  6. Click Create API token. Cloudflare will show you the Access Key ID and the Secret Access Key. Copy and save these immediately. You will not be able to see the Secret Access Key again.

Step 1.3: Set Up Oracle Cloud (VM Hosting)

  1. Sign up for an Oracle Cloud Free Tier account.
  2. Navigate to Compute -> Instances.
  3. Click Create instance. Configure it as follows:
    • Name: tg-bot-alpha (you will repeat this for beta and gamma).
    • Image and shape: Choose an Ubuntu image (e.g., 22.04). Use the default free-tier eligible Ampere (ARM) or AMD (x86) shape.
    • Networking: Ensure "Assign a public IPv4 address" is checked.
    • Add SSH keys: Generate an SSH key pair on your local machine and paste the public key content here. This is how you will log in.
  4. Click Create. Wait for the instance to be provisioned and get its Public IP address.
  5. Repeat this process two more times to create your tg-bot-beta and tg-bot-gamma instances.

Step 1.4: Set Up Telegram Bots

  1. Open Telegram and talk to the @BotFather.
  2. Use the /newbot command to create your first bot. Let's call it Alpha Bot. BotFather will give you a BOT_TOKEN. Save it.
  3. Repeat the /newbot command two more times to create Beta Bot and Gamma Bot, saving each of their unique tokens.

Phase 2: Server Preparation (On Each Oracle VM)

Connect to each of your Oracle VMs via SSH (e.g., ssh ubuntu@<your_public_ip> -i /path/to/your/private_key) and perform the following steps on all three machines.

Step 2.1: Install Dependencies

# Update the system
sudo apt-get update && sudo apt-get upgrade -y

# Install Python, pip, and venv
sudo apt-get install -y python3-pip python3.11-venv git

# Install build-essential for some Python packages
sudo apt-get install -y build-essential

Step 2.2: Set Up Project Directory

# Clone your project (replace with your repo URL if you have one)
git clone <your_repository_url> telegram_script
cd telegram_script

# Or if you're uploading the file manually:
# mkdir telegram_script && cd telegram_script
# (Upload the final .py script into this directory)

# Create a Python virtual environment
python3 -m venv venv

# Activate the environment
source venv/bin/activate

# Create a requirements.txt file
cat > requirements.txt << EOF
telethon
nest_asyncio
python-dotenv
cryptography
asyncpg
boto3
aiolimiter
EOF

# Install the Python packages
pip install -r requirements.txt

Phase 3: Application Configuration (Crucial Step)

On each server, you will create a .env file that contains the shared credentials and the unique credentials for that specific instance.

Step 3.1: Create .env file for the Alpha Instance (on tg-bot-alpha VM)

# In the ~/telegram_script directory
nano .env

Paste the following content, filling in your actual secrets:

# SHARED ACROSS ALL INSTANCES
MASTER_SECRET="your_super_secret_master_password_here"
NEON_CONNECTION_STRING="postgresql://user:[email protected]/dbname"
R2_ENDPOINT_URL="https://<your_account_id>.r2.cloudflarestorage.com"
R2_ACCESS_KEY_ID="your_r2_access_key_id"
R2_SECRET_ACCESS_KEY="your_r2_secret_access_key"
R2_BUCKET_NAME="tg-script-backups"
BOT_API_ID="your_main_telegram_api_id" # Your personal API ID from my.telegram.org
BOT_API_HASH="your_main_telegram_api_hash"

# --- INSTANCE-SPECIFIC: ALPHA ---
INSTANCE_ID="tg_bot_alpha"
INSTANCE_NAME="Telegram Bot Alpha"
BOT_TOKEN="token_for_your_alpha_bot_from_botfather"
ORACLE_REGION="us-ashburn-1" # Or your instance's region

Step 3.2: Create .env files for Beta and Gamma (on their respective VMs) Repeat the nano .env step on the Beta and Gamma VMs, but use their specific INSTANCE_ID, INSTANCE_NAME, and BOT_TOKEN. The shared credentials will be the same.

Phase 4: Deployment & Running the Bot

Step 4.1: Database Initialization On only one of your instances (e.g., the Alpha instance), run the script once manually to allow it to create the database tables.

# Make sure you are in ~/telegram_script and the venv is active
python "ai_studio_code/claude-gpt code enhanced 2 (MODIFIED).py"

You should see logs indicating "Database tables verified/created." Once you see it listening for messages, you can stop it with Ctrl+C. The tables are now ready.

Step 4.2: Create systemd Service for Persistence This will ensure your bot restarts automatically if the server reboots or if the script crashes. On each of the three VMs, create a service file.

# On the ALPHA instance:
sudo nano /etc/systemd/system/tg-bot-alpha.service

Paste the following content. Make sure to replace <your_username> with your actual username on the VM (e.g., ubuntu).

[Unit]
Description=Telegram Bot - Alpha Instance
After=network.target

[Service]
User=<your_username>
Group=<your_username>
WorkingDirectory=/home/<your_username>/telegram_script
ExecStart=/home/<your_username>/telegram_script/venv/bin/python /home/<your_username>/telegram_script/claude-gpt.py"
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
  • Repeat this on the Beta instance, creating /etc/systemd/system/tg-bot-beta.service.
  • Repeat this on the Gamma instance, creating /etc/systemd/system/tg-bot-gamma.service.

Step 4.3: Start and Enable the Services On each respective VM, run the following commands:

# On ALPHA VM:
sudo systemctl enable tg-bot-alpha.service
sudo systemctl start tg-bot-alpha.service
sudo systemctl status tg-bot-alpha.service # Check if it's running

# On BETA VM:
sudo systemctl enable tg-bot-beta.service
sudo systemctl start tg-bot-beta.service
sudo systemctl status tg-bot-beta.service

# On GAMMA VM:
sudo systemctl enable tg-bot-gamma.service
sudo systemctl start tg-bot-gamma.service
sudo systemctl status tg-bot-gamma.service

Your three bot instances are now deployed, running independently but sharing the same backend database and backup location. You can now start a conversation with "Forwarder Alpha Bot", "Forwarder Beta Bot", or "Forwarder Gamma Bot" on Telegram.


=======================================================

SHARED CREDENTIALS (COPY THIS SECTION TO ALL VMs)

=======================================================

A secure, unique password you create. Used to encrypt user data.

IMPORTANT: Use the SAME secret on all instances.

MASTER_SECRET="your_super_secret_master_password_here"

The connection string for your Neon database from the Neon dashboard.

NEON_CONNECTION_STRING="postgresql://user:[email protected]/dbname"

Your personal API credentials from my.telegram.org.

BOT_API_ID="12345678" BOT_API_HASH="your_32_character_api_hash_from_telegram"

Your Cloudflare R2 backup configuration details.

R2_ENDPOINT_URL="https://<your_account_id>.r2.cloudflarestorage.com" R2_ACCESS_KEY_ID="your_r2_access_key_id" R2_SECRET_ACCESS_KEY="your_r2_secret_access_key" R2_BUCKET_NAME="tg-forwarder-backups"

=======================================================

INSTANCE-SPECIFIC CREDENTIALS (CHANGE THIS PER VM)

=======================================================

A unique ID for this specific instance.

INSTANCE_ID="tg_bot_alpha"

A user-friendly name for this instance.

INSTANCE_NAME="Telegram Bot Alpha"

The unique bot token from @BotFather for this instance.

BOT_TOKEN="token_for_your_alpha_bot_from_botfather"

The physical region of your Oracle Cloud VM.

ORACLE_REGION="us-ashburn-1"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment