tech234a · June 10, 2023 14:03 · Jun 9, 2023 · Jun 9, 2023
diff --git a/README.md b/README.md
@@ -10,6 +10,7 @@ This open source version of Reddit lacks some features from the current site, mo
 * new traffic stats
 * chat (you can still send messages directly to users, just not in chat format)
 * [some other API scopes/endpoints](https://www.diffchecker.com/wtIwRNPM/)
+* [anti-spam/anti cheating code](https://github.com/reddit-archive/reddit/wiki/FAQ#is-this-all-of-the-code)
 
 As a proof of concept, here is how to set up a custom Reddit server and use it with **unmodified** third-party Reddit apps. For this use, we will make it appear to your devices that this server is running at reddit.com; other users will continue to be able to access the real Reddit normally. 
 

diff --git a/README.md b/README.md
@@ -0,0 +1,212 @@
+# Using unmodified third-party Reddit apps with a custom server
+
+The [upcoming price increase for the Reddit API](https://old.reddit.com/r/apolloapp/comments/13ws4w3/had_a_call_with_reddit_to_discuss_pricing_bad/) has resulted in the developers of many third-party Reddit apps to [announce plans](https://old.reddit.com/r/apolloapp/comments/144f6xm/apollo_will_close_down_on_june_30th_reddits/) to [shut down](https://old.reddit.com/r/redditsync/comments/144jp3w/sync_will_shut_down_on_june_30_2023/) their [apps](https://old.reddit.com/r/redditisfun/comments/144gmfq/rif_will_shut_down_on_june_30_2023_in_response_to/) on [June 30](https://old.reddit.com/r/ReddPlanet/comments/144glbz/an_unfortunate_goodbye/). Many have encouraged these developers to update their apps to use a different server. Until 2017, the source code for Reddit and its API were open source, and [this code is still available as an archive](https://github.com/reddit-archive/reddit).
+
+This open source version of Reddit lacks some features from the current site, most notably:
+* new Reddit
+* built-in image/video uploading
+* `.i`/`.compact` is still available
+* new Modmail and Modmail APIs
+* new traffic stats
+* chat (you can still send messages directly to users, just not in chat format)
+* [some other API scopes/endpoints](https://www.diffchecker.com/wtIwRNPM/)
+
+As a proof of concept, here is how to set up a custom Reddit server and use it with **unmodified** third-party Reddit apps. For this use, we will make it appear to your devices that this server is running at reddit.com; other users will continue to be able to access the real Reddit normally. 
+
+Standard disclaimers: Follow the instructions at your own risk. These instructions are a proof of concept and may not be suitable for production use. I have heard that sites that previously used Reddit code ended up finding it difficult to maintain and replaced it with a custom rewrite. Additionally, the license agreement for the code states that you need to have "Powered by reddit" somewhere on the webpages from your server. Also, I'm not interested in hosting a public server myself but hopefully these instructions are helpful for someone else.
+
+## Setting up the server
+
+1. The Reddit source code currently only runs on Ubuntu 14.04. While anyone who would want to host a public server should update/recompile dependencies to get it to run on a newer operating system, this is okay for the proof of concept. Ubuntu 14.04 will receive security updates until April 2024. Note: most newer packages do not support Ubuntu 14.04, and you may have issues with outdated SSL certificates installed on the system. Download and install [VirtualBox](https://www.virtualbox.org/wiki/Downloads) and also download the [Ubuntu 14.04.6 server image](https://releases.ubuntu.com/14.04/ubuntu-14.04.6-server-amd64.iso). (The desktop image will also work if you'd like a GUI.)
+2. Create a new VM. Select the ISO you downloaded, skip unattended installation, and give it at least 4096MB of memory. I would also recommend giving it multiple CPUs. The default hard disk size is fine.
+3. Install the OS normally, but set your username to `reddit`. Select "Choose packages manually" at the end.
+4. Once in the system, login and install any security updates: `sudo apt update && sudo apt upgrade -y`.
+5. Now we'll install Reddit, using the [script available in the repo](https://github.com/reddit-archive/reddit/wiki/reddit-install-script-for-Ubuntu) with a minor environment variable tweak:
+```bash
+wget https://raw.github.com/reddit/reddit/master/install-reddit.sh
+chmod +x install-reddit.sh
+sudo REDDIT_DOMAIN=reddit.com ./install-reddit.sh
+```
+Press `y` when prompted.
+
+6. Set up port forwarding for your virtual machine. In the menu bar above your VM, go to `Machine>Settings>Network>Advanced>Port Forwarding` and add the following rule:
+    1. Host port: 8443, Guest port: 443
+Leave protocol as TCP, leave host IP and guest IP blank. Name doesn't matter. Save the changes.
+7. Now update your configuration. Run `cd ~/src/reddit/r2/ && nano development.update` and change the file so it looks as follows:
+```ini
+# after editing this file, run "make ini" to
+# generate a new development.ini
+
+[DEFAULT]
+# global debug flag -- displays pylons stacktrace rather than 500 page on error when true
+# WARNING: a pylons stacktrace allows remote code execution. Make sure this is false
+# if your server is publicly accessible.
+debug = false
+
+disable_ads = true
+disable_captcha = true
+disable_ratelimit = true
+disable_require_admin_otp = true
+
+domain = reddit.com
+oauth_domain = oauth.reddit.com
+https_endpoint = https://reddit.com
+
+disable_wiki = false
+uncompressedJS = true
+
+min_membership_create_community = 0
+
+plugins = 
+
+media_provider = filesystem
+media_fs_root = /srv/www/media
+media_fs_base_url_http = http://%(domain)s/media/
+
+[server:main]
+port = 8001
+```
+Save using `Ctrl-X` then `y`. Then run `make ini` followed by `sudo reddit-restart`.
+
+8. Some differences in the API between the open source version of Reddit and the current version of Reddit break third-party apps, but can easily be fixed using a reverse-proxy. Users of the app and website connect to the reverse proxy server, which then forwards the request to the actual Reddit instance, making any necessary compatibility adjustments to incoming requests and outgoing responses along the way.
+9. On your host machine, download and install/extract the latest version of [OpenResty](https://openresty.org/en/download.html), which is an nginx-based server with Lua scripting support. Update `conf/nginx.conf` to read as follows:
+```conf
+worker_processes  1;
+
+events {
+    worker_connections  1024;
+}
+
+
+http {
+    map_hash_bucket_size 512;
+
+    include       mime.types;
+    default_type  application/octet-stream;
+
+    sendfile        on;
+
+    keepalive_timeout  65;
+
+
+    # remove unsupported OAuth scopes
+    map $request_uri $new_uri {
+        /api/v1/authorize.compact?client_id=5JHxEu-4wnFfBA&response_type=code&state=RedditKit&redirect_uri=apollo%3A%2F%2Freddit-oauth&duration=permanent&scope=account%2Ccreddits%2Cedit%2Cflair%2Chistory%2Cidentity%2Clivemanage%2Cmodconfig%2Cmodflair%2Cmodlog%2Cmodothers%2Cmodposts%2Cmodself%2Cmodwiki%2Cmysubreddits%2Cprivatemessages%2Cread%2Creport%2Csave%2Csubmit%2Csubscribe%2Cvote%2Cwikiedit%2Cwikiread%2Cmodcontributors%2Cmodtraffic%2Cmodmail%2Cstructuredstyles /api/v1/authorize.compact?client_id=5JHxEu-4wnFfBA&response_type=code&state=RedditKit&redirect_uri=apollo%3A%2F%2Freddit-oauth&duration=permanent&scope=account%2Ccreddits%2Cedit%2Cflair%2Chistory%2Cidentity%2Cmodconfig%2Cmodflair%2Cmodlog%2Cmodothers%2Cmodposts%2Cmodself%2Cmodwiki%2Cmysubreddits%2Cprivatemessages%2Cread%2Creport%2Csave%2Csubmit%2Csubscribe%2Cvote%2Cwikiedit%2Cwikiread%2Cmodcontributors%2Cmodtraffic;
+        /api/v1/authorize?client_id=qnjy_qcqie9-Zg&response_type=code&state=RedditKit&redirect_uri=narwhal://oauth&duration=permanent&scope=account,creddits,edit,flair,history,identity,livemanage,modconfig,modflair,modlog,modothers,modposts,modself,modwiki,mysubreddits,privatemessages,read,report,save,submit,subscribe,vote,wikiedit,wikiread /api/v1/authorize?client_id=qnjy_qcqie9-Zg&response_type=code&state=RedditKit&redirect_uri=narwhal://oauth&duration=permanent&scope=account,creddits,edit,flair,history,identity,modconfig,modflair,modlog,modothers,modposts,modself,modwiki,mysubreddits,privatemessages,read,report,save,submit,subscribe,vote,wikiedit,wikiread;
+    }
+
+    # HTTPS server
+    
+    server {
+       listen       443 ssl;
+       server_name  localhost;
+
+       ssl_certificate      reddit.com+1.pem;
+       ssl_certificate_key  reddit.com+1-key.pem;
+
+       ssl_session_cache    shared:SSL:1m;
+       ssl_session_timeout  5m;
+
+       ssl_ciphers  HIGH:!aNULL:!MD5;
+       ssl_prefer_server_ciphers  on;
+
+       location / {
+            if ($new_uri) {
+                return 301 $new_uri;
+            }
+
+            access_by_lua_block
+            {
+                -- replace unsupported sorts with similar options
+                local requri = ngx.var.uri
+                if requri:find("best.json") then
+                    ngx.req.set_uri(ngx.re.gsub(requri, "best.json", "hot.json"), false)
+                end
+                if requri:find("rising.json") then
+                    ngx.req.set_uri(ngx.re.gsub(requri, "rising.json", "new.json"), false)
+                end
+                -- correct type identifiers for posts/comments
+                ngx.req.read_body()
+                local req = ngx.req.get_body_data()
+                local newreq, n, err = ngx.re.gsub(req, "&thing_id=t3_", "&thing_id=t5_")
+                newreq, n, err = ngx.re.gsub(newreq, "&id=t3_", "&id=t5_")
+                ngx.req.set_body_data(newreq)
+            }
+
+            proxy_ssl_name $host;
+            proxy_ssl_server_name on;
+            proxy_ssl_verify off;
+            proxy_set_header Host $host;
+            proxy_set_header Accept-Encoding "";
+            proxy_pass https://127.0.0.1:8443;
+
+            # correct type identifiers in responses
+            header_filter_by_lua_block { ngx.header.content_length = nil }
+            body_filter_by_lua_block {
+                local body = ngx.arg[1]
+                local contenttype = ngx.header.content_type
+                if contenttype == nil then contenttype = "" end
+                if body and (contenttype:find("application/json")) then
+                    body = ngx.re.gsub(body, '"kind": "t3"', '"kind": "AWARD"')
+                    body = ngx.re.gsub(body, '"kind": "t4"', '"kind": "SUBREDDIT"')
+                    body = ngx.re.gsub(body, '"kind": "t5"', '"kind": "LINK"')
+                    body = ngx.re.gsub(body, '"kind": "t6"', '"kind": "MESSAGE"')
+
+                    body = ngx.re.gsub(body, '"kind": "AWARD"', '"kind": "t6"')
+                    body = ngx.re.gsub(body, '"kind": "SUBREDDIT"', '"kind": "t5"')
+                    body = ngx.re.gsub(body, '"kind": "LINK"', '"kind": "t3"')
+                    body = ngx.re.gsub(body, '"kind": "MESSAGE"', '"kind": "t4"')
+                end
+                ngx.arg[1] = body
+            }
+       }
+    }
+
+}
+```
+10. Create `reddit.com+1.pem` and `reddit.com+1-key.pem` using [mkcert](https://github.com/FiloSottile/mkcert/releases). Download `mkcert` and run `mkcert -install` followed by running `mkcert reddit.com "*.reddit.com"` in your `conf` directory.
+
+11. On your host, update your `HOSTS` file or use NextDNS/Pi-hole to point `reddit.com` and `*.reddit.com` your host IP address (127.0.0.1 will work to start). (The specific domains that need to be pointed are reddit.com, www.reddit.com, oauth.reddit.com, and ssl.reddit.com). (Revert this change when you want to connect to the real Reddit).
+
+12. From your server install directory on your host, start the reverse proxy server by running `nginx` (on Windows it will appear to freeze and nothing will be output, this is normal). (You can stop the reverse proxy server by running `nginx -s quit` and reload config by running `nginx -s reload` in a different terminal window).
+
+13. Now we will create the client IDs for the third party Reddit apps. On the host, go to https://reddit.com/prefs/apps/ and create the admin account. Make an account with the username `reddit` and any password. If you get a 500 error on signup, just try logging in anyway, it should work. Reddit normally generates client IDs randomly, but we need it to generate a specific client ID in order for third party apps to work unmodified. To do this, you'll need to make a temporary change to the Reddit source code in the VM and be sure to **revert it once the app is created**. To specify a client ID, you need to change line 58 of `~/src/reddit/r2/r2/models/token.py` from `kwargs["_id"] = cls._generate_unique_token()` to `kwargs["_id"] = 'CLIENT_ID_HERE'` (replace `CLIENT_ID_HERE` with the appropriate value), then restore it to what it was when you're done. You can make this edit using `nano`, and do `sudo reddit-restart` to make sure the change is applied. Then, back on the https://reddit.com/prefs/apps/ webpage, create the app as an Installed App. The values for a few popular iOS apps are as follows:
+    * Apollo: client ID: `5JHxEu-4wnFfBA`, redirect URI: `apollo://reddit-oauth`
+    * Narwhal: client ID: `qnjy_qcqie9-Zg`, redirect URI: `narwhal://oauth`
+    * BaconReader: client ID: `up6Arjatrs1nzw`, redirect URI: `baconreader://redditauth`
+
+(I did not omit the client IDs because Reddit does not consider them to be private information; Reddit emails the user this information every time they log into one of these apps.)
+You can also find these values in the URL of the Reddit login page for a given app. I found that the app creation worked best from the admin account.
+
+## Connecting a mobile device to the custom Reddit instance
+1. I would recommend logging out of any mobile apps before beginning this process.
+
+2. First, you'll need to install the root certificate authority used to sign your fake reddit.com SSL certificate onto your mobile device. (HTTPS certificates are built on a chain of trust, if you don't install the certificate your device will correctly not trust the server that is pretending to be reddit.com).
+    * For the next step, you'll need your host IP address/hostname. You can find your host IP using `ipconfig` on Windows, or `ifconfig` on some other platforms. ([Tailscale](https://tailscale.com/) could make this easier. If you do use Tailscale, be sure to update your Tailscale settings in the web admin console to use the custom NextDNS configuration.) On your host, you can find the certificate by running `mkcert -CAROOT`.
+    * On iOS, you can install the certificate from Safari by clicking on a link that leads to the `.pem` file. The easiest way to do this is navigate to the folder containing your certificate on the host, run `python -m http.server 8000` and then in Safari, navigate to `http://[your host ip]:8000/` and click the link to the `.pem` file (not the key). You may also alternatively be able to download the certificate as an attachment in the iOS Mail app. After that, open iOS settings, go to `General>VPN, DNS, & Device Management`, and install the downloaded configuration profile. Finally, go to `Settings>General>About>Certificate Trust Settings` and turn on trust for the installed certificate.
+    * Instructions will differ for Android.
+
+3. Set up custom DNS to point your mobile device to the custom Reddit server.
+    * This can most easily be done using NextDNS with a custom configuration, and should also be possible using Pi-hole.
+    * Make an account on https://my.nextdns.io/, create a configuration and go to the Settings tab, and add a rewrite for `reddit.com` and set the answer to the IP address (or hostname if using Tailscale) of the host that you are running your reverse proxy server on.
+    * If you use Apollo, I also recommend going to the denylist tab and blocking `apolloreq.com` to prevent the queries being made to the custom server from being counted towards the developer's API usage statistics tracking.
+
+4. Activate the DNS, either by connecting to Tailscale or using the NextDNS app with the custom configuration specified.
+
+5. Because the default configuration disallows admins/employees from logging into third-party apps, create a new user with any username to use within the apps.
+
+6. Open and optionally log into your desired app.
+
+7. The app should be mostly usable.
+    * Browsing, posting, commenting, upvoting, etc. should mostly work.
+    * In-app Modmail will not work because of missing APIs, but many other moderation features appear to be working.
+    * Search may be possible to configure but I couldn't get it to work; [see instructions in the Reddit source archive](https://github.com/reddit-archive/reddit/tree/master/solr).
+    * Support for sending emails (confirmation, password reset, newsletter, etc.) is not configured.
+    * Because best sort and rising sort did not exist in the open source version of Reddit, the reverse proxy replaces best sort with hot sort and rising sort with new sort.
+    * The reverse proxy also corrects some type IDs in the API to better match what the apps are expecting; for this proof of concept this is essentially implemented as a find and replace which could possibly mess up content in a few rare scenarios, though this seems unlikely.
+    * Some apps will not immediately be able to log in and will present an error on the authorization page because they request an invalid scope that was not present in the open source version of Reddit. You can fix this by adding an entry in the map in `nginx.conf` for the reverse proxy that redirects the login URL used by the app to one that does not include the invalid scopes, and then reloading the server. See the map section of `nginx.conf` which provides examples for Apollo and Narwhal; BaconReader did not request any of the invalid scopes and thus did not require an entry. The scopes that would need to be removed (if present) are `structuredstyles`, `modmail`, `livemanage`, and `modnote`.
+
+8. To connect to real Reddit instead, simply log out and close your app then switch off NextDNS/Pi-hole/Tailscale. (You can leave the certificate installed for convenience. On iOS, you can turn off trust if you'd like.)
+
+## A request for third-party app developers
+Before shutting down your apps, please consider pushing a simple app update to enable specifying a custom client ID and Reddit API domain, as this would greatly simplify setup for allowing users to connect to real Reddit with a custom client ID or to connect to a custom Reddit server.