Intercepts HTTPs Traffic with Python & mitmproxy
================================================


Introduction
------------
Modern applications usually make use of back-end API servers to provide their services. With a non-transparent HTTPs proxy, which intercepts the communication between clients and servers (aka the man-in-the-middle scheme), you can easily manipulate both API requests and responses.

This manual helps you create your own proxy with Python and [mitmproxy/libmproxy](http://mitmproxy.org). Mitmproxy ships with both a standalone command-line tool (`mitmproxy`) and a Python library (libmproxy).


Requirements
------------
* Python, >= 2.7.3
    - install through `brew install python` on Mac OS X
* mitmproxy
    1. `git clone https://github.com/cortesi/mitmproxy.git`
    2. `cd mitmproxy`
    3. `sudo python setup.py install`
* netlib, version matching mitmproxy
    1. `git clone https://github.com/cortesi/netlib.git`
    2. `cd netlib`
    3. `sudo python setup.py install`
* PyOpenSSL, >= 0.13
    1. install OpenSSL development package (through `sudo apt-get install libssl-dev` on Ubuntu)
    2. download `pyOpenSSL-0.13.tar.gz` from [PyPi website](https://pypi.python.org/pypi/pyOpenSSL)
    3. `tar xvf pyOpenSSL-0.13.tar.gz`
    4. `cd pyOpenSSL-0.13`
    5. `python setup.py build`
    6. `sudo python setup.py install`
* pyasn1, >= 0.1.2
    - `pip install pyasn1`

**Note:** In my experience, mitmproxy depends on the *latest* netlib and PyOpenSSL, which cannot be installed from Pip. You may download the source tarball and install them manually.


Generate SSL Private Key and Certificate
----------------------------------------
1. `openssl genrsa -out mitmproxy.key 2048`
2. `openssl req -new -x509 -key mitmproxy.key -out mitmproxy.crt -days 3650 -subj /CN=MitmProxy`
3. `cat mitmproxy.key mitmproxy.crt > mitmproxy.pem`
4. install `mitmproxy.crt` on you device (desktop browser, iPhone, Android, etc.)


Example of `proxy.py`
---------------------
```python
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
from libmproxy import controller, proxy
import os, sys, re, datetime, json

class InterceptingMaster (controller.Master):
  def __init__ (self, server):
    controller.Master.__init__(self, server)

  def run (self):
    while True:
      try:
        controller.Master.run(self)
      except KeyboardInterrupt:
        print 'KeyboardInterrupt received. Shutting down'
        self.shutdown()
        sys.exit(0)
      except Exception:
        print 'Exception catched. Intercepting proxy restarted'
        pass

  def handle_request (self, msg):
    timestamp = datetime.datetime.today().strftime("%Y/%m/%d %H:%M:%S")
    client_ip = msg.client_conn.address[0]
    request_url = '%s://%s%s' % (msg.scheme, .msg.host, msg.path)
    print '[%s %s] %s %s' % (timestamp, client_ip, msg.method, request_url)

    msg.reply()

  def handle_response (self, msg):
    # simple substitution for https://example.org/api/users/:id.json
    if "example.org" in msg.request.host:
      regex = re.compile("/api/users/(\d+).json")
      match = regex.search(msg.request.path)
      if match and msg.content:
        c = msg.replace('"private_data_accessible":false', '"private_data_accessible":true')
        if c > 0: print '[Example.org] Private info of user #%s revealed' % (match.groups()[0])

    # JSON manipulation for https://example.com/api/v2
    if ("example.com" in msg.request.host) and ("action=user_profile" in msg.request.content):
      msg.decode() # need to decode the message first
      data = json.loads(msg.content) # parse JSON with decompressed content
      data["access_granted"] = true
      msg.content = json.dumps(data) # write back our changes
      print '[Example.com] Access granted of user profile #%s' % data["id"]

    # Response inspection for https://example.net
    if "example.net" in msg.request.host:
      data = msg.get_decoded_content() # read decompressed content without modifying msg
      print '[Example.net] Respones: %s' % data

    msg.reply()


def main (argv):
  config = proxy.ProxyConfig(
    cacert = os.path.expanduser("./mitmproxy.pem"),
  )
  server = proxy.ProxyServer(config, 8080)
  print 'Intercepting Proxy listening on 8080'
  m = InterceptingMaster(server)
  m.run()

if __name__ == '__main__':
  main(sys.argv)
```