Skip to content

Instantly share code, notes, and snippets.

@arossouw

arossouw/clj-scraper

Created June 6, 2019 10:07
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save arossouw/4d6cefad52a599fb716d9eb8b1331c83 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save arossouw/4d6cefad52a599fb716d9eb8b1331c83 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. arossouw created this gist Jun 6, 2019.
    33 changes: 33 additions & 0 deletions clj-scraper
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,33 @@
    (ns scraper.core)

    ;; params
    ;; method is either "post" or "get" or any http method
    ;; url is the url for example 'http://www.example.com'
    ;; calling the function
    ;; user=> (http-set-header "post" login_url)
    ;; {:header {:method "post", :url "http://yourip/#/signin?disableAutoSignin=yes'"}}

    (defn http-set-header [method url]
    (assoc {} :header (merge {:method method :url (str url)})))


    ;; url: provide url , 'http://www.example.com
    ;; header: header response, example
    ;; (str (http-set-header "post" 'http://yourip/#/signin?disableAutoSignin=yes'))
    ;; "{:header {:method \"post\", :url \"http://yourip/#/signin?disableAutoSignin=yes'\"}}"

    ;; (defn get-request-header [url request]
    ;; (client/get url {:headers (str request}))

    ;; user=> (get-request-header login_url request)
    ;; user=> (str request)
    ;; "{:header {:method \"post\", :url \"http://yourip/#/signin?disableAutoSignin=yes'\"}}"
    ;; user=> (get-request-header data_url request)
    ;; user=>


    (defn get-request-header [url header-data]
    (client/get (str url) {:header (str header-data)}))

    (defn header-to-string [header]
    (str header))