Scrape 4chan for alive images I'm trying to learn Clojure recently and I thought writing a simple web app would be a good way to dive in.

This function gets the list of alive threads from the API and reduces, filters and maps until I'm left with a list of image names (which I can append to "http://i.4cdn.org/wg/" to get the URL's.).

This is the first time I'm using defer/@ so I'm not that sure if I used them correctly.

Here is the code:

(defn scrape []
  (let
    [{:keys [status hey body error] :as resp}
      @(http/get "http://a.4cdn.org/wg/threads.json")]
    (-&gt;&gt;
      (json/read-str (resp :body))
      (map #(%1 "threads"))
      (reduce concat [])
      (map #(%1 "no"))
      (map #(str "http://a.4cdn.org/wg/thread/" %1 ".json"))
      (map http/get)
      (map deref)
      (map #(%1 :body))
      (map json/read-str)
      (map #(%1 "posts"))
      (reduce concat [])
      (filter #(not (nil? (%1 "tim"))))
      (map #(str (%1 "tim") (%1 "ext")))
      (print))))
Don't need :as resp if only body is ever used.
I'd also drop the :keys destructuring if it's not reused.
While couple of those maps make sense, this sequence is also pretty
repetitive, I'd suggest condensing repeated invocations into a single
call at least.
The (not (nil? ...)) is probably the same as the value itself, so
just (filter #(%1 "tim")) should suffice.
The function should most likely return the list instead of printing
it so the files can be examine further in another part of the code.
Maybe use mapcat instead of the map/reduce/concat combination?


The deref is fine I think, it also depends on how the calls should be
batched together (or not).

All in all:

(ns fourchan.core
  (:require [clojure.data.json :as json])
  (:require [org.httpkit.client :as http]))

(defn scrape []
  (-&gt;&gt;
   (json/read-str (@(http/get "http://a.4cdn.org/wg/threads.json") :body))
   (mapcat #(%1 "threads"))
   (map #(http/get (str "http://a.4cdn.org/wg/thread/" (%1 "no") ".json")))
   (mapcat #((json/read-str ((deref %1) :body)) "posts"))
   (filter #(%1 "tim"))
   (map #(str (%1 "tim") (%1 "ext")))))