Inverted index in Clojure - performance vs. idiomatic code
You should be able to use reducers to get a good speed boost as the consumption and building of your index can be parallelized.

I have found the best docs for reducers to be the doc strings themselves:
Reducers -- Github

(ns ...
  (:require [clojure.core.reducers :as r]
            [clojure.java.io :as io]
            [clojure.string :as str]))

(defn invert [file]
  (let [f (.getName file)
        tokens (str/split (str/lower-case (slurp file)) p)]
        (-&gt;&gt; tokens
             (r/mapcat #(hash-map % #{f}))
             (into {}))))

(defn build-index 
  [dirname]
  (-&gt;&gt; dirname
       io/file
       (.listFiles)
       (r/map invert)
       (r/fold (r/monoid (partial merge-with union) hash-map))))