How(and when) Clojure Compiles Your Code
How (and When) Clojure Compiles Your Code

 When I started working with Clojure, one of the challenges I faced was understanding
    exactly what Clojure was doing with my code. I was intimately familiar with how Java
    and the JVM works, and tried to slot Clojure into that mental model — but found
    too many cases where my model didn't quite represent reality. The
    docs weren't much
    help: they talked about the features of the language (including compilation), but
    didn't provide detail on what “automatically compiled to JVM bytecode on the
    fly” actually meant.

 I think that detail is important, especially if you come to Clojure from Java or are
    running your Clojure code within a Java-centric framework. And I see enough questions
    on the Internet to realize that not a lot of people actually understand how Clojure
    works. So this post demonstrates a few key points that are the basis of my new mental
    model.

 I'm using Clojure 1.8 for examples, but I believe that everything that I say is
    correct for versions as early as 1.6 and probably before.

 Clojure is a scripting language 

I'll start with some definitions:


 Compiled languages translate source code into an artifact that is then loaded,
     unchanged into the runtime. Any decisions in the code rely on state
     maintained within the executing program.
 Scripting languages load the source code into the runtime, and execute that
     code as it is loading. Scripts may make decisions based on state that they
     manage, as well as global state that has been set by other scripts.


 Clojure is very much a scripting language, even though it compiles its scripts into
    JVM bytecode. All Clojure source files are processed one expression at a time: the
    reader reads characters from the source until it finds a valid expression (a list
    of symbols delimited by balanced parentheses), expands any macros, compiles the
    result to bytecode, then executes that bytecode.

 It doesn't matter whether the source code is entered by hand into the REPL, or read
    from a file as part of a (require ...) form. It's processed a single
    top-level expression at a time.

 Note my term: “top-level” expression. You won't find this term in the
    Clojure docs; they refer to “expressions” and “forms”
    more-or-less interchangeably, but don't differentiate between expressions that
    are nested within other expressions. The reason that I do will become apparent
    later on.

 In my opinion, this form of evaluation is what gives macros their power (the
    oft-proclaimed homoiconicity of the language simply means that they're easier
    to write). A macro is able to use any information that has already been loaded
    into the runtime, including variables that have been created earlier by the same
    or a different script.


 Top-level expressions turn into classes 

 Here's an interesting experiment: start up a Clojure REPL with the following command
    (using the correct path to the Clojure distribution JAR):

java -XX:+TraceClassLoading -jar clojure-1.8.0.jar


 You'll see a long list of classloading messages flash by before you end up at the
    REPL. Now enter a simple expression, such as (+ 1 2); you'll see more 
    messages, as the Clojure runtime loads the classes that it needs. Enter that same
    expression again, and you'll see something like this:

user=> (+ 1 2)
[Loaded user$eval3 from __JVM_DefineClass__]
3


 This message indicates that Clojure compiled that expression to bytecode and then
    loaded the newly-created class to execute it. The class definition is still in
    memory, and you can inspect it. For example, you can look at its superclass
    (I've removed the now-distracting classloader messages):

user=> (.getSuperclass (Class/forName "user$eval3"))
clojure.lang.AFunction


 AFunction is a class within the Clojure runtime; it is a subclass of
    AFn, which implements the invoke() method. With this
     knowledge, it's apparent that the evaluation of this simple expression has four steps:


 Parse the expression (including macro expansion, which doesn't apply to this case)
     and generate the bytes that correspond to a Java .class file.
 Pass these bytes to a classloader, getting a Java Class back.
 Instantiate this class.
 Call invoke on the resulting object.


 You can, in fact, do all of this by hand, provided that you know the classname:

user=> (.invoke (.newInstance (Class/forName "user$eval3")))
3


 OK, so far so good. Now I want to show why earlier I called out a distinction between
    “top-level” expressions and nested expressions:

user=> (* 3 (+ 1 2))
[Loaded user$eval5 from __JVM_DefineClass__]
9


 Here we have an expression that contains a nested expression. However, note that
    only one class was generated as a result. In theory, every expression
    could turn into its own class. Clojure takes a more pragmatic approach, which is
    a good thing for our memory footprint.


 Variables are wrapped in objects 

 To outward appearances, a Clojure variable is similar to a final Java variable:
    you assign it (once) with def, and retrieve its value simply by
    inserting the variable in an expression:

user=> (def x 10)
#'user/x

user=> (class x)
java.lang.Long

user=> (+ x 2)
12


 A hint of the truth can be seen if you attempt to use an unbound var in an
    expression:

user=> (def y)
#'user/y

user=> (+ 2 y)

ClassCastException clojure.lang.Var$Unbound cannot be cast to java.lang.Number  clojure.lang.Numbers.add (Numbers.java:128)


 In fact, variables are instances of
    clojure.lang.Var, which provides functions to get and set
    the variable's value. When you reference a variable within an expression, that
    reference translates into a method call that retrieves the actual value. 

 This allows a great deal of flexibility, including the ability to redefine variables.
    Application code can do this on a per-thread basis using binding and
    set!, or within a call tree using with-redefs. The Clojure
    runtime does much more, such as redefining all variables when you reload a namespace.


 A namespace is not a class 

 For someone coming from a Java background, this is perhaps the hardest thing to
    grasp. A namespace definition certainly looks like a class definition:
    you have a dot-delimited namespace identifier, which corresponds to the path
    where you save the source code. And when you invoke a function from a namespace,
    you use the same syntax that you would to invoke a static method from a Java
    class.

 The first hint that the two aren't equivalent is that the ns macro
    doesn't enclose the definitons within the namespace. Another is that you can
    switch between namespaces at will and add new definitions to each:

user=> (ns foo)
nil
foo=> (def x 123)
#'foo/x
foo=> (ns bar)
nil
bar=> (def x 456)
#'bar/x
bar=> (ns foo)
nil
foo=> (def y 987)
#'foo/y


 You could take the above code snippets, save them in an arbitrary file, and then
    use the load-file function to execute that file as a script. In fact,
    you could write your entire application, with namespaces, as a single script.

 But most (sane) people don't do that. Instead, they create one source file per
    namespace, store that file in a directory derived from the namespace name, and use
    the require function to load it (or more often, a :require
    directive in some other ns declaration).


 Loading code from the classpath: require and load 

 The :require directive is another point of confusion for a Java
    developer starting Clojure. It certainly looks like the import
    statement that we already know, especially when it's used in an ns
    invocation:

(ns example.main
    :require [example.utils :as utils])


 In reality, :require is almost, but not quite, entirely unlike
    import. The Java compiler uses import to load definitions
    from an already-compiled class so that they can be referenced by the class that's
    currently being compiled. On a superficial level, the Clojure runtime does
    the same thing when it sees :require, but it does this by loading
    (and compiling) the source code for that namespace.

 OK, there are some caveats to that statement. First is that require
    only loads a namespace once, unless you specify the :reload option.
    So if the required namespace has already been loaded, it won't be loaded again.
    And if the namespace has already been compiled, and the source file is older than
    the compiled files, then the runtime loads the already compiled form. But still,
    there's a lot of stuff happening as the result of a seemingly simple directive.

 So, let's dig into the behavior of require, along with its step-brother
    load. Earlier I wrote about using load-file to load an
    arbitrary file into the REPL. Here's that file, followed by the command to load and
    run it:

(ns foo)
(def x 123)

(ns bar)
(def x 456)

(ns user)
(do (println "myscript!") (+ foo/x bar/x))


user=> (load-file "src/example/myscript.clj")
myscript!
579


 When you load the file, it creates definitions within the two namespaces, then invokes
    an expression to add them. After loading the file, you can access those variables from
    the REPL:

user=> (* foo/x bar/x)
56088


 The load function is similar, but loads files relative to the classpath.
    It also assumes a .clj extension. I'm using Leiningen, so my classpath
    is everything under src; therefore, I can load the same file like so:

user=> (load "example/myscript")
myscript!
nil


 Wait a second, what happened to the expression at the end of the script? It was still
    evaluated — the println executed — but the result was discarded
    and load returned nil.

 Now let's try loading this same script with require:

user=> (require 'example.myscript :reload)
myscript!
nil


 Different syntax, same result. The two variables are defined in their respective
    namespaces, and the stand-alone expression was evaluated. So what's the difference?

 The first difference is that require gives you a bunch of options.
    For example, you can use :as to create a short alias for the namespace,
    so that you don't have to reference its vars with fully-qualified names. The way
    that the runtime uses these flags is probably worthy of a post of its own.

 Another difference is that require is a little smarter about loading
    scripts: it only loads (and compiles) a script if it hasn't already done so —
    unless, of course, you use the :reload or :reload-all options,
    like I did here. Omitting that option, we see that a second require doesn't
    invoke the println.

user=> (require 'example.myscript)
nil



 Compiling your code (or, :gen-class doesn't do what you might think) 

 As you've seen above, the Clojure runtime normally compiles your code when it's
    loaded, producing the bytes of a .class file but not writing them to
    the filesystem. However, there are times that you want a real, on-disk class.
    For example, so that you can invoke that class from Java (note that you'll still
    need the Clojure JAR on your classpath). Or so that you can reduce startup time
    for a Clojure application, by avoiding load-time compilation (although I think
    this is probably premature optimization).

 The compile function turns Clojure scripts into classes:

user=> (compile 'example.foo)
example.foo


 That was simple enough. Note, however, that I was running in lein repl,
    which sets the *compile-path* runtime global to a directory that it
    knows exists. If you try to execute this function from the clojure.main
    REPL, it will fail unless you create the directory classes.

 Here's the example file that I compiled:

(ns example.foo)

(def x 123)

(defn what [] "I'm compiled!")

(defn add2 [x] (+ 2 x))


 And here are the classes that it produced:

-rw-rw-r--   1 kgregory kgregory     3008 May  7 09:29 target/base+system+user+dev/classes/example/foo__init.class
-rw-rw-r--   1 kgregory kgregory      683 May  7 09:29 target/base+system+user+dev/classes/example/foo$add2.class
-rw-rw-r--   1 kgregory kgregory     1320 May  7 09:29 target/base+system+user+dev/classes/example/foo$fn__1194.class
-rw-rw-r--   1 kgregory kgregory     1503 May  7 09:29 target/base+system+user+dev/classes/example/foo$loading__5569__auto____1192.class
-rw-rw-r--   1 kgregory kgregory      513 May  7 09:29 target/base+system+user+dev/classes/example/foo$what.class


 If you're a bytecode geek like me, you'll of course run javap -c on those
    files to see what they contain (especially fn__1194, which doesn't appear
    anywhere in the source!). Have at it. For everyone else, here are the two things I think
    are important:


 Every function turns into its own class. If you've been reading along,
     you aready knew that.
 The foo__init class is responsible for pulling all of the
     other classes into memory, creating instances of those classes, and
     assigning them to vars in the namespace.


 If you use Leiningen, you've probably noted that it adds a :gen-class
    directive to the main class of any “app” project that it creates. If
    you skim the docs for gen-class you might think this will produce a
    Java class that exposes all of your namespace's functions. Let's see what really
    happens, by adding a :gen-class directive to the example script:

(ns example.foo
  (:gen-class))


 When you compile, the list of classes now looks like this:

-rw-rw-r--   1 kgregory kgregory     1823 May  7 09:31 target/base+system+user+dev/classes/example/foo.class
-rw-rw-r--   1 kgregory kgregory     3009 May  7 09:31 target/base+system+user+dev/classes/example/foo__init.class
-rw-rw-r--   1 kgregory kgregory      683 May  7 09:31 target/base+system+user+dev/classes/example/foo$add2.class
-rw-rw-r--   1 kgregory kgregory     1320 May  7 09:31 target/base+system+user+dev/classes/example/foo$fn__1194.class
-rw-rw-r--   1 kgregory kgregory     1505 May  7 09:31 target/base+system+user+dev/classes/example/foo$loading__5569__auto____1192.class
-rw-rw-r--   1 kgregory kgregory      513 May  7 09:31 target/base+system+user+dev/classes/example/foo$what.class


 Everything's the same, except that we now have foo.class. Looking
    at this class with javap, we find that it contains overrides of the
    basic Object methods: equals(), hashCode(),
    toString(), and clone(). It also creates a Java-standard
    main() function, which looks for the Clojure-standard -main
    (which doesn't exist for our script, so will fail if invoked). But it doesn't expose
    any of your functions.

 Reading the doc more closely, if you want to use :gen-class to expose
    your functions, you need to specify the exposed functions in the directive itself
    — and use a specified naming format that separates the Clojure method
    implementations from the names exposed to Java.


 Pitfalls of compiling your code 

 Let's change the namespace declaration on foo, so that it requires
    bar:

(ns example.foo
  (:require [example.bar :as bar]))


 This results in the the expected classes for foo, but also several
    for bar (which doesn't define any functions):

-rw-rw-r--   1 kgregory kgregory     2219 May  7 09:39 target/base+system+user+dev/classes/example/bar__init.class
-rw-rw-r--   1 kgregory kgregory     1320 May  7 09:39 target/base+system+user+dev/classes/example/bar$fn__1196.class
-rw-rw-r--   1 kgregory kgregory     1503 May  7 09:39 target/base+system+user+dev/classes/example/bar$loading__5569__auto____1194.class
-rw-rw-r--   1 kgregory kgregory     3009 May  7 09:39 target/base+system+user+dev/classes/example/foo__init.class
-rw-rw-r--   1 kgregory kgregory      683 May  7 09:39 target/base+system+user+dev/classes/example/foo$add2.class
-rw-rw-r--   1 kgregory kgregory     1320 May  7 09:39 target/base+system+user+dev/classes/example/foo$fn__1198.class
-rw-rw-r--   1 kgregory kgregory     1891 May  7 09:39 target/base+system+user+dev/classes/example/foo$loading__5569__auto____1192.class
-rw-rw-r--   1 kgregory kgregory      513 May  7 09:39 target/base+system+user+dev/classes/example/foo$what.class


 This makes perfect sense: ff you want to ahead-of-time compile one namespace, you
    probably don't want its dependencies to be compiled at runtime. But recognize that
    the tree of dependencies can run very deep, and will include any third-party
    libraries that you use (poking around Clojars, there aren't a lot of libraries
    that come precompiled).

 There is one other detail of compilation that may cause concern: require
    loads a namespace from the file(s) with the latest modification time. If you have both
    source and compiled classes on your classpath, this could mean that you're not loading
    what you think you are. Fortunately, in practice this primarily affects work in the
    REPL: Leiningen removes the target directory as part of the jar
    and uberjar tasks, so you won't produce an artifact with a source/class
    mismatch.


 Wrap-up 

 This has been a long post, so I'll wrap up with what I consider the main points.


 Startup times for Clojure applications will be longer than for normal Java
     applications, because of the additional step of compiling and evaluating each
     expression. This isn't going to be an issue if you've written a long-running
     server in Clojure, but it does add significant overhead to short-running
     programs (so Clojure is even less appropriate for small command-line utilities
     than Java).
 Pay attention to the Clojure version used by your dependencies, because they
     might rely on functions from a newer version than your application; this
     problem manifests itself as an “Unable to resolve symbol” runtime
     error. While this is a general issue with transitive dependencies, I've found
     that third-party libraries tend to be at the latest version, while corporate
     applications tend to use whatever was current when they were begun.
 As far as I can tell, the Clojure runtime doesn't ever unload the classes that
     it creates. This means that — on pre-1.8 JVMs — you can fill the
     permgen space. Not a big problem in development, but be careful if you use a
     REPL when connected to a production instance.
 Every script that you load adds to the global state of the runtime. Be aware
     that the behavior of your scripts may be dependent on the order that they're
     loaded.


