Sunday, April 22, 2012

Scripting Clojure with Leiningen 2

[2012-May-14] Update: Fixed shebang for more Unix variants

What would it take to write scripts in Clojure (and run them on a terminal) the way we write them in Python, Ruby or Groovy? To pipe scripts together to accomplish bigger tasks? What would it take to throw in argument handling and transitive dependency management capabilities to such scripts? This post attempts to answer some of these questions while illustrating how to use the Leiningen plugin lein-exec to write Clojure scripts.

In particular, I assume you have Leiningen 2 and lein-exec 0.2.0 installed. The shebang feature may require that you use a *nix environment.

Command-line eval

Let us begin with evaluating tiny Clojure S-expressions on the command line.

$ lein2 exec -e "(println \"40 and 50 make\" (+ 40 50))"
40 and 50 make 90


Since the plugin can also evaluate the content from STDIN, we can also do the same thing as above by piping the Clojure code to the plugin:

$ echo "(println \"40 and 50 make\" (+ 40 50))" | lein2 exec
40 and 50 make 90


While clojure.core vars are available, vars from other namespaces may need an explicit reference:

$ lein2 exec -e "(use 'clojure.pprint) (pprint (map inc (range 10)))"
(1 2 3 4 5 6 7 8 9 10)


You would need similar quoting when writing the Clojure scripts due to the fact that S-expressions and scripts are simply evaluated by lein-exec.

Scripting

So, let's start with the first script as a file called fib10.clj that simply prints out first 10 numbers in the Fibonacci series:

;; Taken from http://j.mp/IiT8UK
(def fib-seq
  ((fn rfib [a b]
    (lazy-seq (cons a (rfib b (+ a b)))))
    0 1))

(println (take 10 fib-seq))


We can run it using lein-exec:

$ lein2 exec fib10.clj
(0 1 1 2 3 5 8 13 21 34)


OK, that is fine and dandy but how do we run fib10.clj as an executable? Here's how – you edit the file and put a shebang on the first line:

#!/bin/bash lein-exec

;; Taken from http://j.mp/IiT8UK
(def fib-seq
  ((fn rfib [a b]
    (lazy-seq (cons a (rfib b (+ a b)))))
    0 1))

(println (take 10 fib-seq))


Now you can run the fib10.clj as an executable:

$ chmod a+x fib10.clj
$ ./fib10.clj
(0 1 1 2 3 5 8 13 21 34)


Handling arguments

Scripts often need to work with command-line arguments. While using lein-exec arguments are always passed as string via the clojure.core/*command-line-args* dynamic var – first argument is the name of the script followed by the arguments to the script. Below is an example that takes name and scores and prints out name and average:

#!/bin/bash lein-exec

(defn err-println "println for STDERR"
  [& args]
  (binding [*out* *err*]
    (apply println args)))

(defn parse-int "Parse string as an integer. Abort if invalid."
  [n]
  (try (Integer/parseInt n)
    (catch NumberFormatException e
      (err-println (str \' n \')
                   "is not a valid integer")
      (System/exit 1))))

(defn avg "Given a sequence of numbers return their average."
  [nseq]
  (double (/ (apply + nseq) (count nseq))))

(if (>= (count *command-line-args*) 3)
  (println (second *command-line-args*)
           (avg (map parse-int (drop 2 *command-line-args*))))
  (do (err-println "Usage:" (first *command-line-args*)
                   "name score [score2 ..]")
    (System/exit 1)))


Upon running the script we can inspect how it responds to different input arguments:

$ chmod a+x ./avg.clj
$ ./avg.clj
Usage: ./avg.clj name score [score2 ..]
$ ./avg.clj "Nick Foster"
Usage: ./avg.clj name score [score2 ..]
$ ./avg.clj "Nick Foster" 45 78 65
Nick Foster 62.66666666666667
$ ./avg.clj "Nick Foster" notnum 45 78 65
'notnum' is not a valid integer


Pipeline

Pipeline is a very powerful concept that enables us to chain simple tasks using their STDIN and STDOUT to build more sophisticated tasks. As we would see below, it is entirely possible to build pipe together scripts written in Clojure by simply reading input from STDIN and writing result to STDOUT:

#!/bin/bash lein-exec

(loop []
  (let [c (.read *in*)]
    (when (>= c 0)
      (if (and (>= c (int \A)) (<= c (int \Z)))
        (print (Character/toLowerCase (char c)))
        (print (char c)))
      (recur))))


This script lcase.clj converts the input fed via STDIN to lowercase. Let us see it in action:

$ chmod a+x ./lcase.clj
$ echo "5 Quick brown foxes Jumped over 7 lazy dogs" | ./lcase.clj 
5 quick brown foxes jumped over 7 lazy dogs
$ tree | ./lcase.clj
..result omitted..
$ ls -l ~ | ./lcase.clj
..result omitted..


Transitive dependencies

Scripting with Clojure is fun! It would only be nicer to have the power of transitive dependency management baked into it. Well, pomegranate that is part of Leiningen 2 already makes that possible. lein-exec wraps pomegranate API to provide its own simpler API. The example below shows how to create a demo web service using lein-exec's deps function and Ring:

#!/bin/bash lein-exec

(use '[leiningen.exec :only  (deps)])
(deps '[[ring "1.0.1"]])

(defn handler
  [request]
  {:status 200
   :headers {}
   :body "Hello from Ring!"})

(use 'ring.adapter.jetty)
(run-jetty handler {:port 3000})


The deps function above from leiningen.exec namespace, which is in the lein-exec plugin itself. Here we pull the dpendency ring 1.0.1 which depends on a number of components under it. All those dependencies are pulled in as you run this script. Upon running something this is what you see:

$ chmod a+x ring.clj
$ ./ring.clj
2012-04-22 17:55:35.814:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
2012-04-22 17:55:35.815:INFO::jetty-6.1.25
2012-04-22 17:55:35.824:INFO::Started SocketConnector@0.0.0.0:3000


Depending on whether ring is already downloaded to your local Maven repo, you may see the script downloading them first if they don't already exist.

Working in project context

At times you may want to run a script in the context of a project, so that the code being evaluated has access to the project CLASSPATH and other project resources. All of the above usages of lein-exec support an additional switch -p that does the same thing in project scope.

$ lein2 exec -ep "(use 'foo.bar) (baz :qux)"
..result omitted..
$ cat foo.clj | lein2 exec -p
.. result omitted..
$ lein2 exec -p foo.clj
.. result omitted..


The examples above are applicable when you run them in a project. If you use -p outside of a project it will complain about missing project!

Caveats

While the ability to script in Clojure is fascinating with all the shebang operator, pipeline and dependency management, it's important to know its limitations:

  • As widely known, JVM startup time is a pain even though Leiningen works hard to reduce it.
  • Eval'd code runs in the same JVM that runs Leiningen. There's no easy way to customize that.
  • Currently, the deps function pulls in dependencies only from Maven Central and Clojars.
  • When you run lein-exec in project context, the project map is not accessible to the eval'd code.

Few of these limitations may be addressed in future versions.


I think many Clojure beginners may see scripting as a good way to learn the language and explore Clojure libraries. At the same time, some of the Clojure app/tool projects may find it easy to distribute the app via a script instead of a full blown JAR with instructions on how to run it. Whatever is your impression, feedback and ideas, please share it in comments. You may like to follow me on Twitter: @kumarshantanu

Happy scripting with Clojure!

10 comments:

  1. Thanks for this. I agree with you that scripting is a good way to learn and play with the language and distribute useful code. No sense in letting the "scripting languages" have all the fun. NewLisp takes this approach, and I don't see why Clojure can't be there too. There should be more resources like this article.

    ReplyDelete
  2. "Node.js is the pragmatic choice for ClojureScript command line apps. One of Clojure's weaknesses was that the JVM was not a great choice for these types of scripts. ClojureScript + Node.js effectively solves this problem."
    http://news.ycombinator.com/item?id=2787950

    ReplyDelete
    Replies
    1. While that is a good argument in general, I think when one is heavily invested in the JVM or depends on certain Java libraries or simply does not want to consider anything other than the JVM s/he doesn't have any choice but to use the JVM.

      Delete
  3. Most of this works in Cygwin. The problems I have found are:

    Cygwin shebang seems not to understand "lein exec" so I wrote a short script leinexec:
    lein exec $*

    Cygwin bash seems confused about quoted arguments like "Nick Foster".

    I'm sure I'll explore the Node option once I get more into ClojureScript but this is more direct. Great for making little Clojure tools quickly (even if they may start up a bit slowly)! Thanks!

    ReplyDelete
    Replies
    1. Thanks for the info! Can you please list a sample script that works with Cygwin? I will update the post accordingly and look into how to make it better for the next release of lein-exec.

      Delete
    2. I tried everything in your post except "Working in project context." Apart from the two problems I noted above, the only other thing that did not work is the pipe example with tree, because I don't have tree in Cygwin.

      Delete
    3. Thanks for the feedback; I have updated the lein-exec project on Github with two scripts `lein-exec` and `lein-exec-p` that you can download and use for shebang. I have also updated this blog post to reflect that. Since you don't have `tree` in Cygwin, you can try the example with the `ls` command or whatever that produces uppercase characters in result.

      Delete
  4. Shebang lines are not required to support arguments at all. It is totally acceptable for a unix to stop reading your shebang line after "/bin/sh". Next try please.

    ReplyDelete
    Replies
    1. I have heard of that but didn't run into yet. Can you please let me know which Unix variants don't accept arguments on shebang? This info will help me test/fix it for the next release of lein-exec.

      Delete
  5. Cool! I have been missing the scripting ability of Groovy, and wondering what I'd have to do to emulate that ability in Clojure. Thanks for a very useful blog post!

    ReplyDelete

Disqus for Char Sequence