Clojure RPC, prototyping and early thoughts

Last week, I prototyped an RPC framework, slacker, by clojure and for clojure.

What I did ?

Suppose you have a sets of clojure functions to expose. Define them under a namespace:

(ns slapi)
(defn timestamp []
  (System/currentTimeMillis))

;; ...more functions

Expose the namespace with slacker server, on port 2104:

(use 'slacker.server)
(start-slacker-server (the-ns 'slapi) 2104)

On the client side, we use the `defremote` macro to create a facade for `timestamp` function. This API will keep the client code consistent with local mode.

(use 'slacker.client)
(def sc (slackerc "localhost" 2104))
(defremote sc timestamp)
(timestamp)

Internally, slacker uses aleph for transport and carbonite for serialization. I forked carbonite and made it compatible with clojure 1.2 because the aleph mainline is still running on 1.2.

Going further

High-Order Functions

In clojure, functions are treated as first class value. Within memory, you can pass function as parameter to another function. However, this is not supported by serialization framework. So is it possible to add support for that in future?

Lazy sequence as parameter

This is another interesting feature in clojure function call. You can pass a lazy-sequence to clojure function. In RPC, it requires parameters to be evaluated on the server side.

(defn get-first [& args] (first args))
(apply get-first (range))

Example copied from StackOverflow

Coordinated states between several remote servers

With RPC, we can update states on several servers. So do we need something like distributed dosync:

(defremote a1 update-a1-state)
(defremote a2 update-a2-state)
(dosync-distributed
  (update-a1-state some-value)
  (update-a2-state some-value))

I’m not sure if this is a valid scenario in real world but I think it’s an interesting topic.(distributed STM?)

Conclusion

RPC is the first step to distributed clojure world. I will keep you updated with my prototype.

My response to Spark: Visualize your mercurial commit history from commandline

标题长了些。还是用母语吧。

昨天HackerNews上一个小脚本轰动了,所谓山不在高程序不在小。为了响应这个小脚本,我写了一个更加简单的Mercurial(hg)扩展,帮助你输出hg仓库的提交历史。这个feature和github的直方图有点像,在我看来github的直方图是他们最重要的feature之一,它鞭策着你不断地commit。

hg summary

安装:将这个脚本放在任意一处,在你的hgrc中添加:

[extensions]
summary = /path/to/your/script

由于很简单,就不按照mercurial的规范发布了。

I cloned Beanstalkd with Clojure

I just cloned Beanstalkd with clojure and you can find this project on github. It’s a light weight task queue that “producers” could put tasks in and “workers” are blocked to reserve them and process them. It will help you to split expensive operations (sending email, etc) to background.

The features:

  • Almost compatible with Beanstalkd protocol
  • Full support on Beanstalkd verbs: put/reserve/release/bury/kick
  • Persistence with binary write-ahead logs
  • Simple JMX monitoring

The networking layer is based on @ztellman‘s libraries, aleph, lamina and gloss. And I also received great help from him on implementing the protocol with gloss.

This is not my first clojure project but the largest one. I add marginalia in the dev-dependencies, you can generate well formatted code and docs with `lein marg`.

By the way, this project is not well tested. I’m still working on it to find potential issues and to fix them. Do not use it on any production environment before you have carefully gone through the code base. I just made this announcement here to let people know this project if they are interested in this topic.

And always, Any fork/contribution is welcomed.

Squealer (a test framework for Pig script) is using jip

Squealer is a framework written in Jython to test your Apache Pig scripts, by Mark Roddy. It’s now using jip to resolve Java dependencies. On huge dependencies of Hadoop, jip could be great helpful to setup Squealer.

To get started, it is recommended to create a standalone Jython environment for Squealer:
virtualenv -p /usr/local/bin/jython –no-site-packages squealer-env

Activate the environment
cd squealer-env
. bin/activate

Install jip with pip
pip install jip

Download squealer from bitbucket project page, extract it to somewhere. Install it with jip:
jython setup.py install

Dependencies will be downloaded from Maven Central. You just wait for it to finish.

Start a Jython interpreter with ‘jython-all‘ and now you can import squealer.

Clojure Developers You Should Follow

介绍一下Clojure社区的活跃人物,有助于大家更好地学习和了解clojure,以下罗列的前提是包含但不限于,排名亦不分先后。

Rich Hickey (richhickey)

Clojure的创始人。

Michael Fogus (fogus)

The joy of clojure一书的作者,clojure社区的活跃人物,Clojure/core的成员。参与和维护着很多clojure项目。

James Reeves (weavejester)

Compojure等一系列web框架的作者。是Clojure web开发方面的先驱,还维护着大量的clojure项目。

Mark McGranaghan (mmcgrana)

Clojure web规范Ring的创始人,还维护着clj-redis等项目。

Stuart Hallowa (stuarthalloway)

Programming Clojure一书的作者,Clojure的核心开发人员之一。

Christophe Grand (cgrand)

cgrand是clojure web开发方面的专家,维护了enlive moustache等库。

Phil Hagelberg (technomancy)

Clojure构建工具leiningen的作者。

Zach Tellman (ztellman)

Zach是框架aleph/lamina/gloss的作者,是clojure网络编程的专家。

Nathan Marz (nathanmarz)

Nathan是twitter的clojure dev,维护着一些与hadoop相关的clojure项目。

以上列举的人都可以在github和twitter上找到。