Monthly Archives: December 2011

Extend slacker server with interceptors

Posted on by
Reply

An interceptor framework was introduced in slacker 0.3.0. It’s designed to allow user to add custom functionality without hacking into the internal of slacker.

Like many server frameworks, slacker abstracts the request processing as a pipeline. The request object is modified by adding or updating attributes through each node of the pipeline. So it’s easy to add your interceptor into the pipeline, with which you can get the data before and after function executed.

To create such an interceptor, you should use the slacker.interceptor/definterceptor macro and slacker.interceptor/definterceptor+ macro:

(definterceptor name
:before interceptor-function
:after interceptor-function)

(definterceptor+ name [arguments]
:before interceptor-function
:after interceptor-function)

definterceptor+ can accept arguments so you can configure the interceptor when you use it.

See a simple example:

(definterceptor log-function-call
  :before (fn [req] (println (str "calling " (:fname req))) req))

(definterceptor+ log-function-call-prefixed [prefix]
  :before (fn [req] (println (str
                               (if (fn? prefix) (prefix) prefix)
                               " calling "
                               (:fname req)))
                    req))

Then, add it to your slacker server by

(use '[slacker.interceptor])
(import '[java.util Date])
(start-slacker (the-ns 'slapi) 2104
  :interceptors (interceptors log-function-call
                              (log-function-call-prefixed
                                (fn [] (.toString (Date.)))))

Now you can log every function call of your slacker server.

For more detail about the interceptor framework, especially the request data, please check the wiki page.

In slacker 0.3.0, there is a built-in interceptor to stats function calls. You can find it at slacker.interceptors.stats. The stats data is expose via JMX. You can also write monitoring application to retrieve the data.

And there will be more built-in interceptors in 0.4.0, includes function call time stats and logging.

使用Clojure Thread Macro的心得

Posted on by
2

Thread Macro是clojure里一个很强大的宏，他帮助你简化嵌套函数的调用，比如

(str (inc (count [:a :b])))

就可以利用thread macro简写成

(-> [:a :b] count inc str)

->>和->类似，区别在于->>把值作为函数的最后一个参数传入。

简单的功能介绍完了，接下来就遇到问题了。我需要功能，能够接受一个或多个函数，然后把这些函数组成一个pipeline。这时很自然想到->是一个好帮手。也许只需要一个类似这样的form就可以了： #(apply -> % [funcs])。结果失败了，因为->是个宏，所以根本不能用apply。于是想到有apply-macro吗？有，或者说曾经有过。在contrib中曾经有一个apply-macro，不过被强烈不推荐使用。到这里，这条路堵死了，惟一的办法就是把->放到API之外，放到用户代码里去。

放到用户代码里，你需要写一个详细的说明文档并且告诉用户他必须这么做。然而在clojure世界里有一个更好的办法就是再写一个宏把->包装起来。这么做看似多此一举，其实是保持了API的一致性。通过宏，我们可以把自己的API延伸到用户代码中去。或者说，通过一个类似DSL的宏，给一些并不优雅的API一个缓冲，也为API日后的演化留下空间。

这里还要扯开一句关于宏的开发。clojure中所谓code is data，主要就是体现在宏里。原本在多数其他语言里，宏是不能求值的。但是在clojure里，由于code is data的缘故，宏是可以求值的。所有的输入数据都是list，你可以做first/reverse这样的操作。但是有一点要注意的是，宏中求得的值和代码里的值是不一样的。例如{:a inc}这样一个字面量，在宏里是可以通过:a做求值的，然后这里得到的并非一个函数（function），而是一个符号（symbol）。再者，调试宏的时候你可能会被这样的结果困惑：

(defmacro a [f] (println (:a f)))
(a {:a 1}) ; ==> prints 1
(def b {:a 1})
(a b) ; ==> prints nil

字面量可以，同值的变量就不行了。原因还是那句，宏里不能求值。

继续谈->。这个宏其实远没有你想象的那么驯服。遇到复杂一点的情况：

(def m {:a inc})
(-> 2 (get m :a) str)

这个写法对吗？str是个函数，(get m :a)返回的是inc也是个函数，貌似正确。运行之后却报错get的参数数量错误。所以千万不要忘了->是个宏，(get m :a)这里是不会求值到inc的，直接作为一个list被宏吞下去。在宏里只能通过符号的组合变化来生成代码，那么一不小心，就没有inc什么事了。

于是，你可能想到这里需要一个确切的函数，就好比这样：

(def m {:a inc})
(-> 2 (fn [x] ((get m :a) x)))

也许这样就好多了，我们放了一个匿名函数，并不要求宏去求值，因为这个匿名函数会被宏生成到新的代码里。里面的get也会在运行时求值。看似没什么问题，可是一运行还是没有期待的结果，居然返回了一个匿名函数！而对这个匿名函数求值得到的也是一个错误的结果！简直有点无厘头了。

呵呵，不故弄玄虚了。我们用macroexpand看看发生了什么。

这是用匿名函数包装以前

(macroexpand-1 '(-> 2 (get m :a)))
(get 2 m :a)

->居然只是简单地把2放到了get这个form里面！

再看看用匿名函数包装后的结果

(macroexpand-1 '(-> 2 (fn [x] ((get m :a) x))))
(fn 2 [x] ((get m :a) x))

和刚才一样，2被放到了第一个form的第一个参数位置！得到的是一个非法的form。

那么既然->只是简单地把第一个参数放到后面form的首个参数的位置，那么这个宏正确的使用方法其实是

(def m {:a inc})
(-> 2 ((fn [x] ((get m :a) x))))

再加一层括号！

(macroexpand-1 '(-> 2 ((fn [x] ((get m :a) x)))))
((fn [x] ((get m :a) x)) 2)

可见，->虽然是个功能强大的宏，但宏终归只是宏，和函数的区别是明显的。在使用的时候，不能完全按照函数的习惯。

如果你想了解实际的代码，可以参考slacker 0.3.0里的这个interceptor框架：
https://github.com/sunng87/slacker/blob/master/src/slacker/interceptor.clj
上面提到的难处，多半也都是在开发这个框架时亲身经历的。

slacker 0.2.0 is out

Posted on by
Reply

Slacker 0.2.0 has been pushed to clojars today. Connection pooling and json serialization are available in this release.

Connection Pool

Generally, pooling connection is a good idea in high concurrence application. To make slacker for real world, connection pool support is a high-prioritized feature in its development. The new connection pool is backended by commons-pool which you might familiar with. To use connection pool, just create slacker client with a new function `slackerc-pool`

(def scp (slackerc-pool "localhost" 2104))

Then you can use this pool just like a single client.

Some options are available to configure the pool by your wish:

  • :max-active, max connections opened by the pool
  • :exhausted-action
    • :fail throw an exception when pool exhausted.
    • :block block current thread and wait until max-wait exceed (throw an exception)
    • :grow automatically create new connection and add it to pool
  • :max-wait max wait time before throwing an exception
  • :min-idle minimal number of pool hold idle connections

The options are inherited from GenericObjectPool, you can find detailed information from their javadoc.

JSON Serialization

slacker just added json serialization provided by clj-json. According to my test, clj-json is 1x faster than carbonite in serialization.

(def sc (slackerc "localhost" 2104 :content-type :json))

However, with json serialization, you may lost some clojure types like keyword and set in type conversion. You should be caution when using json as serialization method.

In next release, I am planning to use fastjson as json lib which provides option to write type name into json so it could be a full featured serialization for clojure. And fastjson is claimed even faster than jackson.

Performance

slacker gains high performance with its non-blocking server, serialization and direct function call. As tested on a dual 6 core server, it reaches 10000+ TPS for a single client (50 connections, 50 threads). The server just use 35% CPU so I consider it could have even more TPS if there is two or more client machines.

So if you are interested in some benchmarks, you can test it with client like this. All the requests are using synchronous call because I believe it’s the most common case you use slacker.

Next steps

Inspired by discussion in cn-clojure mailing list, I’m going to add HTTP transport for slacker. With HTTP transport, it’s easier to debug and evaluate your clojure functions, it also makes slacker available to ClojureScript.

At lst, thanks Zach Tellman for reviewing my client code.

从GNOME网站安装exaile-doubanfm-gnome-shell-extension

Posted on by
3

最近GNOME发布了期待已久的extension.gnome.org，这个网站允许你直接通过浏览器安装和管理gnome-shell扩展，有点类似app store的感觉，混乱的~/.local/share/gnome-shell/extensions/终于有了一个官方的界面。

网站开通的第一时间，我提交了exaile-doubanfm-gnome-shell-extension，经过review和修改，这个扩展也得到了进一步的完善，适配了gnome-shell 3.2的风格。

你可以从这个地址直接安装启用
https://extensions.gnome.org/extension/24/exaile-doubanfm-control/

它会在exaile douban.fm启动后显示一个菜单在gnome-shell上，你可以通过这个菜单进行基本的操作。

如果喜欢，别忘了在extension.gnome.org上vote一下 ：）

Slacker 0.1.0 is out

Posted on by
2

Glad to roll out the first release of the slacker framework. Slacker is a clojure RPC framework on top of a TCP binary protocol. It provides a set of non-invasive APIs for both server and client. The remote invocation is completely transparent to user.

In addition to APIs introduced in last post, asynchronous approach is supported in client API :

(defremote remote-func :async true)
@(remote-func)

If you add option `:async` to defremote, then the function facade will return a promise. You have to deref it by yourself. Also you can use the `:callback` option in defremote to specify a callback function.

(defremote remote-func :callback #(println %))
(remote-func)

This gives you much more flexibility of using remote function. But be careful it will break consistency between local and remote mode.

To use slacker, add it to your project.clj

:dependencies [[info.sunng/slacker "0.1.0"]]

You can find examples on the github page.