"Slacker Cluster 0.12: Grouping"

What are Slacker and Slacker Cluster

Slacker is my side project started in late 2011. The goal of Slacker project is to provide a high performance RPC system for clojure, with elegant API. Slacker doesn't ruin your code. Your remote invocation looks exactly same as local, from code. That means you can switch back and forth at same time.

Slacker Cluster is a support module for running Slacker servers with multiple instances. Cluster enabled slacker server will publish all its served namespaces to Zookeeper cluster. The Cluster enabled client reads and watches these meta data. The most important feature of Slacker Cluster is you can add or remove servers without changing client configuration.

Grouping in Slacker Cluster

Started in 0.11, then enhanced in 0.12, Slacker Cluster now has flexible grouping choices for your scenario. In Slacker Cluster, grouping means which server(s) to call on a particular invocation.

There and four kinds of grouping for you: :random, :leader, :all and custom.

:random

By default, Slacker cluster clients use :random grouping: select a random server from server list. Random grouping works great for stateless services. It automatically balances load of each server.

:leader

Slacker servers selects leader for each namespace they expose. So at any time there will be one and only one leader node for every namespaces. The :leader grouping routes all invocations onto the leader node. This is required when your server has state, and you have to ensure the consistency and availability.

:all

As the name suggests, :all grouping routes invocations on every node at same time. In other words, it's broadcast. Note that this grouping might change your function return values. In :random and :leader mode, there's only one server called, just like local invocation. In :all, there's chances several servers are called and several values returned. I will talk about how to deal with these return values later.

Custom

You can also provide a function for dynamic grouping. For requested namespace, function and arguments, you can specify any server(s) or grouping option.

Grouping results

Grouping may break original behavior of you code by returning multiple values from multiple servers. But you still have full control over it. There are four types of value you can specify for results aggregation: :single, :vector, :map and custom function.

In short words:

  • :single returns the first valid result, and behavior same as calling single server or local invocation. This is the default value.
  • :vector returns a vector of all results.
  • :map returns a map of all results, indexed by server addresses.
  • Custom aggregation function accepts the results and allows you to merge the values.

Grouping exceptions

What happens when remote function threw exceptions? The grouping exception option defines that. When set to :all, the client will raise an error only if all remote nodes broken. Otherwise, the broken result will be ignored and only valid results will apply grouping-results rules. The opposite option is :any, which mean client will raise error when any of calls is broken.

Granularity

The grouping options can be set to client level:

(clustered-slackerc "cluster-name" "127.0.0.1:2181" :grouping :leader)

or function level:

(defn-remote sc slacker.example.api/timestamp
  :grouping :all
  :grouping-results :single)

Conclusion

Slacker Cluster has been used in our Avos Cloud backend for service integration. Feel free to let me know if you have interests or questions with this library.