Here comes the Sun » 把戏

Visualizing OpenStreetMap Nanjing Contribution

sunng — Fri, 20 Apr 2012 07:02:40 +0000

早上在prismatic上看到mapbox的一篇博客，介绍通过TileMill可视化OSM的贡献者，非常酷。于是我在南京的地图上也做了一个这样的可视化。

一个详细的大图在这里。虽然只做了南京的五个主要贡献者，基本上涵盖了大部分数据。

图例就不专门输出了
[user = 'Sunng'] { marker-fill: @magenta;}
[user = 'fuwuyuan'] { marker-fill: @blue;}
[user = 'sinopitt'] {marker-fill: @yellow;}
[user = 'larryy'] {marker-fill: @green;}
[user = 'zhengz'] {marker-fill: @red;}

MapBox家的东西真的非常酷，这家的技术以nodejs为主，围绕osm开发了不少产品。最近比较大的新闻，比如4sq转到osm上，其实就是转到这家的osm服务上。有兴趣你可以关注一下！

为WebWorker设置正确的路径

sunng — Thu, 22 Mar 2012 09:32:32 +0000

WebWorker的路径通常是写在代码源文件中，而且这个路径并非其相对父js文件的相对路径，而似乎是相对页面的路径。所以指定一个正确的可随处部署的路径变得有些麻烦。昨天有人给HeatCanvas提了这个问题我才想到上网搜索了一下，有一个还算挺不错的办法。

写一个getPath函数，从document里找到父js的路径，拼到Worker的名字上。对heatcanvas.js这个文件来说就是：

HeatCanvas.getPath = function() {
var scriptTags = document.getElementsByTagName("script");
for (var i=0; i<scriptTags.length; i++) {
var src = scriptTags[i].src;
var pos = src.indexOf("heatcanvas.js");
if (pos > 0) {
return src.substring(0, pos);
}
}
return "";
};

因此现在HeatCanvas已经解决了这个路径问题，现在这个库应该更好用了。当然如果你改了我的文件名我就无话可说了。

Visualize Reddit upvotes by subreddit

sunng — Thu, 27 Oct 2011 11:59:46 +0000

这是我reddit上所有的upvote在各个subreddit上的分布情况，这个情况还是可以说明我是个普通青年。

排在前几位的分别是

Programming
Linux
Python
f7u12
Clojure
Ubuntu

如果你还不了解什么是Reddit：Reddit是一个巨大的社会书签+论坛网站，他的频道叫做subreddit，每个频道有一个相应的主题，涵盖了从IT技术到新闻到生活的各个角落。

这些数据是通过下面的Clojure程序获得（使用reddit.clj库），并通过jfreechart展现出来的。

Substract a vector from a matrix in Octave

sunng — Fri, 21 Oct 2011 15:39:31 +0000

假设你有一个矩阵：

A = [1,2;3,4;5,6]

以及一个矢量：

B = [2,5]

你希望对A的每一行元素对元素地减B，例如第一行

[1-2,2-5]

但你不希望用循环完成这个工作，那么你可能想到利用B创建一个和A一样维度的矩阵然后进行.-：

C = [2,5;2,5;2,5]
A .- C

在Octave中可以利用repmat这个函数获得C：

C = repmat(B, length(A), 1)

但是对于大矩阵来说这是一种对内存的浪费。

更好的方法是利用bsxfun：

bsxfun(@minus, A, B)

bsxfun在octave的文档中似乎鲜有提及。

手动设置Fedora15网卡

sunng — Sat, 28 May 2011 07:16:29 +0000

今天上午的Fedora更新，版本为0.8.999.3的NetworkManager会致使NetworkManager和network service的版本不一致，从而导致网络功能无法使用。而这个时候要通过yum downgrade NetworkManager*降级又没有网络连接可用，悲剧。

这时只有手动设置网卡了。service network restart和 service NetworkManager restart都报告失败，ifconfig查看网卡，只有本地回环启动。

接下来接上网线，通过 ifconfig -a 查看所有的网卡接口，例如我的接口叫做em2.

启动网卡
ifconfig em2 up

查看网卡情况
ifconfig em2

如果没有获得ip，可以通过dhcpclient获得ip
dhclient -4 em2

再查看网卡情况，如果获得了ip，可以尝试ping www.baidu.com，如果出现network unreachable的报错，需要再配置一下路由信息，其中gw是你的网关：
route add default gw 192.168.1.1

这时你的网络应该暂时可用了，立刻执行yum downgrade NetworkManager*降级吧。

Correct source file encoding with one liner

sunng — Tue, 08 Mar 2011 13:44:09 +0000

Amoeba项目最早的代码可以追溯到2008年了，其中有多个作者贡献代码，因为一直在Windows下开发，所以没有使用UTF8编码，最近大家统一到UTF8，遇到了代码编码不正确的问题。

于是我们需要统一解决一下这个问题：

iconv -f gbk -t utf8 -o ConnectionManager.java ConnectionManager.java

这样可以把gbk编码的源文件转换为UTF8，原地转换。

推广到整个代码目录，用find和xargs做，xargs通过-I来制定一个占位符。

find . -name "*.java" -type f -perm +600 -print | xargs -I _ iconv -f gbk -t utf8 -o _ -c _

结果发现iconv运行中报了错，进一步检查发现一部分代码正常转换了，另一部分却乱码了。原来，两个作者的代码分别是gbk和gb2312，这iconv转换的时候两种编码并不兼容。这就麻烦了，必须对代码分别处理才可以，区别代码的编码，暂时就用Java源文件里的作者名字。又看了一下find似乎没有对文件内容过滤的条件，不过不要紧，我们可以用xargs做：

find . -name "*.java" -type f -perm +600 -print | xargs -I _ sh -c 'grep -q hexianmao _ && iconv -f gb2312 -t utf8 -o _ -c _ '

对这位hexianmao作者的代码，我们利用grep进程的返回值来进行判断。grep的-q参数相当于>/dev/null。这里有一个tricky的地方，再xargs里我们不能直接用&&来组合命令，不过可以通过sh -c这样的方式，并且其中的占位符会被xargs合适地替换掉。

这样执行之后，这位作者的gb2312代码就被成功转换了。而另一部分作者的gbk代码也可以用同样方式解决了。

Fix ogg/oga not play in Firefox 3.6

sunng — Tue, 15 Feb 2011 15:05:44 +0000

Native audio support was introduced in since Firefox 3.5 . Ogg is one of the media format supported by Firefox. However, sometimes you may find it doesn’t work even if you set the right source path. And you just check the network status of audio element:

document.getElementsByTagName(“audio”)[0].networkState

Then you get the constant of a 4, which is HTMLMediaElement.NETWORK_NO_SOURCE.

This is because firefox checks the Content-Type header to make sure it’s a media file. (Webkit based browsers don’t have this restriction.) However, ogg format is not configured on most http servers. You can check the content type by:
curl -I

Take Apache as example, you can add following content to your configuration file:
AddType audio/ogg .oga
AddType video/ogg .ogv .ogg

For more, check this article:
https://developer.mozilla.org/en/Configuring_servers_for_Ogg_media

The 4k Story

sunng — Fri, 08 Oct 2010 14:10:37 +0000

从 Redis 2.0 开始，Redis的作者就不断地被问道，你为什么要自己造一个VM轮子呢。尽管作者在FAQ里说明了，但是仍然有很多不同意见。

反向代理Varnish的开发人员，Poul-Henning Kamp 写了一篇文章，What’s wrong with 1975 programming ?，锋芒毕露，矛头直指竞争对手Squid，顺便也打击一大片牵连到了Redis的作者Antirez。他说：

I have spent many years working on the FreeBSD kernel, and only rarely did I venture into userland programming, but when I had occation to do so, I invariably found that people programmed like it was still 1975.

Kamp兄有来到user-space之后一夜回到解放前的感觉，又好像摇晃着饮料瓶子对着Antirez说：你Out啦！也许是因为作者就是个内核开发者，所以Varnish对操作系统的Virtual Memory机制充分信任，把Squid对内存的手动管理称为wasted work。”So Welcome to Varnish, a 2006 architecture program. ”

还有用户也提出

Redis doesn’t use OS swap. According to Salvatore Sanfilippo, the creator of Redis, it was because the page size of 4KB was too big. I personally don’t think that helps but it’d be better if Redis preallocated specified amount of buffer pool and bring related objects to the same page to increase locality of reference, instead of letting the heap manager blindly fragment objects. In my opinion, the page size of 32 bytes is too small, considering that the hardware architectures and the compilers are optimized for the conventional page size. In that scale, even the latency of reading something from RAM could be dominant (RAM is too slow for CPU, therefore it’s got L1/L2 cache), and RAM has the pipelined burst mode to pre-fetche memory contents at a few clock cycles, before they are actually requested.

5号，Antirez在博客上写了回击 What’s wrong with 2006 programming?，他认为：

OS Swap在一些情况下会导致客户端阻塞
4K大小的Page可能包含很多key，其中总有一些被访问到，导致操作系统无法swap这些page
使用自己实现的Paging为程序提供了极大的自由度，包括作者提到的2.2将会引入的数据压缩、新的数据结构以及自定义的过期算法

前段时间，Foursquare用MongoDB时，因为Sharding方法一些疏漏把大量的数据集中到了一台机器上，导致一台EC2实例内存耗尽无法工作。Mongodb的内部机制就是mmap，我的同事做过相关的测试，当内存耗尽时，读写操作都使用磁盘，这时mongodb的性能是完全无法使用的。事后10gen的开发人员Horowitz总结出现问题的原因总结出现问题的原因时，其中很重要的一点是

Document size is less than 4k. Such documents, when moved, may be too small to free up pages and, thus, memory.

看了这个原因，Redis的作者Twitter上大喜：”Real world instance of my 4k page + small objects concerns”

Visualize call tree of a C function

sunng — Fri, 24 Sep 2010 15:55:19 +0000

Requirement

You want to visualize a call hierarchy of a C function.

Solution

Utilities you need are listed below:

GNU cflow
cflow2vcg
graphviz

Take ‘rdbSaveBackground’ (redis/rdb.c) for example:

cflow --format=posix --omit-arguments --level-indent='0=\t' --level-indent='1=\t' --level-indent=start='\t' -m 'rdbSaveBackground' ~/osprojects/redis/src/rdb.c | cflow2dot | dot -Tjpg -o rdb.jpg

Output:

Source: unix diary

The post is brought to you by lekhonee v0.7

Bayeux Protocol

sunng — Sat, 17 Jul 2010 05:45:41 +0000

运行一个CometD Demo非常简单，只要创建一个Maven项目即可（CometD Howtos）：
$ mvn archetype:generate -DarchetypeCatalog=http://cometd.org

maven会提示用户选择archetype，包括cometd的版本1、版本2，jetty6、jetty7的实现，以及客户端dojo或jquery的实现。这里可以选择最新的：
http://cometd.org -> cometd-archetype-dojo-jetty7 (2.0.0 – CometD archetype for creating a server-side event-driven web application)

项目创建完成后执行mvn jetty:run即可，打开http://127.0.0.1:8080/{artifactId}即可。

CometD的协议包容了各种主要的浏览器，比如在Chromium 5上，dojo采用WebSocket实现；而在不支持WebSocket的Firefox 3上，通过long-polling实现。Bayuex是一个应用协议，CometD是Bayuex的实现，类似鸡与蛋的关系。

有了昨天在Chromium上看WebSocket协议的经验，先看一下CometD的WebSocket实现：
握手。客户端请求/{artifactId}/cometd/handshake
包含Header

GET /cometd-jetty/cometd/handshake HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: 127.0.0.1:8080
Origin: http://127.0.0.1:8080
Cookie: JSESSIONID=12jqq6hbsfkfic8vzqpevxtrw

这是标准的WebSocket握手协议，服务端返回：

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: http://127.0.0.1:8080
WebSocket-Location: ws://127.0.0.1:8080/cometd-jetty/cometd/handshake

双方完成WebSocket连接的建立。客户端通过websocket发送JSON，进行bayuex的握手：

[{"version":"1.0","minimumVersion":"0.9","channel":"/meta/handshake","supportedConnectionTypes":["websocket","long-polling","callback-polling"],”advice”:{“timeout”:60000,”interval”:0},”id”:”1″}]

服务端返回JSON，下发clientId完成握手：

[{"channel":"/meta/handshake","clientId":"8g6dbnlqr2k6jfo1tdpaeb7iw","version":"1.0","successful":true,"minimumVersion":"1.0","id":"1","supportedConnectionTypes":["websocket","long-polling","callback-polling"]}]

握手完成，bayuex连接建立。

在Demo中，客户端添加了一个handshake的listerner

function _metaHandshake(handshake)
{
if (handshake.successful === true)
{
cometd.batch(function()
{
cometd.subscribe('/hello', function(message)
{
dojo.byId('body').innerHTML += '

Server Says: ' + message.data.greeting + '

';
});
// Publish on a service channel since the message is for the server only
cometd.publish('/service/hello', { name: 'World' });
});
}
}

所以在完成握手后，客户端发送一个批量请求，subscribe /hello频道，并且向/service/hello发送json格式的消息。向/service channel发送的信息表示客户端与服务端的单独通信，不会被转发给其他客户端。
id用于区分每个请求，bayuex spec规定向/meta和/service发送的请求必须包含id字段，用于标示请求响应。
请求的内容最终聚合为一个Json

[{"channel":"/meta/subscribe","subscription":"/hello","id":"2","clientId":"8g6dbnlqr2k6jfo1tdpaeb7iw"},{"channel":"/service/hello","data":{"name":"World"},"id":"3","clientId":"8g6dbnlqr2k6jfo1tdpaeb7iw"}]

服务端发回响应，id=2的请求成功，订阅/hello频道成功

[{"channel":"/meta/subscribe","successful":true,"id":"2","subscription":"/hello"}]

之后，服务端发回/hello channel的消息

[{"channel":"/hello","data":{"greeting":"Hello, World"}},{"channel":"/service/hello","successful":true,"id":"3"}]

客户端还要定期发送连接请求保持连接

[{"channel":"/meta/connect","connectionType":"websocket","advice":{"timeout":0},"id":"4","clientId":"8g6dbnlqr2k6jfo1tdpaeb7iw"}]

服务端返回，连接成功

[{"channel":"/meta/connect","advice":{"reconnect":"retry","interval":2500,"timeout":15000},"successful":true,"id":"4"}]

connect请求是用于在客户端和服务端维持连接， Bayeux标准中提到(1, 2)：

A transport MUST maintain one and only one outstanding connect message. When a HTTP response that contains a /meta/connect response terminates, the client MUST wait at least the interval specified in the last received advice before following the advice to reestablish the connection

The client MUST maintain only a single outstanding connect message. If the server does not have a current outstanding connect and a connect is not received within a configured timeout, then the server SHOULD act as if a disconnect message has been received.

至此，cometd客户端就可以在/hello频道上订阅、发布消息了。
在Chromium上，所有的操作都在一个WebSocket连接上完成。

而当断开连接时，客户端向服务端发送

[{"channel":"/meta/disconnect","id":"188","clientId":"a8iutjvfp7dtwhzrfujeonk5q"}]

服务端响应

[{"channel":"/meta/disconnect","successful":true,"id":"188"}]

Bayuex基本上就可以理解为一个websocket上的应用协议了。

再看看Firefox 3.6上的实现。Firefox 3.6不支持WebSocket，所有的通信只能通过XHR来实现。
握手，通过一个xhr post请求实现：

POST /{artifactId}/cometd/handshake HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 115
Connection: keep-alive
Content-Type: application/json;charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://127.0.0.1:8080/{artifactId}/
Content-Length: 182
Cookie: JSESSIONID=fjnyxb28raih1cnaljrijl1ic
Pragma: no-cache
Cache-Control: no-cache

服务器端响应：

HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Set-Cookie: BAYEUX_BROWSER=df92-h8q89f416mutgbpxrwb8185u;Path=/
Content-Length: 213
Server: Jetty(7.1.5.v20100705)

[{"channel":"/meta/handshake","clientId":"9185k23lo482oq1po3ivxup2cj","version":"1.0","successful":true,"minimumVersion":"1.0","id":"1","supportedConnectionTypes":["websocket","long-polling","callback-polling"]}]

握手完成，执行客户端定义的回调。发送bayeux请求，通过一个新的XHR上
[{"channel":"/meta/subscribe","subscription":"/hello","id":"2","clientId":"9185k23lo482oq1po3ivxup2cj"},{"channel":"/service/hello","data":{"name":"World"},"id":"3","clientId":"9185k23lo482oq1po3ivxup2cj"}]

服务端同时返回三个bayuex的请求响应

[{"channel":"/meta/subscribe","successful":true,"id":"2","subscription":"/hello"},{"channel":"/hello","data":{"greeting":"Hello, World"}},{"channel":"/service/hello","successful":true,"id":"3"}]

客户端开始发送连接请求

[{"channel":"/meta/connect","connectionType":"long-polling","advice":{"timeout":0},"id":"4","clientId":"9185k23lo482oq1po3ivxup2cj"}]

注意这里使用的是long-polling方式，这是由dojo针对浏览器特性决定的。

Long-polling server implementations attempt to hold open each request until there are events to deliver; the goal is to always have a pending request available to use for delivering events as they occur, thereby minimizing the latency in message delivery.

如果没有新消息，服务端阻塞十秒后返回

[{"channel":"/meta/connect","successful":true,"id":"7"}]

客户端接收到返回立刻发起新的connect请求

当有新消息时，阻塞在服务器端的connect请求会立即返回，同时带回新的消息，如

[{"channel":"/hello","data":{"name":"555"},"id":"6"},{"channel":"/meta/connect","successful":true,"id":"619"}]

而如果是本客户端publish的新消息，会在请求成功的响应中返回，不会影响connect连接，如：

[{"channel":"/hello","data":{"name":"nihao"},"id":"715"},{"channel":"/hello","successful":true,"id":"715"}]

断开时，仍然是通过xhr post一条bayuex命令到服务端

[{"channel":"/meta/disconnect","id":"750","clientId":"9185k23lo482oq1po3ivxup2cj"}]

服务端响应：

[{"channel":"/meta/disconnect","successful":true,"id":"750"}]

至此，通过long polling方式实现bayuex的cometd客户端也描述清楚了。long-polling仍然是通过connect请求来实现pull的方式准实时，与websocket真正push的方式还是存在区别的。

The post is brought to you by lekhonee v0.7