soldat & gefr

我的这一套stack正在走向完整。上次贴了一张soldat-http的图,现在基于soldat的wsgi服务器也已经有了一个基本可以运行的实现,名字叫做gefr(我的命名出处参考这里)。现在gefr上已经可以跑基于bottle框架的wsgi程序了,也就是说一些基于python的web应用可能可以通过jython来运行在soldat上。为了搞定jython的环境,这几天我还花了不少时间做了jip帮我从maven仓库里自动下载依赖的jar包。

soldat和gefr的代码都放在我的bitbucket上:

此外,这两个项目也分别发布到了sonatype oss仓库python cheese shop

现在还有几个问题:

  • soldat在读buffer的时候先获得buffer的limit,再去读相应长度的buffer有时会出现BufferUnderflowException。这个可能存在线程安全问题,现在还没发现。
  • gefr启动之后可以通过在jvisualvm里找到这个进程,但是绑定profiler之后很诡异的是gefr就不再处理请求了。
  • 直接用soldat的处理http请求,吞吐量可以上万;但是在上面加上jython的gefr,再加上bottle框架,同样的功能吞吐量就剩下原来的十分之一了。就是因为没法做profile,所以还不知道时间花到哪里去了。

简单地 announce 一下,这样我有更多的动力来继续把这两个小东西做好。

jip 0.1

The original idea is to create a standalone jython environment, I took traditional Java tools, ant and ivy, to resolve Java dependencies. But the XMLs seem to be verbose and not pythonic. So I decide to create something like pip, that is, resolves and installs dependencies in a pythonic way. It is jip.

jip will automatically download jars and its non-optional runtime dependencies from Maven repositories. By default, jip will search your local repository and maven central repository for the requested artifact. Also, you can create a configuration file to overwrite this.

Virtualenv is required by jip. You must run jip within a standalone environment, created by virtualenv:
virtualenv -p /usr/local/bin/jython jython-env
cd jython-env
source bin/activate

Don’t forget to activate it.

Then download jip with pip:
pip install jip

Now you have pip for python and jip for java. To install a Java package, just type:
jip install <groupId>:<artifactId>:<version>

groupId+artifactId+version is known as the coordinate of a maven artifact. For example, you need spring-core in your jython development:
jip install org.springframework:spring-core:3.0.5.RELEASE

The jars will be stored in javalib directory:
ls javalib

commons-logging-1.1.1.jar     spring-core-3.0.5.RELEASE.jar
spring-asm-3.0.5.RELEASE.jar

And When you installed jip, I will provide you a jython-all command to include dependencies by default. So use jython-all instead of jython to run your program and the shell.

For traditional Java user, there is a resolve subcommand to download dependencies defined in a pom file. This is more maintainable, some of you may prefer this way to typing it one by one.
jip resolve pom.xml

To define custom repositories, place a dot file .jip in your home directory:

[jboss]
uri=http://repository.jboss.org/maven2/
type=remote

[local]
uri=/home/sun/.m2/repository/
type=local

[central]
uri=http://repo1.maven.org/maven2/
type=remote

You may have internal Nexus, just append to this file following the pattern.

Finally, a clean subcommand to remove everything you downloaded.

That’s all. You can find the project at:

Your feedback is appreciate. Fire an issue in github if you find any bugs or new ideas about this tool.

Setup a Jython development environment

This article will show you some basic steps to setup a standalone environment for Jython, with the power of some popular and standard tools.

0. Install essential tools

Take Ubuntu Maverick as an example, before starting, you have to make sure these packages installed on your system.

Essential Python tools (Python is shipped with most Linux distribution by default)
sudo apt-get install python-pip python-virtualenv

Java
sudo apt-get install openjdk-6-jdk openjdk-6-jre

Jython
The Jython package in Ubuntu repository is obsolete. Download Jython installer from official website. A full installation is recommended here.
http://jython.org/downloads.html

As an optional step, I often move Jython direcotry to /usr/local/share/ , and use a symbol link to add Jython to PATH:
sudo ln -s /usr/local/share/jython/bin/jython /usr/local/bin/

Install additional Java tools
sudo apt-get install ant ivy

1. Create a standalone environment

Jython 2.5 is fully compatible with virtualenv, so you can create a standalone environment just like what you do with python:
virtualenv -p /usr/local/bin/jython jython-env
cd jython-env

List the directory, you will see these files and directories:

  • bin/
  • cachedir/
  • Lib/
  • jython.jar
  • registry

Source the activate script:
source bin/activate

2. Download and install python packages

Now the python environment is just ready, you can install python packages as you want. Install bottle, for example:
pip install bottle

3. Download and install Java dependencies

This is the different step. As you know, the most important feature of Jython is availability of Java libraries to Python code. So we need some additional tools to manage Java dependencies. Popular Java lifecycle management tool Maven is not suitable here, because Maven is tightly depends on its archetype. Gradle is also known as a powerful build tool with DSL support. However, Gradle uses Groovy as the scripting language to create build file. Bringing Groovy to a Jython project seems terribly strange.

So I prefer the traditional Ant way to manage dependencies. First of all, create a build file with following content:
build.xml

<project xmlns:ivy="antlib:org.apache.ivy.ant" name="jython-env" default="resolve">
    <target name="resolve" description="retrieve dependencies">
        <ivy:retrieve pattern="javalib/[artifact]-[revision].[ext]" type="jar"/>
    </target>
</project>

Then create an ivy xml to configure dependencies:
ivy.xml

<ivy-module version="2.0">
    <info organisation="info.sunng" module="jython-env" />
    <dependencies>
        <dependency org="commons-lang" name="commons-lang" rev="2.5"/>
    </dependencies>
</ivy-module>

Also, as an optional step, you can overwrite default settings to configure ivy to use maven local repository.
ivysettings.xml

<ivy-settings>
    <settings defaultResolver="maven-or-not"/>
    <resolvers>
        <chain name="maven-or-not" returnFirst="true">
            <filesystem name="maven-local" m2compatible="true" />
            <ibiblio name="ibiblio" m2compatible="true" />
        </chain>
    </resolvers>
</ivy-settings>

All done, now you can download everything. In Ubuntu, default ivy installation, you have to specify ivy path in ant command line:
ant -lib /usr/share/java/ivy.jar

You will see jars in javalib.

4. Reconfigure start up script

The default Jython start up script is not friendly to external Java dependencies. To include jars, you have to use such dirty command line parameters:
jython -Dpython.path=javalib/commons-lang-2.5.jar:javalib/…

To get rid of this, first rename bin/jython to bin/startJython. (Follow the Groovy naming convention)

Then create a new bin/jython script wraps the old one:

#! /bin/bash

JYTHON_CMD="startJython"
JAVA_LIBS="javalib/*.jar"
PYTHON_PATH=""

for lib in $JAVA_LIBS
do
    PYTHON_PATH="$PYTHON_PATH:$lib"
done

#echo $PYTHON_PATH
$JYTHON_CMD -Dpython.path=$PYTHON_PATH

5. Run

Finally, it comes to an end. Type jython to execute the interactive shell:
jython

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_20
Type “help”, “copyright”, “credits” or “license” for more information.
>>> from org.apache.commons.lang import BitField
>>> import bottle

Both Python and Java dependency is accessible. Now it’s time to get your new start.

dbus-python, in case you don't know

上周用dbus-python写exaile插件,实现mpris2.0,非常痛苦,因为几乎没有完整的文档,只有一个tutorials帮助你入门。总结一下我这次quick and dirty的开发所掌握的资料,当然,因为没有太多时间,所以也没有对dbus做更深入的了解,仅仅是in case you don’t know。

DBus Interface Properties

对mpris 2.0中定义的properties用dbus-python应该如何实现呢,似乎没有@dbus.service.properties这种decorator啊。这里的文档对Properties进行了简单的说明,对于python程序,只能这样实现了:

    @dbus.service.method(dbus.PROPERTIES_IFACE, in_signature='ss', out_signature='v')
    def Get(self, interface, prop):
        ...
    @dbus.service.method(dbus.PROPERTIES_IFACE, in_signature='ssv')
    def Set(self, interface, prop, value):
        ...
    @dbus.service.method(dbus.PROPERTIES_IFACE, in_signature='s', out_signature='a{sv}')
    def GetAll(self, interface):
        ...

程序里通过prop判断property名字,返回结果。没办法,就是这样。

指定返回数据的dbus类型

对于dbus接口返回的variant类型,dbus-python会guess_signature。可是这部分的功能有的时候并不如我们想象,比如返回一个dict类型,其中的value有字符串有list有int,而out_signature是v。这种情况在Property Get的时候非常常见,如果不能手动指定类型,guess_signature得到的结果,value是统一类型,即,int会报错,如果没有int类型,会将所有字符串转成list(dbus的array)

这种情况需要手动指定返回的类型,用dbus.types模块下的类型对python显示包装,对dict类型,需要用dbus.types.Dictionary()包装,并且在构造函数里传入signature=’sv‘及相应的variant_level即可显示告知类型。同样,对list对象,需要用dbus.types.Array()包装。

这样即可绕过guest_signature,如果用dbus-monitor查看,就可以看到类型为variant string了。

Exaile 豆瓣电台插件 0.0.5

趁热依旧打铁。Exaile 豆瓣电台插件 0.0.5

Features

  • 界面快捷键 加心f 跳过s 删除d
  • 调整了按钮的顺序,和官方一致
  • 增强的菜单,可以在豆瓣模式里切换频道、关闭Exaile,无须回到主界面
  • 推荐当前歌曲到豆瓣,可以在配置界面配置推荐语模板

Installation

Wiki page

Downloads

Download from Github

Screenshots

Douban Mode
screenshot_002

Enhanced Douban Mode Menu
screenshot_001

More ?

豆瓣 on Ubuntu