Squealer (a test framework for Pig script) is using jip

Squealer is a framework written in Jython to test your Apache Pig scripts, by Mark Roddy. It’s now using jip to resolve Java dependencies. On huge dependencies of Hadoop, jip could be great helpful to setup Squealer.

To get started, it is recommended to create a standalone Jython environment for Squealer:
virtualenv -p /usr/local/bin/jython –no-site-packages squealer-env

Activate the environment
cd squealer-env
. bin/activate

Install jip with pip
pip install jip

Download squealer from bitbucket project page, extract it to somewhere. Install it with jip:
jython setup.py install

Dependencies will be downloaded from Maven Central. You just wait for it to finish.

Start a Jython interpreter with ‘jython-all‘ and now you can import squealer.

Update on HeatCanvas and gefr

HeatCanvas的百度地图扩展

感谢@lbt05姐的无私贡献,现在HeatCanvas又增加了百度地图API支持。你可以在你的百度地图中使用heat map了。详情可以参考@lbt05姐撰写的文档,而这里是一个简单的live demo。此外,@lbt05姐还贡献了GoogleMap扩展的patch,帮助我解决了地图拖动后Canvas无法覆盖viewport的bug。再次感谢@lbt05姐。

百度地图支持已经汇入主干,可以在github找到它。

Gefr新增 Jetty WSGI Bridge

昨天为gefr新增了Jetty服务器的支持,这样你可以将自己的Python WSGI程序运行在成熟的Jetty服务器上。性能是大家关心的因素,下面是在我的本机(32位CentOS,双核2.8GHz,4G内存)上一个粗略的测试结果:

100并发,20000请求:

Server Software: gefr-jetty/0.3dev
Server Hostname: localhost
Server Port: 8088

Document Path: /
Document Length: 19 bytes

Concurrency Level: 100
Time taken for tests: 3.120827 seconds
Complete requests: 20000
Failed requests: 0
Write errors: 0
Total transferred: 2584515 bytes
HTML transferred: 380665 bytes
Requests per second: 6408.56 [#/sec] (mean)
Time per request: 15.604 [ms] (mean)
Time per request: 0.156 [ms] (mean, across all concurrent requests)
Transfer rate: 808.44 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 114.1 0 3000
Processing: 0 7 18.5 5 1776
Waiting: 0 6 18.5 4 1776
Total: 0 12 115.8 6 3015

Percentage of the requests served within a certain time (ms)
50% 6
66% 7
75% 8
80% 8
90% 10
95% 13
98% 70
99% 74
100% 3015 (longest request)

如果打开KeepAlive,吞吐量可以达到10000以上:

Server Software: gefr-jetty/0.3dev
Server Hostname: localhost
Server Port: 8088

Document Path: /
Document Length: 19 bytes

Concurrency Level: 100
Time taken for tests: 1.749189 seconds
Complete requests: 20000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 20000
Total transferred: 3062142 bytes
HTML transferred: 380266 bytes
Requests per second: 11433.87 [#/sec] (mean)
Time per request: 8.746 [ms] (mean)
Time per request: 0.087 [ms] (mean, across all concurrent requests)
Transfer rate: 1709.36 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 23
Processing: 0 8 8.0 6 85
Waiting: 0 8 8.0 6 85
Total: 0 8 8.1 6 85

Percentage of the requests served within a certain time (ms)
50% 6
66% 7
75% 8
80% 9
90% 11
95% 20
98% 41
99% 43
100% 85 (longest request)

你可以根据gefr的文档安装它。

jip.embed: On-the-fly classpath resolution for Jython

jip 0.7 introduces a module called jip.embed, which allows you to add libraries to your code in the runtime as you declare them. With jip.embed, you don’t have to download jars manually and append them to your -Dpython.path. You just pick your editor, import jip.embed, code your business, then save and run it.

Code example:

import jip.embed
jip.embed.require('commons-lang:commons-lang:2.6')
from org.apache.commons.lang import StringUtils

print StringUtils.reverse('jip rocks')

Output:

skcor pij

(jip will print some log here if dependencies are included for first time)

So please check out my new released jip, 0.7:
https://github.com/sunng87/jip

For virtualenv user, you can install full-featured jip via pip:
$ pip install jip

To install jip globally, download the package from python cheese shop and run:
$ jython setup.py install

jip 0.5.1 released

I just rolled out jip 0.5.1 as a bugfix version of 0.5. It has been published to pypi and you can install it with pip install jip in your virtualenv.

From 0.5, you have new features available:

  • User-Agent ‘jip/0.5‘ is added to http request.
  • New command `freeze` just like pip.
  • Improved jar downloading. By default, at most 3 jars are downloaded in parall.

In 0.5.1, bugs were fixed:

  • Repositories defined in pom are now included for install command.
  • Placeholder resolving of #{pom.groupId} is corrected. (instance)
  • urllib2.URLError is caught to prevent dump.

For any problem and feature request, please refer to github issue tracker.

Convert a Python function to Java anonymous class

When calling Java with Jython, anonymous inner class might be an issue because there is no such equivalent in Python.

In GUI programming, jython made additional effort on AWT event processing. You can pass a python function as some types of event listener.

def change_text(event):
    print 'Clicked!'

button = JButton('Click Me!', actionPerformed=change_text)
frame.add(button)

Described in the Definitive Guide of Jython:

This works because Jython is able to automatically recognize events in Java code if they have corresponding addEvent()* and *removeEvent() methods. Jython takes the name of the event and makes it accessible using the nice Python syntax as long as the event methods are public.

However, it does not work with Runnable, Callable and many other interfaces designed for anonymous usage.

To solve the incompatibility, I created a small decorator that converts Python function to a Java object.

Thanks to python’s dynamic magic, we can create class in runtime and assign modified method to a particular instance. The conversion is performed once the function loaded. Also, you can pass something as arguments to constructor.

With anonymous_class decorator, the example above can be written as:

from javax.swing import JButton, JFrame
from java.awt.event import ActionListener

frame = JFrame('Hello, Jython!',
            defaultCloseOperation = JFrame.EXIT_ON_CLOSE,
            size = (300, 300)
        )

@anonymous_class(ActionListener, "actionPerformed")
def change_text(dummy, event):
    print 'Clicked!'

button = JButton('Click Me!', actionPerformed=change_text)
frame.add(button)
frame.visible = True