Update on HeatCanvas and gefr

HeatCanvas的百度地图扩展

感谢@lbt05姐的无私贡献,现在HeatCanvas又增加了百度地图API支持。你可以在你的百度地图中使用heat map了。详情可以参考@lbt05姐撰写的文档,而这里是一个简单的live demo。此外,@lbt05姐还贡献了GoogleMap扩展的patch,帮助我解决了地图拖动后Canvas无法覆盖viewport的bug。再次感谢@lbt05姐。

百度地图支持已经汇入主干,可以在github找到它。

Gefr新增 Jetty WSGI Bridge

昨天为gefr新增了Jetty服务器的支持,这样你可以将自己的Python WSGI程序运行在成熟的Jetty服务器上。性能是大家关心的因素,下面是在我的本机(32位CentOS,双核2.8GHz,4G内存)上一个粗略的测试结果:

100并发,20000请求:

Server Software: gefr-jetty/0.3dev
Server Hostname: localhost
Server Port: 8088

Document Path: /
Document Length: 19 bytes

Concurrency Level: 100
Time taken for tests: 3.120827 seconds
Complete requests: 20000
Failed requests: 0
Write errors: 0
Total transferred: 2584515 bytes
HTML transferred: 380665 bytes
Requests per second: 6408.56 [#/sec] (mean)
Time per request: 15.604 [ms] (mean)
Time per request: 0.156 [ms] (mean, across all concurrent requests)
Transfer rate: 808.44 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 114.1 0 3000
Processing: 0 7 18.5 5 1776
Waiting: 0 6 18.5 4 1776
Total: 0 12 115.8 6 3015

Percentage of the requests served within a certain time (ms)
50% 6
66% 7
75% 8
80% 8
90% 10
95% 13
98% 70
99% 74
100% 3015 (longest request)

如果打开KeepAlive,吞吐量可以达到10000以上:

Server Software: gefr-jetty/0.3dev
Server Hostname: localhost
Server Port: 8088

Document Path: /
Document Length: 19 bytes

Concurrency Level: 100
Time taken for tests: 1.749189 seconds
Complete requests: 20000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 20000
Total transferred: 3062142 bytes
HTML transferred: 380266 bytes
Requests per second: 11433.87 [#/sec] (mean)
Time per request: 8.746 [ms] (mean)
Time per request: 0.087 [ms] (mean, across all concurrent requests)
Transfer rate: 1709.36 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 23
Processing: 0 8 8.0 6 85
Waiting: 0 8 8.0 6 85
Total: 0 8 8.1 6 85

Percentage of the requests served within a certain time (ms)
50% 6
66% 7
75% 8
80% 9
90% 11
95% 20
98% 41
99% 43
100% 85 (longest request)

你可以根据gefr的文档安装它。

Gefr with multiple backends support

I have been working on gefr this week to add apache MINA backend for it. The goal of gefr is to provide adapters between Python’s WSGI and Java network infrastructures. In cPython world, most WSGI servers are implemented with C code(fapws, meinheld, etc.), which is not applicable on Jython platform. However, the Java world has its own reliable network foundations. So my gefr turns them into WSGI servers. The ultimate goal of gefr is to make it possible to port any of your WSGI web applications to Jython seamlessly.

There has been many attempts to use Jython in Java web development. Most of them is focusing on applying Jython in Servlet framework. This is helpful, but it doesn’t take the whole advantage of Python web programming. The most valuable thing in Python web development is its various web frameworks and agile tools. But inside servlet framework, we can hardly apply them, the only things we have is Jython’s beautiful syntax. That should not be enough.

OK. You are eager to meet gefr? Let’s get hands dirty with some setup. I just assume that you have some essential software installed in the right place. Is it true for you?

Create Jython virtual environment:

$ virtualenv -p /usr/local/bin/jython gefr-test
$ cd gefr-test
$ source bin/activate

Install jip. jip is a great tool to manage Java dependencies for your Jython application. jip>=0.3 is required in this guide, and pip will handle this for you.
$ pip install jip

Install gefr. Since gefr is a pure Python package, pip is still helpful.
$ pip install gefr

Now create a jip configuration file called .jip in your virtual home. We should add sonatype oss repository.

[repos:oss]
uri=http://oss.sonatype.org/content/repositories/snapshots/
type=remote

[repos:central]
uri=http://repo1.maven.org/maven2/
type=remote

[repos:local]
uri=/home/sun/.m2/repository/
type=local

New in jip 0.3, we can use a single command to resolve dependencies of gefr.
$ jip install-dependencies info.sunng.gefr:gefr:0.2-SNAPSHOT

OK. Then everything is ready. Create your first Jython WSGI application. Actually, there is no difference.
I just copy the example of gefr here, you can find it in the code repository.

#! /usr/bin/python

__author__="Sun Ning <classicning@gmail.com>"
__date__ ="$Jan 5, 2011 10:14:27 PM$"

import sys
from gefr.core import Gefr

def simple(environ, start_response):
    """a static app"""
    headers = []
    headers.append(('Content-Type', "text/html"))
    start_response("200 OK", headers)

    return ["<h1>it works.</h1>"]

if __name__ == "__main__":
    from optparse import OptionParser
    parser = OptionParser()
    parser.add_option('-b', '--backend', dest='backend', help='server backend')
    (options, args) = parser.parse_args()

    if options.backend == 'soldat':
        from gefr.backends.soldat import SoldatBackend as backend
    elif options.backend == 'mina':
        from gefr.backends.mina import MinaBackend as backend
    else:
        print 'backend %s not supported.' % options.backend
        sys.exit(1)

    g = Gefr(simple, backend)
    g.start()

With the command line options, you can control which backend to be used. Now start it with:
$ jython-all example.py -b soldat
or
$ jython-all example.py -b mina

Soldat is another network framework written by myself. And MINA, you must know. Open your browser, redirect it to http://localhost:8080 , now you see it works.

Sure, that’s not enough for your needs. Let’s have an advanced example. Among all the web frameworks in Python, I like bottle most. So we just build a simple application with GET and POST in bottle.

$ pip install bottle
$ pip install mako
$ mkdir bottleapp
$ cd bottleapp

The bottle application looks like:
echo.py

# -*- coding:utf-8 -*-

from bottle import Bottle, mako_view, request
from gefr.core import Gefr
from gefr.backends.mina import MinaBackend

bapp = Bottle()

@bapp.get('/')
@mako_view('edit')
def display_form():
    return dict(ip=request.environ.get('REMOTE_ADDR'))

@bapp.post('/')
@mako_view('show')
def display_content():
    content = request.POST.get('content')
    return dict(content=content)

Gefr(bapp, MinaBackend).start()

The simple mako templates are:

edit.tpl

<html>
<head>
    <title>Bottle on MINA</title>
</head>
<body>
    <h1>Bottle on MINA</h1>
    <p>User from ${ip}, leave your words please:</p>
    <form action="/" method="post">
        <textarea cols="60" rows="20" name="content"></textarea>
        <p><input type="submit" /></p>
    </form>
</body>
</html>

show.tpl

<html>
    <head>
        <title>Bottle on MINA</title>
    </head>
    <body>
        <h1>Bottle on MINA</h1>
        <pre>${content | h}</pre>

    </body>
</html>

Start the bottle app:
$ jython-all echo.py

This is rather simple but it covers the most basic features of a web server.

I should say thank you for reaching here. Gefr is still an experimental project in early development. So there must be bugs and potential performance improvements in it. Feel free to contact me if you are also interested in it. The project is hosted on bitbucket:
https://bitbucket.org/sunng/gefr

gefr API updates

早上收到jythonet作者的一封邮件,受到启发,我打算扩展一下gefr这个WSGI adapter。原本gefr是写给soldat用来测试的,结果发现现在出现了买椟还珠的效果,既然大家更加关注这个,我决定多花一些时间在上面。不过话说回来gefr确实是Jython Web程序的新思路,过去大家都在想怎样把Jython放进Servlet中,gefr的思路是把Java的服务器实现放到WSGI后面。未来会有gefr-netty, gefr-mina, gefr-servlet等等出现,如果真的存在一个可靠的backend,也许这种思路也不失为一个办法,毕竟python的web框架更吸引人,而java的基础设施相对可靠,各取所长。

根据这个目的,现在gefr 0.2的API更新为:

即在创建Gefr的时候需要告知backend类型,目前还只有soldat一种后端。未来也可能通过自动检测让这个参数成为可选。

gefr 0.2已经在开发中,您可以通过这里关注:
https://bitbucket.org/sunng/gefr/overview

soldat & gefr

我的这一套stack正在走向完整。上次贴了一张soldat-http的图,现在基于soldat的wsgi服务器也已经有了一个基本可以运行的实现,名字叫做gefr(我的命名出处参考这里)。现在gefr上已经可以跑基于bottle框架的wsgi程序了,也就是说一些基于python的web应用可能可以通过jython来运行在soldat上。为了搞定jython的环境,这几天我还花了不少时间做了jip帮我从maven仓库里自动下载依赖的jar包。

soldat和gefr的代码都放在我的bitbucket上:

此外,这两个项目也分别发布到了sonatype oss仓库python cheese shop

现在还有几个问题:

  • soldat在读buffer的时候先获得buffer的limit,再去读相应长度的buffer有时会出现BufferUnderflowException。这个可能存在线程安全问题,现在还没发现。
  • gefr启动之后可以通过在jvisualvm里找到这个进程,但是绑定profiler之后很诡异的是gefr就不再处理请求了。
  • 直接用soldat的处理http请求,吞吐量可以上万;但是在上面加上jython的gefr,再加上bottle框架,同样的功能吞吐量就剩下原来的十分之一了。就是因为没法做profile,所以还不知道时间花到哪里去了。

简单地 announce 一下,这样我有更多的动力来继续把这两个小东西做好。