Don't repeat yourself: distribute jython package with jip.dist

As a new feature in jip 0.4, we can use some helpers from jip.dist to simplify package distribution. With jip.dist, you can define Java dependencies for your jython package. In an environment with jip, dependencies will be automatically installed when user uses pip to get you package.

We have two different approaches allow you to choose.

Approach 1, Define dependencies in POM

This is the standard maven way. To describe your jython package and its dependencies, create a pom.xml in your project. The directory hierarchy looks like:

├── app
│   ├── module1
│   │   ├── __init__.py
│   ├── core.py
│   ├── __init__.py
├── LICENSE
├── MANIFEST.in
├── pom.xml
├── README
└── setup.py

In pom.xml, just add dependencies as you do with Maven.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" <br="" />  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;

    ...

    <dependencies>
        <dependency>
            <groupid>org.slf4j</groupid>
            <artifactid>slf4j-api</artifactid>
            <version>1.6.1</version>
        </dependency>

        <dependency>
            <groupid>org.slf4j</groupid>
            <artifactid>slf4j-log4j12</artifactid>
            <version>1.6.1</version>
        </dependency>

        ...

    </dependencies>

    <repositories>
        <repository>
            <id>sonatype-oss-sonatype</id>
            <url>http://oss.sonatype.org/content/repositories/snapshots/</url>
        </repository>
    </repositories>
</project>

You can also define repositories in pom.xml if you use custom repository. (for example, jboss, java.net)

Remember to add pom.xml in your MANIFEST.in, to ensure the file will be packaged into source package:

include pom.xml

Approach 2, Define dependencies with Python

You may be tired with endless XML configuration. jip allows you to define dependencies with python, just like gradle with groovy.

In your setup.py, add something like:

requires_java = {
    'dependencies':[
        ## (groupdId, artifactId, version)
        ('org.slf4j', 'slf4j-api', '1.6.1'),
        ('org.slf4j', 'slf4j-log4j12', '1.6.1'),
        ('info.sunng.soldat', 'soldat', '1.0-SNAPSHOT'),
        ('org.apache.mina', 'mina-core', '2.0.2')
    ],
    'repositories':[
        ('sonatype-oss-snapshot', 'http://oss.sonatype.org/content/repositories/snapshots/')
    ]
}

Then pass it to setup(). The keyword argument require_java is jip specific.

setup(
    ...
    requires_java=requires_java,
    ...)

Use jip’s setup wrapper

To use jip’s power, the only difference is to use setup() from jip.dist instead of setuptools or distutils.

from jip.dist import setup

Then publish your jython package to Python Cheese Shop:
$ jython setup.py sdist upload

Internally, jip uses setuptools. So you can still do jython setup.py develop .

And jip 0.4 is available under MIT License. You are free to use jip.dist in your code.

For your user

You should write a guide forcing users to use your jython application within virtualenv. And install jip as a prerequisite:
$ pip install jip

Then simply install your package with pip:
$ pip install <your-package-name>

No additional step required!

So please just release your jython package with jip !

For more information:

Enhanced jip to simplify Jython module distribution

As you might notice, the installation of gefr is too complex, requiring several manual actions. User have to remember the long maven coodinator to resolve dependencies. wtf!

So I have been working whole day to simplify the process. 

Currently, gefr’s approach is to upload pom.xml to a public maven repositoy (sonatype oss). And the source is uploaded to pypi. Pip will find the source and install it. Jip will find the plain pom and resolve it. Because once pip finished the installation, the source package will be erased and jip can never find the pom. So I have to distribute them seperately.

It would be better to invoke jip right before pip exits. I did some investigation about post-install script. And lucky enough, distutils allows you to override default install command. We can use it to invoke jip, in the scope of setup scrript. It does make sense.

Another problem is the original design of .jip configuration file. From jip 0.2, .jip is available as environmen-scoped: we define some custom repositories and the whole environment shares them. But if we have multiple projects in the same environment, jip may waste time to find public artifact in a private repository. Even worse, jip may load invalid artifact from private repos. In the new design, private repositories are defined in pom.xml, as project-scoped, just like most java build tools.

With new jip, to distribute a jython package, you should write a pom in the same way of Java, specifying the dependencies and custom repositories if you have. Then modify your MANIFEST.in to include it to your source package. At last, in the setup.py, define a new install command to call jip and pass the command into the setup() . From the new approach, only ‘pip install’ is required for end user. Super easy!

Upon the new usage of jip(as a library), I am also considering to migrate from GPL to LGPL or MIT. I have little knowledge about conflicts between licenses. So if you have some ideas or concerns, feel free to let me know.

Gefr with multiple backends support

I have been working on gefr this week to add apache MINA backend for it. The goal of gefr is to provide adapters between Python’s WSGI and Java network infrastructures. In cPython world, most WSGI servers are implemented with C code(fapws, meinheld, etc.), which is not applicable on Jython platform. However, the Java world has its own reliable network foundations. So my gefr turns them into WSGI servers. The ultimate goal of gefr is to make it possible to port any of your WSGI web applications to Jython seamlessly.

There has been many attempts to use Jython in Java web development. Most of them is focusing on applying Jython in Servlet framework. This is helpful, but it doesn’t take the whole advantage of Python web programming. The most valuable thing in Python web development is its various web frameworks and agile tools. But inside servlet framework, we can hardly apply them, the only things we have is Jython’s beautiful syntax. That should not be enough.

OK. You are eager to meet gefr? Let’s get hands dirty with some setup. I just assume that you have some essential software installed in the right place. Is it true for you?

Create Jython virtual environment:

$ virtualenv -p /usr/local/bin/jython gefr-test
$ cd gefr-test
$ source bin/activate

Install jip. jip is a great tool to manage Java dependencies for your Jython application. jip>=0.3 is required in this guide, and pip will handle this for you.
$ pip install jip

Install gefr. Since gefr is a pure Python package, pip is still helpful.
$ pip install gefr

Now create a jip configuration file called .jip in your virtual home. We should add sonatype oss repository.

[repos:oss]
uri=http://oss.sonatype.org/content/repositories/snapshots/
type=remote

[repos:central]
uri=http://repo1.maven.org/maven2/
type=remote

[repos:local]
uri=/home/sun/.m2/repository/
type=local

New in jip 0.3, we can use a single command to resolve dependencies of gefr.
$ jip install-dependencies info.sunng.gefr:gefr:0.2-SNAPSHOT

OK. Then everything is ready. Create your first Jython WSGI application. Actually, there is no difference.
I just copy the example of gefr here, you can find it in the code repository.

#! /usr/bin/python

__author__="Sun Ning <classicning@gmail.com>"
__date__ ="$Jan 5, 2011 10:14:27 PM$"

import sys
from gefr.core import Gefr

def simple(environ, start_response):
    """a static app"""
    headers = []
    headers.append(('Content-Type', "text/html"))
    start_response("200 OK", headers)

    return ["<h1>it works.</h1>"]

if __name__ == "__main__":
    from optparse import OptionParser
    parser = OptionParser()
    parser.add_option('-b', '--backend', dest='backend', help='server backend')
    (options, args) = parser.parse_args()

    if options.backend == 'soldat':
        from gefr.backends.soldat import SoldatBackend as backend
    elif options.backend == 'mina':
        from gefr.backends.mina import MinaBackend as backend
    else:
        print 'backend %s not supported.' % options.backend
        sys.exit(1)

    g = Gefr(simple, backend)
    g.start()

With the command line options, you can control which backend to be used. Now start it with:
$ jython-all example.py -b soldat
or
$ jython-all example.py -b mina

Soldat is another network framework written by myself. And MINA, you must know. Open your browser, redirect it to http://localhost:8080 , now you see it works.

Sure, that’s not enough for your needs. Let’s have an advanced example. Among all the web frameworks in Python, I like bottle most. So we just build a simple application with GET and POST in bottle.

$ pip install bottle
$ pip install mako
$ mkdir bottleapp
$ cd bottleapp

The bottle application looks like:
echo.py

# -*- coding:utf-8 -*-

from bottle import Bottle, mako_view, request
from gefr.core import Gefr
from gefr.backends.mina import MinaBackend

bapp = Bottle()

@bapp.get('/')
@mako_view('edit')
def display_form():
    return dict(ip=request.environ.get('REMOTE_ADDR'))

@bapp.post('/')
@mako_view('show')
def display_content():
    content = request.POST.get('content')
    return dict(content=content)

Gefr(bapp, MinaBackend).start()

The simple mako templates are:

edit.tpl

<html>
<head>
    <title>Bottle on MINA</title>
</head>
<body>
    <h1>Bottle on MINA</h1>
    <p>User from ${ip}, leave your words please:</p>
    <form action="/" method="post">
        <textarea cols="60" rows="20" name="content"></textarea>
        <p><input type="submit" /></p>
    </form>
</body>
</html>

show.tpl

<html>
    <head>
        <title>Bottle on MINA</title>
    </head>
    <body>
        <h1>Bottle on MINA</h1>
        <pre>${content | h}</pre>

    </body>
</html>

Start the bottle app:
$ jython-all echo.py

This is rather simple but it covers the most basic features of a web server.

I should say thank you for reaching here. Gefr is still an experimental project in early development. So there must be bugs and potential performance improvements in it. Feel free to contact me if you are also interested in it. The project is hosted on bitbucket:
https://bitbucket.org/sunng/gefr

jip 0.2 released

As you may know, jip is a dependency management tool for Jython/Java development. It resolves and downloads Java packages from maven-compatible repositories. jip also follows some best practices, encouraging you to use a portable and standalone environment (virtualenv) for your Jython development.

It has been four months since the initial release. In version 0.2, I made following changes for you:

  • Improved console output format
  • Correct scope dependency management inheritance
  • Snapshot management, alpha
  • Environment independent configuration
  • Bug fixes

You can find the typical usage doc on github:
https://github.com/sunng87/jip

jip-0.2 has been published to pypi so you can install or upgrade it with:
easy_install -U jip

Please do remember to use it inside virtualenv!

南京的第一个周末

回了南京,最大的变化就是心里踏实多了,从上大学以后几乎就没有再在家里好好呆过。虽然年纪轻轻就告老还乡有点不太给力,但是这样实在是比在外乡遥遥无期地飘着感觉要好(飘着好,遥遥无期地飘着不好)。

上班的地方在江宁,你可以从sunng.info页面的location标签上找到我。那个园区应该是聚集了一些软件公司,不过规模比显然比不上浦东软件园(还一二三期)。当然啦,规模大有个毛用,稍微晚几分钟食堂还要排队,快算了吧。

交通比想象的要方便,换一次公交车就从家门口到园区门口了。即使是早高峰时间,也只有在市区的一条路上稍微堵一下,大部分时间都是飞驰的专车,夫复何求!!换车的地方在未来的京沪高铁附近,将来南京南站也建在那一带,施工的时候还是尘土飞扬的,这点比张江要差点。(嗯,也就跟张江比比吧)

小公司有小公司的好处,新人去了没有繁文缛节的洗脑。开发岗位的直接用linux桌面,我把它看作一种福利,虽然是个rhel5,但好歹有自己配置的余地。当然即使这样,Windows还是少不了,Office是主要的原因,不过现在虚拟机是它永恒的家了。其他嘛,印象深刻的是公司的OpenDNS规则配得很厉害,好在firefox有个配置项叫做network.proxy.socks_remote_dns,你懂的。

周末在南京就可以随心所欲了。今天是新赛季中超联赛第一轮,下午3点半奥体又要热闹了。去年舜天队不给力,在盛大又特别忙,全年我就看了第一个主场的比赛,结果还输了(十年来第一次在现场输球)!!初中高中的时候要忙学习,大学的时候工作室周六开会,所以这么多年都没有条件保持看球的出勤率。现在时候到了,这是一种怎样的情怀了。

除去安逸的事情以外,我最近在看UEAP,再和@rlove的Linux System Programming对照看比较有感觉,不过我好像动手偏少了。另外还要看一下一个python的wsgi server叫做meinheld它为什么这么快呢。。

最后还要恭喜工作室战友、某著名互联网企业优秀员工、抢了04地理(我几乎忘了我是地理学院的了)第一婚的成功人士tiger新婚愉快!(除了The connection was reset以外就是成功人士的定义了)