pieceofpy.com

Today I was having a chat today about Pylons vs. Django and for the most part it was pretty diplomatic. We got to talking about the Admin interface the Django has. Which you don’t have to do any extra boiler plate for, it is just there for you. With Pylons you have to use something like FormAlchemy or use Turbogears to get a similar style admin interface for your models and data.

Since we were sitting at a computer, I went ahead brought up a quick project and did a little demo of the paster shell. Sure, it involves typing and it isn’t as pretty or “fast” as an admin panel, but he didn’t even know it existed. One of the common things he mentioned was, “if I want to change the menus that are dynamically defined” or “if a username needs to be changed” .. and the application itself doesn’t have a custom admin panel, with Pylons he had to do raw SQL.

$paster shell pylons_config.ini

All objects from demo.lib.base are available
Additional Objects:
   mapper     -  Routes mapper object
   wsgiapp    -  This project's WSGI App instance
   app        -  paste.fixture wrapped around wsgiapp

>>> error_user = meta.Session.query(model.User).filter_by(username='wwitzel 3').one()
>>> # nice thing about this, is you also will get exceptions throw if more than one record exists
>>> error_user.username
u'wwitzel 3'
>>> error_user.username = 'wwitzel3'
>>> meta.Session.commit()
>>> menu_typo = meta.Session.query(model.Menu).filter_by(id=1).one()
>>> menu_typo.value
u'Abuot'
>>> menu_typo.value = 'About'
>>> meta.Session.commit()

So that is a very simple example of how one would use the paster shell to update some bad data in the database while ensuring integrity of your custom model and extension code. After I showed this to my friend he wasn’t as concerned about the lack of a web interface for administration within Pylons.

I asked the question I Stackoverflow and maybe it was too generic for the site, since it just got trolled with “Google keyword” by some d-bag. So I deleted it and figured I’d throw it up on my blog a see about getting some feedback from the people who read this pile about. The reason I ask this is mainly because I am preparing to do some updated screencasts for Pylons.

I’ve seen multiple ways referenced in official docs and I have done it a few different ways myself. I am using Pylons and I am curious what the best practices are for this common scenario?

I have used something similar to this for auto-magically making the conversion happen.

# The auto-magic version
# I pulled this off a blog, forget the source.
def _sa_to_dict(obj):
    for item in obj.__dict__.items():
        if item[0][0] is '_':
            continue
        if isinstance(item[1], str):
            yield [item[0], item[1].decode()]
        else:
            yield item

def json(obj):
    if isinstance(obj, list):
        return dumps(map(dict, map(_sa_to_dict, obj)))
    else:
        return dumps(dict(_sa_to_dict(obj)))

# here is the controller
@jsonify
def index(self, format='html'):
    templates = Session.query(Template).all()
    if format == 'json':
        return json(templates)

I have also done the version where you use the jsonify decorator and build your dictionary manually, something like this, which is ok if I need to define some custom behavior for my JSON, but as the default behavior seems excessive.

@jsonify
def index(self, format='html'):
    if format == 'json':
        q = Session.query
        templates = [{'id': t.id,
                      'title': t.title,
                      'body': t.body} for t in q(Template)]
        return templates

I’ve also created an inherited SA class which defines a json method and have used that on all my objects to convert them to JSON. Similar to the the fedora extensions.

Maybe I missed some obviously library out there or some obvious helper in the Pylons packages, but I feel like this is a very common task being done a dozen different ways between docs, source, and my own personal projects. Curious what others are doing / using.

UPDATE / 13 March 2009: snakefight 0.3 now has a –include-jar option, prefer that to using my hack.

After reading P. Jenvey’s blog post about Deploying Pylons Apps to Java Servlet Containers I immediately downloaded the Jython 2.5 beta and installed snakefight to give it a try. One of our services where I work is a Pylons based application. It is deployed using paster and Apache ProxyPass. Our main application is written in Java and is deployed as a war under Jetty. So if I can get my Pylons application built as a war and deployed that way, it would greatly simplify our deployment process.

$ sudo /opt/jython25/bin/easy_install snakefight
$ /opt/jython25/bin/jython setup.py develop
$ /opt/jython25/bin/jython setup.py bdist_war --paster-config dev_r2.ini
... output of success and stuff ...
$ cp dist/project-0.6.8dev.war /opt/jetty/webapps

Now I visit my local server and hit the project context. I get some database errors, kind of expected them. So for the time being, I’ll be running this directly using Jython to speed up the debugging process. A quick googling of my DB issues turns up zxoracle for SQLalchemy which uses Jython zxJDBC. I install that in to sqlalchemy/databases as zxoracle.py and give it another go. Changing the oracle:// lines in my .ini file to now read zxoracle:// Now it can’t find the 3rd party Oracle libraries (ojdbc.jar).

$ cd ./dist
$ jar xf project-0.6.8dev.war
$ cd WEB-INF/lib
$ ls
# no ojdbc.jar as expected ...
$ cd ~/project
$ export CLASSPATH=/opt/jython25/jython.jar:/usr/lib/jvm/java/jre/lib/ext/ojdbc.jar
$ /opt/jython25/bin/jython /opt/jython25/bin/paster serve --reload dev_r2.ini

Now it is looking a little better and it able to find the jar, but still a DB issue, now with SQLalchemy library. Not having a ton of time to investigate, I decide to try rolling back my SQAlachemy version for Jython. Turns out rolling back to 0.5.0 fixed the issue. I’ll be investigating why it was breaking with 0.5.2 soon ™. So now I rerun it, and get a new error.

AttributeError: 'ZXOracleDialect' object has no attribute 'optimize_limits'

I decide I am just going to go in to the zxoracle.py and add optimize_limits = False to the ZXOracleDialect. No idea what this breaks or harms, but I do it anyway and rerun the application. Success! Every thing is working now. No liking the idea of having to manually insert the Oracle jar in to the WEB-INF/lib and not really wanting to much around with environment variables, I also implemented a quick and dirty include-java-libs for snakefight, the diff for command.py is below. This allows me to pass in a : separated list of jars to include in the WEB-INF/lib. EDIT: The diff I posted isn’t needed since I put it on my hg repo. You can grab it from here.

So now I am back to building my war. Just as before.

$ /opt/jython25/bin/jython setup.py bdist_war --paste-config dev_r2.ini --include-java-libs /opt/jython25/extlibs/ojdbc.jar
running bdist_war
creating build/bdist.java1.6.0_12
creating build/bdist.java1.6.0_12/war
creating build/bdist.java1.6.0_12/war/WEB-INF
creating build/bdist.java1.6.0_12/war/WEB-INF/lib-python
running easy_install project
adding eggs (to WEB-INF/lib-python)
adding jars (to WEB-INF/lib)
adding WEB-INF/lib/jython.jar
adding Paste ini file (to dev_r2.ini)
adding Paste app loader (to WEB-INF/lib-python/____loadapp.py)
generating deployment descriptor
adding deployment descriptor (WEB-INF/web.xml)
created dist/project-0.6.8dev-py2.5.war
$ cp dist/project-0.6.8dev-py2.5.war /opt/jetty/webapps
$ sudo /sbin/service jetty restart

And presto! I am in business. My pylons application is deployed under Jetty and all the selenium functional tests are passing. I am sure there is probably a easier, neater, or cleaner way to do all this, but this was my first iteration through and also my first time ever deploying a WAR to a java servlet container so all in all I am happy with the results. Performance seems about the same as when running the application with paster serve, but Jetty does use a little more memory than before (expected I guess).

So I am playing around in Firefox with XMLHttpRequest. Looking in to a way to facilate a server update to a client without have to refresh the page or use Javascript timers. So the long-live HTTP request seems the way to go.

This little app will at most have 20-30 connections at once, so I am not worried about the open connection per client. The data it calculates is rather large and intensive to gather, so I paired it with the cache decorator snippet found on ActiveState and used in Expert Python Programming. This example feeds a cached datetime string. The caching lets different client receive the same data during the cache process. There is some lag between the updates since they all set their sleep at different points, there may be away around this though.

So here is my basic index.html.

<body>
<em>This will push data from the server to you every 5 seconds .. enjoy!</em>
<ul id="container"></ul>

<script>
var div = document.getElementById('container');
function handleContent(event)
{
  var xml_packet = event.target.responseXML.documentElement;
  div.innerHTML += '<li>' + xml_packet.childNodes[0].data + '</li>';
}
(function () {
    var xrequest = new XMLHttpRequest();
    xrequest.multipart = true;
    xrequest.open("GET","/server/index",false);
    xrequest.onload = handleContent;
    xrequest.send(null);
})();

</script>
</body>

Now the controller code itself.

class ServerController(BaseController):
    def index(self):
        response.headers['Content-type'] = 'multipart/x-mixed-replace;boundary=test'
        return data_stream()

def data_stream(stream=True):
    yield datetime_string()

    while stream:
        time.sleep(5)
        yield datetime_string()

@memorize(duration=15)
def datetime_string():
    content = '--test\nContent-type: application/xml\n\n'
    content += '<?xml version=\'1.0\' encoding=\'ISO-8859-1\'?>\n'
    content += '<message>' + str(datetime.datetime.now()) + '</message>\n'
    content += '--test\n'

    return content

Also the decorator code for good measure.

cache = {}

def is_old(entry, duration):
    return time.time() - entry['time'] > duration

def compute_key(function, args, kw):
    key = pickle.dumps((function.func_name, args, kw))
    return hashlib.sha1(key).hexdigest()

def memorize(duration=10):
    def _memorize(function):
        def __memorize(*args, **kw):
            key = compute_key(function, args, kw)

            if (key in cache and not is_old(cache[key], duration)):
                return cache[key]['value']
            result = function(*args, **kw)
            cache[key] = {'value': result, 'time':time.time()}
            return result
        return __memorize
    return _memorize

Full working demo will be available in the HG repos shortly.

A friend having issues installing Pylons on Windows XP with Python 2.6 gave me the idea to do this quick write up. So here it is, the 6 step method for running Pylons on Windows XP.

• Download Python.
• Add Python to your path and launch a command prompt.
• Download ez_setup.py, python ez_setup.py
• Download simplejson, python setup.py –without-speedups install
• easy_install Pylons
• easy_install formbuild
• Do a quick test; paster create –template=pylons

And that is all she wrote. Pretty easy. The reason we install simplejson seperate is because the default behavior is to build with speedups and well .. by default, that behavior won’t work on a standard Windows XP machine. So we install it seperate to avoid any conflicts.

In preparation for a production deployment of a new Pylons app, I’ve been looking in to different deployment methods. In an effort to to be /. safe and Diggable when the new application launches, we’ve decided on 4 server deployment.

• 1 nginx server
• 2 pylons (paster) servers
• 1 postgresql server

I built nginx from the source without issues. The default install location of /usr/local/nginx works for me. You’ll need to make any init scripts and install them, see your platform doucmentation for how to do this. You’ll also want to be sure to add the new log dir to any log stats/consolidating/trimming jobs you run.

Here is the important parts of the nginx configuration for proxying to the Paster servers. Also be sure you adjust your keep alive and connection timeout settings, if you have just a standard Ajaxy Web 2.0 application, you’ll want to kick that down to 5 5 or 5 10. They default is way to high unless you’re doing constant streaming of live updates or something to that degree.

worker_processes  2;
events {
    worker_connections  1024;
}
http {
    client_body_timeout   5;
    client_header_timeout 5;
    keepalive_timeout     5 5;
    send_timeout          5;

    tcp_nodelay on;
    tcp_nopush  on;

    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   1;
    gzip_http_version 1.0;
    gzip_min_length   0;
    gzip_types        text/plain text/html text/css;
    gzip_vary         on;

    upstream pasters {
        server 10.3.0.5:5010;
        server 10.3.0.6:5011;
    }
    server {
        listen       80;
        server_name  localhost;

        location / {
            proxy_pass http://pasters;
            proxy_redirect default;
        }
    }

The paster servers are setup like this, I put them both in the same .ini and setup them up in the tpl. This lets me do an easy_install , setup-app based deployment without having to manually edit the ini to change the port numbers, which is error prone. This also lets you adjust and tune per server, instead of deploying 1 server section and changing it for each. Example would be if one server was way more powerful, you could tune it and then use the weighting in nginx to prefer that server. All without having to edit the ini after deployment.

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 5010
use_threadpool = True
threadpool_workers = 10

[server:main2]
use = egg:Paste#http
host = 0.0.0.0
port = 5011
use_threadpool = True
threadpool_workers = 10

Using 10 1000 on Apache bench gave me some good results. 85 requests per second to either of the standalone Paster servers. 185 requests per second when balanced with nginx. For fun, I deployed a third on my database server and was pleased to see 250 requests per second. Then I deployed 3 per server. So a total of 9 paster instances and was able to see 1080 requests per second. I also increased the thread of each from 10 to 25 , this uses more memory, but enables a higher RPS.

Getting closer to the estimated 2,500 needed to survive a /. and should survive the estimated 1,000 from a high Digg.

In the world of MVC and RESTful services, the old addage fat models, skinny controllers is something I’m sure you’ve constantly seen and read about. So what does it really mean? How do you benefit? Is it the silver bullet for MVC development? What are the draw backs?

Using the latest versions of Pylons and SQLalchemy (0.9.7rc2 and 0.5.0rc1 respectivly) we can implement this methodology pretty easily. We’ll use formencode schemas to handle the basic input validation and then keep our business logic in the controller itself.

Here is what a controller method using this concept might look like.

class MemberController(BaseController):

    def __before__(self):
        if session.has_key('memberid'):
            c.memberid = session['memberid']

    @validate(schema=model.forms.schema.SubscriptionSchema(), form='new')
    def create(self):
        subscription = model.Subscription(c.memberid, **self.form_result)
        meta.Session.save(subscription)
        meta.Session.commit()
        return redirect_to(controller='member', action='account')

The schema validation affords us the luxury of being able to just pass our data directly to the model. The __before__ method checks the session for the memberid assigned at login and gives us access to it, further keeping our method nice and clean. The model would implement the business logic, in this case since this is creating a new subscription, it would just sum now() and deltatime(days=days) to determine the expired.

This model could later be expanded upon, say for example you added an upgrade methods to your controller. Now, the same subscription model could be used with some added logic. The model could now have a static prorate method to expire the existing account and make room for creating the new subscription. I’ve pushed the example source to my github, hopefully this will get your brain juices flowing. If I get bored, I’ll toss together a complete working example and check it in.

Source for this post can be found at
http://trac.pieceofpy.com/pieceofpy/browser/fat-models-skinny-controllers