pieceofpy.com

After a few drinks at the hotel bar, we head to bed to get ready for Day 3 of OOPSLA.

Keynote: Pyrimids and Software or something…
I showed up late for this, but from what I gathered, it was a comparison to the Pyrmids and software. It was really facinating and I enjoyed Mark Lehner’s talk about finding the lost city and everything .. but something this heavy at 8:30 in the morning is well .. yeah, neat.

Introducing New Ideas Into Your Organization
Linda Rising (http://lindarising.com for those of you living in a cave) gave a great tutorial on how people can get new idea into their organization. Everyone has had to deal with this at one time or another no matter where they have worked. People don’t like change. They don’t wanna change if they don’t have to, even if all the reasons are good and they agree with the new idea, in the end, if they don’t want to change, they won’t. As my wife always says. “I want you to WANT to take out the garbage.”

Quick side note, I was speaking with Ricardo Lopez on the day previous to Linda’s tutorial and his advice was to make sure you have enough in your savings for when you get fired. Now as funny as it sounds, his reason, which was being fearless and with standing pressures to make concessions was directly inline with the ideas and patterns Linda provided in her tutorial.

Back to the tutorial. I arrived early (mistake), so the other early arriver’s and myself ended up being actors in a play. The play of how to introduce new ideas into your organization. The play included Innovators, Early Adopters, Connectors, etc.. and how/when you can use these different types of people in the organization to our benefit and pitfalls to avoid so they don’t become enemies.

The Cliff notes version, get the easy ones on board first (innovators like new cool stuff), put a small step of the new idea in place with this group. Now take it to the early adopters (how depends on your corporate culture), get them involved and maybe add another step. Continue to promote this style growth until you have enough good results from this new thing to ask some of the good ol’ boys who might have middle managements ear, continue on this path with Brown Bags and cookie offerings. At this point the movement itself is viral and you’ll need to start tending to the delicate process of getting the upper management stamp of approval.

Anyway, the tutorial was a pattern language for introducing new ideas based of her latest book and I must say, I really enjoyed the tutorial and acting out my part. Though the pattern language should be adjusted and tailored to fit your corporate culture, I think it is an excellent start for anyone having problems getting no ideas in to their company, be it agile development or more water fountains. The tutorial finished with a set of group sessions tasked with solving a member of each groups problem. You can find more information about these patterns and her on Linda’s website mentioned at the start of this post.

Afterwards
After the keynote and tutorial, my wife and I ate at the hotel restaurant and called it a night. This conference is fun and loaded with great information and conversation, but leaves my brain so tired at the end of the day it is hard to to much else but eat some food and go to sleep.

After a goodnight sleep, I mange to NOT be late for this mornings tutorial sessions. Today I have planned to attend Advanced Software Architecture and Pattern-Oriented Software Architecture: A Pattern Language for Distributed Computing.

Advanced Software Architecture
Marwan Abu-Khalil was the presenter for this tutorial. This wasn’t as hands on as I usually like from my tutorial sessions, but with that being said, there was a lot of good knowledge being tossed around.

We cover the best practices and outlined a rough path one should follow when designing software. Now it was mentioned frequently and often, that this was just the most common path encountered and that each project would lend itself to a different, there inherit lines the hard part of software architecture. If you could just put it in a patterns book, everyone would be good at it. The quote frequently used was “All architecture is design, but not all design is architecture.”

Brainstorm session instigated by this tutorial session.
As a developer I myself tend to loose focus of the bigger picture as I dive deep in to the inner workings of some specific implementation or core sub-system. This course gives a nice high-level perspective of what the software architecture should focus on. For micro sized development teams; 1-5 developers, these things become even more important, since in most cases, every developer on a such a small team will be responsible for some of the most complicated areas of the system. This goes against the rule of thumb that architects, though they should be involved in the implementation, should not be involved in developing any of the core or complex systems of the architecture, so to prevent them from loosing focus in the details.

I agree with this, but as mentioned, in micro teams this can be very hard, if no impossible to do (1 man “team”). This is why introducing things like peer code review and daily standups are so important. Most people seem to have this idea that the smaller you are, the less you need to apply todays agile methods, on the countrary, staying agile with more developers as you can decouple subsystems and spread them across multiple developers or teams. Staying objective and keeping systems decoupled and preventing architectural drift in a one or two man team is even harder than say three, five man teams.

Pattern-Oriented Software Architecture
This tutorial was done by Doug Schmidt, side note, he’s been living in Nashville the last few years doing his research and work out of Vanderbilt University.

This was a great tutorial. The subtitle was A Pattern Language for Distributed Computing. But it wasn’t so much about completely new patterns, in fact any on fimilar with any of the GoF, PLOP, or POSA books would of received a lot of review. This was about using many proven patterns that we’ve used in stand-alone archictectures for a long time, take those and establish a pattern language for solving the problems we encoutned in todays distributed enviroments.

A pattern mentioned which I must of missed from the POSA 2 book, but new it immediately because I have implemented it under different names was the Active Object pattern. Quickly, Active Object decouples the execution from the invokation of a method. Good coverage of other patterns as well. The handout includes pros and cons in a distributed enviroment for all the patterns presented.

I did take issue with some of the cons being things like “Harder to test” and “Harder to debug”. I just kept thinking .. if you mock out the interactive components so that your tests are decoupled from any implementation, it seems no harder to test the Active Object pattern vs. any other piece of software. Questions like what if the request comes back out of order? What if the request processor goes offline? What do you do with the queue? What if you get a response for a request that is no longer valid? No more room in your queue? Etc, etc.. All of these conditions can and should be covered by tests. I don’t have issues with testing or debuging my most recent Active Object implementation.

There was also a nice closing discussion about utilizing multi-core, embedded devices giving us a new reason to remember all the cycle and memory conservation techniques every said we could forget and if existing functional languages would be the wave of the future to take advantage of advancing core technology. Schmidt conclusion was we will need a new language or a dramatic extention to deal with these new concurrency issues … I just kept thinking Stackless.

In the end, it was another standard issue Schmidt tutorial, it did not disappoint. Fast pace and lots of invited Q&A and viewer interaction. Even if you don’t like the guy or his ideas, it is worth it to see him give a talk at least once. Random side note .. he refered to Python once, following the words “novelty languages like…”

I didn’t proof this at all, corrections welcomed.

After the seven hour drive to Nashville, we settled in to our hotel and hit the bar. Getting the brain ready for todays activities. I’ll be updating at the end of each Tutorial or Keynote through out the day, but will only publish the entries at the end of day, so bare with me if the entrys seem like many posts in one.

Green Bar for C++
This was the first turotial I atteneded. The one I had picked was canceled and this was my runner up. The tutorial was handled by the author CUTE Peter Sommerlad. The presentation was good and packed full of content. I have been out of the C++ game for a while now, but his presentation of CUTE integrated in to the new Eclipse has sparked my interest once again and really made me want to take on another C++ project. Maybe an optimized parser this extends Python bindings .. just have to find something to parse.

They were very well setup for the course. Course was very interactive and hands on. I spent the second half of the course implementing the classic Numeric to Roman converted using Eclipse and CUTE with TDD practices. The practices themselves were all refresher for me, but getting a chance to put my hands on some C++ felt good. It had been over a year since I really got dirty in C++. They also had multi-platform installs of Eclipse and a CUTE package archive on a flash drive. Everyone in the tutorial was up and running in no time. I made not of this for sure, in case my any of my recent proposals to speak get accepted.

I was also able to see other peoples approach to the same problem which is always insigntful and thought provoking, no matter the language. I spoke with Peter breifly after the tutorial. The functionality of the CUTE framework itself only took about 2 days of development. The integration in to Eclipse is a different monster. Thanks for the free pen and the swiss chocolate.

Building SOAs with Web Services
Olaf from IBM gives us a great tutorial on using SOAs to build web services. The introduction was an abstract overview of SOA (Service Oriented Archiecture) with some touches on concepts and motivation behind SOAP and WSDLs.

He then spoke about some real world implementations and showed some examples and diagrams of those implementations. After which we began the interactive part of walking through a very simple implementation that tied a PHP script to a Java SOA via Tomcat/Axis.

Coverage was also given to user experiences and best practices and identifying when a given problem lends itself to the SOA (WSDL/SOAP) solution. Overall it was a good tutorial and getting a chance to see the Web Tools plugin for Eclipse in action generating bottom-up from existing POJO and making the WSDL contract was neat.

Afterwards
After the day at OOPSLA my wife and I at some St. Louis style BBQ down on Broadway. It was very good. I have a personal goal to eat nothing but BBQ for dinner during my time in Nashville.

I read through the guidelines and I didn’t see anything against me publishing my submission for Pycon. So I’d like any feedback you might have on this tutorial outline.

I have to ask myself “Who is the target audience?”. A developer with some programming, maybe not Python experience who does web programming. A developer with some basic familiarity with web frameworks, maybe having used rails, Turbogears, Django, or even Pylons before. A developer who uses Pylons but hasn’t really looked at what it can do for them?

I also keep hemming and hawing over the web services the “hard way” section. Why show someone the way not to do it? My argument for it is I find the positive impact of doing it the easy way, with the RESTful controller/routes template to be so much greater when it directly follows the common, home grown style I see all over.

I’ve kept it off the main page, since the proposal is rather long.
Continue reading ‘PyCon 2009 Submission’ »

We have a new theme? You like? If not, blame commenter rholmes, it is his fault. Seriously though. In a previous post he brought up a very good point, the site looked like hell if you were browsing with images off. Well this new theme looks better with images off and overall it isn’t too horribly bad. So, if you don’t like it, suggest one, just make sure it looks good with images turned off.

Now to the SCM part. After about a week of playing with Github and Lighthouse, I found them both to be great products. They integrated well with each other and the tools for working with Git are available under every platform. If I didn’t already have my own server and experience deploying Trac and Mercurial I would use both these services without question. That being said, though I have enjoyed my time working with those tools, I’ve migrated my Github source and Lighthouse tickets over to a newly installed Trac 0.11, full circle.

End result. If you like tinkering. If you like managing your own installations or you have some customization/integration you would like to do, use Trac and Mercurial, otherwise use Github and one of the great ticket systems it integrates with; I enjoyed Lighthouse.

For fun, here is the circle of ticketing and scm life I’ve gone through over the last 6 years or so.

• cvs and PHP Ticket
• svn and home grown Python tickets
• svn and Trac
• Github and Lighthouse
• mercurial and Trac

Have to wonder what is next. A lot of people at work have been asking me why Github or mercurial? Why Wayne? True, that most of the time I am using the repositories for me, myself, and I (De La Soul), but the benefits extend just beyond handling version control for an arbitrary number of developers and clean merges. I’ll do a write up soon.

All the sourcecode for this site is now located at: http://trac.pieceopfy.com/pieceofpy
I’ll be updating all the old links through the other posts to reflect this. Fun.

You see lots of examples on the net for SQLalchemy. Implementing a blog, implementing a wiki, even other articles on implementing tags. Some are good, some are pretty poor, and some are just plain out of date. After some researching on best practices for implementing a Tag system with SQLalchemy I’ve come up with the solution you are about to read.

I’ve pulled these examples from real world production code. Just renamed them and shortened them up a little for the blog post. I pulled the naming convention right from SimpleSite example for Pylons. Here is the the table layout. Simple. A page, tag, and relation table.

page_table = sa.Table("page", meta.metadata,
    sa.Column("id", sa.types.Integer, sa.schema.Sequence('page_seq_id', optional=True), primary_key=True),
    sa.Column("name", sa.types.Unicode(100), nullable=False),
)

tag_table = sa.Table("tag", meta.metadata,
    sa.Column("id", sa.types.Integer, sa.schema.Sequence('taq_seq_id', optional=True), primary_key=True),
    sa.Column("name", sa.types.Unicode(50), nullable=False, unique=True),
)

pagetag_table = sa.Table("pagetag", meta.metadata,
    sa.Column("id", sa.types.Integer, sa.schema.Sequence('pagetag_seq_id', optional=True), primary_key=True),
    sa.Column("pageid", sa.types.Integer, sa.schema.ForeignKey('page.id')),
    sa.Column("tagid", sa.types.Integer, sa.schema.ForeignKey('tag.id')),
)

Now the important part, the mapper. The mapper is what is going to tell sqlalchemy what you are trying to do and how to handle and relate those ForeignKeys. It does the heavy lifting so you don’t have to.

class Tag(object):
    pass

class Page(object):
    pass

orm.mapper(Tag, tag_table)
orm.mapper(Page, page_table, properties = {
    'tags':orm.relation(Tag, secondary=pagetag_table, cascade="all,delete-orphan"),
})

This does two things. It setups the relationship and also uses the built-in cascade rule from SQLalchemy to ensure that no orphan tags are left in the database.

So now we can use this model setup like so. Here, I’ve just started up my paster shell so I could work through some quick usage examples.

page = model.Page()
page.name = "Example Page"

tag = model.Tag()
name = "tag"

page.tags.append(tag)
meta.Session.save(page)
meta.Session.commit()

tag_q = meta.Session.query(model.Tag)
tags = tag_q.all()
len(tags)

# filter pages by tag(s)
page_q = meta.Session.query(model.Page)
pages = page_q.join('tags').filter_by(name="tag").all()

# delete-orphans does the work for us here...
meta.Session.delete(pages[0])
meta.Session.commit()

tags = tag_q.all()
len(tags)

# tag cloud anyone?
# see the source code linked below for a properly weighted tag cloud.
tag_q = meta.Session.query(func.count("*").label("tagcount"), model.Tag)
tag_r = tag_q.filter(model.Tag.id==model.pagetag_table.c.tagid).group_by(model.Tag.id).all()

# what about pages with related tags?
page_q = meta.Session.query(model.Page)

taglist = ["tag1", "tag2"]
tagcount = len(taglist)
page_q.join(model.Page.tags).filter(model.Tag.name.in_(taglist)).\
group_by(model.Page.id).having(func.count(model.Page.id) == tagcount).all()

Ok, now the fun part, what about all related tags? An intersection between an arbitrary number of many-to-many relationships? For that I added a static method to my tag class. Something like this.

class Tag(object):
    @staticmethod
    def get_related(tags=[]):
        tag_count = len(tags)

        inner_q = select([pagetag_table.c.pageid])
        inner_w = inner_q.where(
            and_(pagetag_table.c.tagid == Tag.id,Tag.name.in_(tags))
        ).group_by(pagetag_table.c.pageid).having(func.count(pagetag_table.c.pageid) == tag_count).correlate(None)

        outer_q = select([Tag.id, Tag.name, func.count(pagetag_table.c.shipid)])
        outer_w = outer_q.where(
            and_(pagetag_table.c.pageid.in_(inner_w),
            not_(Tag.name.in_(tags)),
            Tag.id == pagetag_table.c.tagid)
        ).group_by(pagetag_table.c.tagid)

        related_tags = meta.Session.execute(outer_w).fetchall()
        return related_tags

A big thanks to PHP-Cake and TagSchema for the ideas, concepts, and implementation examples.

You can find the actual code that this blog was the basis for at:
http://trac.pieceofpy.com/pieceofpy/browser/tags-sqlalchemy

A friend having issues installing Pylons on Windows XP with Python 2.6 gave me the idea to do this quick write up. So here it is, the 6 step method for running Pylons on Windows XP.

• Download Python.
• Add Python to your path and launch a command prompt.
• Download ez_setup.py, python ez_setup.py
• Download simplejson, python setup.py –without-speedups install
• easy_install Pylons
• easy_install formbuild
• Do a quick test; paster create –template=pylons

And that is all she wrote. Pretty easy. The reason we install simplejson seperate is because the default behavior is to build with speedups and well .. by default, that behavior won’t work on a standard Windows XP machine. So we install it seperate to avoid any conflicts.

In preparation for a production deployment of a new Pylons app, I’ve been looking in to different deployment methods. In an effort to to be /. safe and Diggable when the new application launches, we’ve decided on 4 server deployment.

• 1 nginx server
• 2 pylons (paster) servers
• 1 postgresql server

I built nginx from the source without issues. The default install location of /usr/local/nginx works for me. You’ll need to make any init scripts and install them, see your platform doucmentation for how to do this. You’ll also want to be sure to add the new log dir to any log stats/consolidating/trimming jobs you run.

Here is the important parts of the nginx configuration for proxying to the Paster servers. Also be sure you adjust your keep alive and connection timeout settings, if you have just a standard Ajaxy Web 2.0 application, you’ll want to kick that down to 5 5 or 5 10. They default is way to high unless you’re doing constant streaming of live updates or something to that degree.

worker_processes  2;
events {
    worker_connections  1024;
}
http {
    client_body_timeout   5;
    client_header_timeout 5;
    keepalive_timeout     5 5;
    send_timeout          5;

    tcp_nodelay on;
    tcp_nopush  on;

    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   1;
    gzip_http_version 1.0;
    gzip_min_length   0;
    gzip_types        text/plain text/html text/css;
    gzip_vary         on;

    upstream pasters {
        server 10.3.0.5:5010;
        server 10.3.0.6:5011;
    }
    server {
        listen       80;
        server_name  localhost;

        location / {
            proxy_pass http://pasters;
            proxy_redirect default;
        }
    }

The paster servers are setup like this, I put them both in the same .ini and setup them up in the tpl. This lets me do an easy_install , setup-app based deployment without having to manually edit the ini to change the port numbers, which is error prone. This also lets you adjust and tune per server, instead of deploying 1 server section and changing it for each. Example would be if one server was way more powerful, you could tune it and then use the weighting in nginx to prefer that server. All without having to edit the ini after deployment.

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 5010
use_threadpool = True
threadpool_workers = 10

[server:main2]
use = egg:Paste#http
host = 0.0.0.0
port = 5011
use_threadpool = True
threadpool_workers = 10

Using 10 1000 on Apache bench gave me some good results. 85 requests per second to either of the standalone Paster servers. 185 requests per second when balanced with nginx. For fun, I deployed a third on my database server and was pleased to see 250 requests per second. Then I deployed 3 per server. So a total of 9 paster instances and was able to see 1080 requests per second. I also increased the thread of each from 10 to 25 , this uses more memory, but enables a higher RPS.

Getting closer to the estimated 2,500 needed to survive a /. and should survive the estimated 1,000 from a high Digg.

In the world of MVC and RESTful services, the old addage fat models, skinny controllers is something I’m sure you’ve constantly seen and read about. So what does it really mean? How do you benefit? Is it the silver bullet for MVC development? What are the draw backs?

Using the latest versions of Pylons and SQLalchemy (0.9.7rc2 and 0.5.0rc1 respectivly) we can implement this methodology pretty easily. We’ll use formencode schemas to handle the basic input validation and then keep our business logic in the controller itself.

Here is what a controller method using this concept might look like.

class MemberController(BaseController):

    def __before__(self):
        if session.has_key('memberid'):
            c.memberid = session['memberid']

    @validate(schema=model.forms.schema.SubscriptionSchema(), form='new')
    def create(self):
        subscription = model.Subscription(c.memberid, **self.form_result)
        meta.Session.save(subscription)
        meta.Session.commit()
        return redirect_to(controller='member', action='account')

The schema validation affords us the luxury of being able to just pass our data directly to the model. The __before__ method checks the session for the memberid assigned at login and gives us access to it, further keeping our method nice and clean. The model would implement the business logic, in this case since this is creating a new subscription, it would just sum now() and deltatime(days=days) to determine the expired.

This model could later be expanded upon, say for example you added an upgrade methods to your controller. Now, the same subscription model could be used with some added logic. The model could now have a static prorate method to expire the existing account and make room for creating the new subscription. I’ve pushed the example source to my github, hopefully this will get your brain juices flowing. If I get bored, I’ll toss together a complete working example and check it in.

Source for this post can be found at
http://trac.pieceofpy.com/pieceofpy/browser/fat-models-skinny-controllers

I’ve enabled Gravatar for comments, if you don’t know what that is, you can find out all you need to know at http://gravatar.com

I have also setup a github account for all the source code from this blog. It is pretty easy to use and should a multi-contributor project stem from this blog, we’ll have a place for it. Find out more at http://github.com or browse the repo at:

http://github.com/wwitzel3/pieceofpy/tree/master

UPDATED
I took down the github repo and setup a mercurial repo with trac on top of it.
http://trac.pieceopfy.com/pieceofpy

See Blog themes and SCM for more details.