authored by Wayne Witzel III

I've moved! Checkout the latest posts on waynewitzel.com

This site will remain online to preserve links. All new content will be at the new domain.

This is a story about Pyramid, pyres, some Python 3.3 porting, and just how easy it is to get a task queue working with Pyramid and redis.

The story starts with John Anderson convincing me to work on notaliens.com. In the process of developing the Sites portion of the project we decided we wanted to implement a task queue for capturing, storing, and generating the thumbnails of the sites that are submitted to notalienss.

After looking over a few choices, John had some previous experience with pyres at SurveyMonkey so we pushed forward with that.

During the process of implementing the task queue we discovered that pyres wasn't Python 3.3 compatible. As the flagship community site for the Pyramid project, we felt maintaining Python 3.3 support was important.

So we had a choice, switch to an already Pyhton 3.3 compatible task queue system or take on porting pyres to Python 3.3. We talked about some other options like using celery or retools, but we decided we liked the API and the simplicity of pyres so much that we could take on the porting effort.

Fortunately for us the pyres project had some great tests. This made the process of porting pretty simple. You can actually see the diff for the pull request we submitted to pyres.

Google Summer of Code is right around the corner. In fact student applications are already open. The Apache Software Foundation has been accepted as one of the mentoring organizations. This is great news for Allura which is part of the ASF Incubator, because we will be eligble to mentor and accept student proposals for working on Allura.

So if you are intersted in working on a open project hosting platform that is written in Python, I encourage you to register and submit a proposal to the Apache Software Foundation for working on the Allura project. You can find the tickets that are part of the Allura GSoC 2013 on the Apache COMDEV Jira.

You can also find useful information about the GSoC on the ASF GSoC FAQ Page and on the Allura GSoC Wiki page.

1. What's the coolest Python application, framework, or library you have discovered in 2012?

matplotlib - This library is a python 2D plotting library that creates great figures that you can use to provide a visual representation of many data. You can generate plots, histograms, bar charts, etc. The API for figure creation is very simple and straight forward and I have really enjoyed using it over the last few months.

2. What new programming technique did you learn in 2012

The programming I do is a pretty equal mix of brand new code and extensions to existing code. This year I took the time to learn how to better refactor existing code. That can mean a lot of things to a lot of people, but I specifically focused on refreshing my knowledge of code smells as defined in Refactoring - Fowler (1999) and taking the time to learn how to make better code from the start by re-reading Clean Code - Martin (2008).

Everytime I re-read books such as these, they almost feel completely new. The years that have passed have given me a completely different learning expereince from the exact same text.

3. Which open source project did you contribute to the most in 2012? What did you do?

Allura - To be fair, I work for SourceForge so I get to contribue to Allura as part of my day job. The contributions range from bug fixes to backend updates to tools and extensions.

Pyramid - This is the project I contributed to the most that was not part of my day job. That said, the overall level of contributions wasn't that high, just a few bug fixes, minor JSON renderer update, documentation tweaks, and some Pyramid related blog posts. You can see them here.

4. Which Python blog or website did you read the most in 2012

Reddit, but to be fair, a lot of the stories I went to on Reddit were because they were linked in the Pycoder's Weekly newsletter. So maybe Pycoder's Weekly should take the credit.

5. What are the top things you want to learn in 2013

In 2013 I hope to continue learning more about mobile development, use AngularJS for something other than a trivial demo, and continue my general quest to always be a better developer than I am today. I've also been learning a lot of nutrition and other health sciences and will probably contintue to focusing on learning more about them in 2013.

6. What is the top software, application, or library you wish someone would write in 2013

I think it would be really awesome if there was a recipe application that allowed you to enter in your macronutrient needs for the day (Protein, Fat, Carbs) and spit out a list of recipes, based on a selected meal frequency, that would make you hit all of your requirements.

Want to do your own list? here's how:

• copy-paste the questions and answser them in your blog
• tweet it with #2012pythonmeme hashtag

When writing web application that expose an API, at some point we end up needing to serialize our custom objects. We create these objects ourselves and other times we are using third-party libraries.

Fortunately the latest Pyramid master has new features that allow you to easily define a serialization protocol for custom objects.

We provide you with two approaches, the first one will feel familar if you have used TurboGears. The second allows you to handle the serialization of any object, even those third-party ones you didn't write.

Using __json__

Using the __json__ method is great when you want an easy way to serialize your own objects. All you need to do is define a __json__ method on your class and use the default json renderer on your view. Like this.

class CustomObject(object):
    def __init__(self, name, email):
        self.name = name
        self.email = email
        self.timestamp = datetime.utcnow()
    def __json__(self, request):
        return dict(
            name=self.name,
            email=self.email,
            timestamp=self.timestamp.isoformat(),
            )

@view_config(route_name="custom_object", renderer="json")
def custom_object(request):
    from objects import CustomObject
    results = dict(
        count=2,
        objects=[
            CustomObject('Wayne Witzel III', 'wayne@pieceofpy.com'),
            CustomObject('Fake Person', 'fake.person@pieceofpy.com'),
            ],
        )

return results

You can see here, this is taking the non-serializable datetime that is part of your custom object and turning it into a serializable string using the isoformat call. The default json renderer looks for this special __json__ method. Once you have that defined, there is nothing more for us to do. As long as the return of __json__ is serializable, everything is handled for us. Even when returning lists of custom objects, like say the results of a SQL query.

Using add_adapter

Now if extending the object itself isn't desirable, you can use a custom adapater. This uses a type checking approach, that registers a serialization method for a specific type. It has a little more setup than the __json__ approach above, but is great when dealing with built-in types of third party objects.

First we create a method that knows how to serialize our object.

class ThirdPartyObject(object):
    def __init__(self):
        self.value = Decimal(0.1)

def third_party_adapter(obj, request):
    return dict(
        value=str(obj.value),
        )

So here, we define a very simple adapter that knows how to deal with our Decimal value. Now in our __init__ we need to tell Pyramid about this custom method.

json_third_party = JSON()
json_third_party.add_adapter(ThirdPartyObject, third_party_adapter)
config.add_renderer('json_third_party', json_third_party)
config.add_route('third_party', '/third_party.json')

Finally we can now tell our view to use our newly registered json_third_party renderer when our view returns.

@view_config(route_name="third_party", renderer="json_third_party")
def third_party(request):
    from objects import ThirdPartyObject
    results = dict(
        count=1,
        objects=[
            ThirdPartyObject(),
            ],
        )
    return results

As you can see if is very easy to configure Pyramid to JSON serialize custom objects even if you aren't the original author or don't want to modify the code of an object.

Resources

You can try out this feature by checking out the latest copy of Pyramid master from Github. A working Pyramid demo for this blog post is available here.

Reference: Renderers