authored by Wayne Witzel III

Pyramid Task Queue with pyres and redis

On July 14, 2013 In python Permalink

This is a story about Pyramid, pyres, some Python 3.3 porting, and just how easy it is to get a task queue working with Pyramid and redis.

The story starts with John Anderson convincing me to work on notaliens.com. In the process of developing the Sites portion of the project we decided we wanted to implement a task queue for capturing, storing, and generating the thumbnails of the sites that are submitted to notalienss.

After looking over a few choices, John had some previous experience with pyres at SurveyMonkey so we pushed forward with that.

During the process of implementing the task queue we discovered that pyres wasn't Python 3.3 compatible. As the flagship community site for the Pyramid project, we felt maintaining Python 3.3 support was important.

So we had a choice, switch to an already Pyhton 3.3 compatible task queue system or take on porting pyres to Python 3.3. We talked about some other options like using celery or retools, but we decided we liked the API and the simplicity of pyres so much that we could take on the porting effort.

Fortunately for us the pyres project had some great tests. This made the process of porting pretty simple. You can actually see the diff for the pull request we submitted to pyres.

Read and Post Comments

Allura - Google Summer of Code 2013

On April 23, 2013 In python Permalink

Google Summer of Code is right around the corner. In fact student applications are already open. The Apache Software Foundation has been accepted as one of the mentoring organizations. This is great news for Allura which is part of the ASF Incubator, because we will be eligble to mentor and accept student proposals for working on Allura.

So if you are intersted in working on a open project hosting platform that is written in Python, I encourage you to register and submit a proposal to the Apache Software Foundation for working on the Allura project. You can find the tickets that are part of the Allura GSoC 2013 on the Apache COMDEV Jira.

You can also find useful information about the GSoC on the ASF GSoC FAQ Page and on the Allura GSoC Wiki page.

Read and Post Comments

New Year's Python Meme 2012

On December 29, 2012 In python Permalink

1. What's the coolest Python application, framework, or library you have discovered in 2012?

matplotlib - This library is a python 2D plotting library that creates great figures that you can use to provide a visual representation of many data. You can generate plots, histograms, bar charts, etc. The API for figure creation is very simple and straight forward and I have really enjoyed using it over the last few months.

2. What new programming technique did you learn in 2012

The programming I do is a pretty equal mix of brand new code and extensions to existing code. This year I took the time to learn how to better refactor existing code. That can mean a lot of things to a lot of people, but I specifically focused on refreshing my knowledge of code smells as defined in Refactoring - Fowler (1999) and taking the time to learn how to make better code from the start by re-reading Clean Code - Martin (2008).

Everytime I re-read books such as these, they almost feel completely new. The years that have passed have given me a completely different learning expereince from the exact same text.

3. Which open source project did you contribute to the most in 2012? What did you do?

Allura - To be fair, I work for SourceForge so I get to contribue to Allura as part of my day job. The contributions range from bug fixes to backend updates to tools and extensions.

Pyramid - This is the project I contributed to the most that was not part of my day job. That said, the overall level of contributions wasn't that high, just a few bug fixes, minor JSON renderer update, documentation tweaks, and some Pyramid related blog posts. You can see them here.

4. Which Python blog or website did you read the most in 2012

Reddit, but to be fair, a lot of the stories I went to on Reddit were because they were linked in the Pycoder's Weekly newsletter. So maybe Pycoder's Weekly should take the credit.

5. What are the top things you want to learn in 2013

In 2013 I hope to continue learning more about mobile development, use AngularJS for something other than a trivial demo, and continue my general quest to always be a better developer than I am today. I've also been learning a lot of nutrition and other health sciences and will probably contintue to focusing on learning more about them in 2013.

6. What is the top software, application, or library you wish someone would write in 2013

I think it would be really awesome if there was a recipe application that allowed you to enter in your macronutrient needs for the day (Protein, Fat, Carbs) and spit out a list of recipes, based on a selected meal frequency, that would make you hit all of your requirements.

Want to do your own list? here's how:

  • copy-paste the questions and answser them in your blog
  • tweet it with #2012pythonmeme hashtag
Read and Post Comments

Pyramid - JSON Serialize Custom Objects

On July 17, 2012 In python, pyramid Permalink

When writing web application that expose an API, at some point we end up needing to serialize our custom objects. We create these objects ourselves and other times we are using third-party libraries.

Fortunately the latest Pyramid master has new features that allow you to easily define a serialization protocol for custom objects.

We provide you with two approaches, the first one will feel familar if you have used TurboGears. The second allows you to handle the serialization of any object, even those third-party ones you didn't write.

Using __json__

Using the __json__ method is great when you want an easy way to serialize your own objects. All you need to do is define a __json__ method on your class and use the default json renderer on your view. Like this.


class CustomObject(object):
    def __init__(self, name, email):
        self.name = name
        self.email = email
        self.timestamp = datetime.utcnow()
    def __json__(self, request):
        return dict(
            name=self.name,
            email=self.email,
            timestamp=self.timestamp.isoformat(),
            )

@view_config(route_name="custom_object", renderer="json")
def custom_object(request):
    from objects import CustomObject
    results = dict(
        count=2,
        objects=[
            CustomObject('Wayne Witzel III', 'wayne@pieceofpy.com'),
            CustomObject('Fake Person', 'fake.person@pieceofpy.com'),
            ],
        )

return results

You can see here, this is taking the non-serializable datetime that is part of your custom object and turning it into a serializable string using the isoformat call. The default json renderer looks for this special __json__ method. Once you have that defined, there is nothing more for us to do. As long as the return of __json__ is serializable, everything is handled for us. Even when returning lists of custom objects, like say the results of a SQL query.

Using add_adapter

Now if extending the object itself isn't desirable, you can use a custom adapater. This uses a type checking approach, that registers a serialization method for a specific type. It has a little more setup than the __json__ approach above, but is great when dealing with built-in types of third party objects.

First we create a method that knows how to serialize our object.


class ThirdPartyObject(object):
    def __init__(self):
        self.value = Decimal(0.1)

def third_party_adapter(obj, request):
    return dict(
        value=str(obj.value),
        )

So here, we define a very simple adapter that knows how to deal with our Decimal value. Now in our __init__ we need to tell Pyramid about this custom method.

json_third_party = JSON()
json_third_party.add_adapter(ThirdPartyObject, third_party_adapter)
config.add_renderer('json_third_party', json_third_party)
config.add_route('third_party', '/third_party.json')

Finally we can now tell our view to use our newly registered json_third_party renderer when our view returns.

@view_config(route_name="third_party", renderer="json_third_party")
def third_party(request):
    from objects import ThirdPartyObject
    results = dict(
        count=1,
        objects=[
            ThirdPartyObject(),
            ],
        )
    return results

As you can see if is very easy to configure Pyramid to JSON serialize custom objects even if you aren't the original author or don't want to modify the code of an object.

Resources

You can try out this feature by checking out the latest copy of Pyramid master from Github. A working Pyramid demo for this blog post is available here.

Reference: Renderers

Read and Post Comments

Working with Pyramid and Ming

On January 10, 2012 In python, ming Permalink

As I've been working on Community Cookbook I have encountered some situations using Ming that I am sure many other people have as well. Using Ming as my ODM and I have been very happy with how it is going so I wanted to share with everyone the steps I have taken to integrate Ming into Pyramid. My goal was to make it feel just like you would except a data storage integration to feel. Simple, clean, and easy.

As part of creating this integration, I created my own scaffold to hook everything up. You can download and try the scaffold yourself from my SourceForge repository. Below I will outline some of the key areas I had to configure. I highly reccomend you install and create a sample project with the scaffold when you follow along with this blog post.

First in the main init.py of the project, I had to setup some configuration. If you have created a new project using the scaffold this will be done for you. Here are the important snippets from root init.py

# stockpot/__init__.py

from pyramid.events import NewRequest
import stockpot.models as M

def main(global_config, **settings):
    # ...
    config.begin()
    config.scan('stockpot.models')
    M.init_mongo(engine=(settings.get('mongo.url'), settings.get('mongo.database')))
    config.add_subscriber(close_mongo_db, NewRequest)
    # ...

def close_mongo_db(event):
    def close(request):
         M.DBSession.close_all()
    event.request.add_finished_callback(close)

Now lets take a look at what you are getting from the models import (M). We use init mongo which does exactly what you think and then we use DBSession which I used as a name since it is familar to those who are coming from SQLalchemy. I will just show you the entire file.

# stockpot/models/__init__.py

from ming import Session
from ming.datastore import DataStore
from ming.orm import ThreadLocalORMSession
from ming.orm import Mapper

session = Session
DBSession = ThreadLocalORMSession(session)

def init_mongo(engine):
    server, database = engine
    datastore = DataStore(server, database=database)
    session.bind = datastore
    Mapper.compile_all()

# Here we just ensure indexes on all our mappers at startup.
    for mapper in Mapper.all_mappers():
        session.ensure_indexes(mapper.collection)

# Flush changes and close all connections
    DBSession.flush()
    DBSession.close_all()

from .user import User

Now I want to show you how I did the groupfinder and RequestWithAttribute. Since I imagine most of you looking to use Ming will probably have some type of user model. For the groupfinder and RequestWithAttributes the only special thing I had to do was add an extra step to convert the users id in to a proper bson.ObjectId

from pyramid.decorator import reify
from pyramid.request import Request
from pyramid.security import unauthenticated_userid

import bson
import stockpot.models as M

def groupfinder(userid, request):
    userid = bson.ObjectId(userid)
    user = M.User.query.get(_id=userid)
    return [] if user else None

class RequestWithAttributes(Request):
    @reify
    def user(self):
        userid = unauthenticated_userid(self)
    userid = bson.ObjectId(userid)
    if userid:
        return M.User.query.get(_id=userid)
    return None

Finally here is what my mapped User model might look like. This is a sample taken from a real project with some sensitive elements replaced and/or removed. Use it as an example, but not as a good one.

# stockpot/models/user.py

from hashlib import sha1
from datetime import datetime
from string import ascii_letters, digits
from random import choice

from ming import schema as S
from ming.orm import FieldProperty
from ming.orm.declarative import MappedClass

from pyramid.httpexceptions import HTTPForbidden

from stockpot.models import DBSession

# If you change this AFTER a user signed up they will not be able to
# login until they perform a password reset.
SALT = 'supersecretsalt'
CHARS = ascii_letters + digits
MAX_TRIES = 100

class User(MappedClass):
    class __mongometa__:
        session = DBSession
        name = 'users'
        custom_indexes = [
                dict(fields=('email',), unique=True, sparse=False),
                dict(fields=('username',), unique=True, sparse=False),
                dict(fields=('identifier',), unique=True, sparse=False),
        ]

# This should feel familar to anyone coming from SQLa
    _id = FieldProperty(S.ObjectId)
    username = FieldProperty(str)
    email = FieldProperty(str)
    password = FieldProperty(str, if_missing=S.Missing)
    signup_date = FieldProperty(datetime, if_missing=datetime.utcnow())

# Simple init method
    def __init__(self, *args, **kwargs):
        self.signup_date = datetime.utcnow().replace(microsecond=0)
        self.username = kwargs.get('username')
        if kwargs.get('password'):
            self.password = User.generate_password(kwargs.get('password'),
                    str(self.signup_date))
        self.email = kwargs.get('email', '{0}@example.com'.format(self.username))

# Update method is a little different than you are used to
    # ensure by the time you're calling this everything is validated and safe
    def update(self, *args, **kwargs):
        for k,v in kwargs.items():
            if k == 'password':
                v = User.generate_password(v, str(self.signup_date))
            setattr(self, k, v)

# Standard authenticate method
    @classmethod
    def authenticate(cls, login, password):
        user = cls.query.find({'$or': [{'username':login}, {'email':login}]}).one()
        if user:
            password = User.generate_password(password, str(user.signup_date))
            if password == user.password:
                return user
        else:
            return None

# Password hashing
    @staticmethod
    def generate_password(password, salt):
        password = sha1(password).hexdigest() + salt
        return sha1(password+SALT).hexdigest()

# Username generation, the username generation is very useful if you use social auth
    @staticmethod
    def random_username(_range=5, prefix=''):
        return prefix + ''.join([choice(CHARS) for i in range(_range)])

And there you have it, if you tie all the elements together you get a pretty easy and straightforward Ming integration with Pyramid.

Read and Post Comments
Next Page ยป