authored by Wayne Witzel III

Working with Pyramid and Ming

On January 10, 2012 In python, ming Permalink

As I've been working on Community Cookbook I have encountered some situations using Ming that I am sure many other people have as well. Using Ming as my ODM and I have been very happy with how it is going so I wanted to share with everyone the steps I have taken to integrate Ming into Pyramid. My goal was to make it feel just like you would except a data storage integration to feel. Simple, clean, and easy.

As part of creating this integration, I created my own scaffold to hook everything up. You can download and try the scaffold yourself from my SourceForge repository. Below I will outline some of the key areas I had to configure. I highly reccomend you install and create a sample project with the scaffold when you follow along with this blog post.

First in the main init.py of the project, I had to setup some configuration. If you have created a new project using the scaffold this will be done for you. Here are the important snippets from root init.py

# stockpot/__init__.py

from pyramid.events import NewRequest
import stockpot.models as M

def main(global_config, **settings):
    # ...
    config.begin()
    config.scan('stockpot.models')
    M.init_mongo(engine=(settings.get('mongo.url'), settings.get('mongo.database')))
    config.add_subscriber(close_mongo_db, NewRequest)
    # ...

def close_mongo_db(event):
    def close(request):
         M.DBSession.close_all()
    event.request.add_finished_callback(close)

Now lets take a look at what you are getting from the models import (M). We use init mongo which does exactly what you think and then we use DBSession which I used as a name since it is familar to those who are coming from SQLalchemy. I will just show you the entire file.

# stockpot/models/__init__.py

from ming import Session
from ming.datastore import DataStore
from ming.orm import ThreadLocalORMSession
from ming.orm import Mapper

session = Session
DBSession = ThreadLocalORMSession(session)

def init_mongo(engine):
    server, database = engine
    datastore = DataStore(server, database=database)
    session.bind = datastore
    Mapper.compile_all()

# Here we just ensure indexes on all our mappers at startup.
    for mapper in Mapper.all_mappers():
        session.ensure_indexes(mapper.collection)

# Flush changes and close all connections
    DBSession.flush()
    DBSession.close_all()

from .user import User

Now I want to show you how I did the groupfinder and RequestWithAttribute. Since I imagine most of you looking to use Ming will probably have some type of user model. For the groupfinder and RequestWithAttributes the only special thing I had to do was add an extra step to convert the users id in to a proper bson.ObjectId

from pyramid.decorator import reify
from pyramid.request import Request
from pyramid.security import unauthenticated_userid

import bson
import stockpot.models as M

def groupfinder(userid, request):
    userid = bson.ObjectId(userid)
    user = M.User.query.get(_id=userid)
    return [] if user else None

class RequestWithAttributes(Request):
    @reify
    def user(self):
        userid = unauthenticated_userid(self)
    userid = bson.ObjectId(userid)
    if userid:
        return M.User.query.get(_id=userid)
    return None

Finally here is what my mapped User model might look like. This is a sample taken from a real project with some sensitive elements replaced and/or removed. Use it as an example, but not as a good one.

# stockpot/models/user.py

from hashlib import sha1
from datetime import datetime
from string import ascii_letters, digits
from random import choice

from ming import schema as S
from ming.orm import FieldProperty
from ming.orm.declarative import MappedClass

from pyramid.httpexceptions import HTTPForbidden

from stockpot.models import DBSession

# If you change this AFTER a user signed up they will not be able to
# login until they perform a password reset.
SALT = 'supersecretsalt'
CHARS = ascii_letters + digits
MAX_TRIES = 100

class User(MappedClass):
    class __mongometa__:
        session = DBSession
        name = 'users'
        custom_indexes = [
                dict(fields=('email',), unique=True, sparse=False),
                dict(fields=('username',), unique=True, sparse=False),
                dict(fields=('identifier',), unique=True, sparse=False),
        ]

# This should feel familar to anyone coming from SQLa
    _id = FieldProperty(S.ObjectId)
    username = FieldProperty(str)
    email = FieldProperty(str)
    password = FieldProperty(str, if_missing=S.Missing)
    signup_date = FieldProperty(datetime, if_missing=datetime.utcnow())

# Simple init method
    def __init__(self, *args, **kwargs):
        self.signup_date = datetime.utcnow().replace(microsecond=0)
        self.username = kwargs.get('username')
        if kwargs.get('password'):
            self.password = User.generate_password(kwargs.get('password'),
                    str(self.signup_date))
        self.email = kwargs.get('email', '{0}@example.com'.format(self.username))

# Update method is a little different than you are used to
    # ensure by the time you're calling this everything is validated and safe
    def update(self, *args, **kwargs):
        for k,v in kwargs.items():
            if k == 'password':
                v = User.generate_password(v, str(self.signup_date))
            setattr(self, k, v)

# Standard authenticate method
    @classmethod
    def authenticate(cls, login, password):
        user = cls.query.find({'$or': [{'username':login}, {'email':login}]}).one()
        if user:
            password = User.generate_password(password, str(user.signup_date))
            if password == user.password:
                return user
        else:
            return None

# Password hashing
    @staticmethod
    def generate_password(password, salt):
        password = sha1(password).hexdigest() + salt
        return sha1(password+SALT).hexdigest()

# Username generation, the username generation is very useful if you use social auth
    @staticmethod
    def random_username(_range=5, prefix=''):
        return prefix + ''.join([choice(CHARS) for i in range(_range)])

And there you have it, if you tie all the elements together you get a pretty easy and straightforward Ming integration with Pyramid.

Read and Post Comments

SQLalchemy Cleanup Challenge

On October 13, 2011 In python, sqlalchemy Permalink

Yesterday I found myself writing some very interesting SQLalchemy. The problem is I have a date column in PostgreSQL that is stored as epoch time, so it is just an Interger column. I need to group by year,month and grab the total count of status='A' groups for that year,month combination.

Here is what I came up with, can you make it cleaner? Faster? I am curious to see the different variations people come up with.

        pg_date_part_month = sa.func.date_part('month',
                sa.func.to_timestamp(Group.register_time))
        pg_date_part_year = sa.func.date_part('year',
                sa.func.to_timestamp(Group.register_time))

group_month_select = ( 
            db.query(
                sa.sql.label('year', pg_date_part_year),
                sa.sql.label('month', pg_date_part_month),
                sa.sql.label('total', sa.func.count(Group.status))
            )   
            .filter_by(status='A')
            .group_by(pg_date_part_year)
            .group_by(pg_date_part_month)
            .group_by(Group.status)
            .order_by(pg_date_part_year)
            .order_by(pg_date_part_month)
        )
Read and Post Comments

Using SQLAlchemy Custom Types to Convert Integers to DateTime

On October 12, 2011 In python, sqlalchemy Permalink

Today I was working on fetching out some data from an existing PostgreSQL server and generating some BSON output that would later be imported in to MongoDB. One of the problems I ran in to was that I needed to format the timestamps easily for each row of data.

Searching the internet I ran across this blog post by Ralph Bean, which does just that, but at a level that was well beyond what I needed. So taking away some inspiration from Ralph's blog post, I decided to just go with a Custom Type.

from time import mktime
from datetime import datetime

class IntegerDateTime(types.TypeDecorator):
    """Used for working with epoch timestamps.

Converts datetimes into epoch on the way in.
    Converts epoch timestamps to datetimes on the way out.
    """
    impl = types.INTEGER
    def process_bind_param(self, value, dialect):
        return mktime(value.timetuple())
    def process_result_value(self, value, dialect):
        return datetime.fromtimestamp(value)

Then in my reflected table, I just override the column that holds the integer representation of the datetime I want.

group_table = sa.Table('groups', metadata,
    sa.Column('register_time', IntegerDateTime),
    autoload=True,
    include_columns=[
        'group_id',
    'register_time',
    'type'
    ],
)

Now when we query and begin to use our results, register_time will be a DateTime object making it very easy to do any timedelta arithmetic or string formatting.

Read and Post Comments

Working for Geeknet

On October 05, 2011 In python Permalink

I joined Geeknet full time this week as a Senior Software Engineer on the SourceForge team. I am really looking forward to the challenge ahead and working with everyone on the team. As part of the SourceForge team I will be helping to make SourceForge better for the current users and working on improvements that will attract new projects to SourceForge. My personal goal is to make SourceForge part of the day to day vocabulary like it used to be. I want see SourceForge back in the top three when people ask "Where is a good place to host my code?" or "If you were creating an OSS project, where would you put?". I believe that this team can do that and I am really excited for the future.

I will be candid here, when I first thought of working for Geeknet on the SourceForge team, I was thinking probably the same thing you are ... isn't that only CVS and SVN? That is just for hosting downloads right?

So as part of my initial curiosity of what people currently think of SourceForge versus what is actually happening on the site I did some googling and read up on anything that mentioned SourceForge that was less than 6-months old. I found that it wasn't so much that SourceForge wasn't keeping pace with the other options out there, but that it wasn't marketing the fact that it was keeping pace with the other options out there.

Here are a list of improvements that I didn't know about until I did some research.

  • SourceForge supports Git and Mercurial (and Bazzar).
  • You can fork and submit merge requests with Git and Mercurial.
  • You don't have to get approval for projects.
  • There is a user project space for creating miscellaneous repositories.
  • Integrated Tracker / Commit messages. [#TICKET]
  • 9 out of 10 Ninjas recommend it.

When you combine that with one of the best mirror networks out there, live IRC support, and a great community; You get a pretty nice resource. And at a price of FREE.99 (like beer), it was all I needed to know that this was where I wanted to work. Contributing to an long-time open source project that paved the path for other project hosting sites while helping to keep that open spirit alive and well.

Read and Post Comments

Scheduling Posts with Blogofile

On August 28, 2011 In python Permalink

I wanted to be able to schedule posts with blogofile and this was the quickest way I could think of to do it. If someone knows of a better way, than please comment cause I would love to read about.

My change is pretty simple, if the date in the YAML header is > than now, throw a PostProcessing exception and continue on with the next post. I added a cronjob that runs blogofile build every hour, so this solution sucks for blogs that take a long time to build or if you want precision scheduling, but my site builds fast and I am ok with an hour delay.

diff -r e370cb5a903f blog/_controllers/blog/post.py
--- a/blog/_controllers/blog/post.py    Tue Aug 23 23:16:37 2011 -0400
+++ b/blog/_controllers/blog/post.py    Sun Aug 28 19:03:28 2011 -0400
@@ -70,7 +70,14 @@
     def __str__(self):
         return repr(self.value)

+class PostProcessException(Exception):

+    def __init__(self, value):
+        self.value = value
+
+    def __str__(self):
+        return repr(self.value)
+                
 class Post(object):
     """
     Class to describe a blog post and associated metadata
@@ -179,7 +186,11 @@
             self.slug = slug

if not self.date:
-            self.date = datetime.datetime.now(pytz.timezone(self.__timezone))
+            self.date       = datetime.datetime.now(pytz.timezone(self.__timezone))
+        else:
+            if self.date > datetime.datetime.now(pytz.timezone(self.__timezone)):
+                raise PostProcessException('Post date is in the future.')
+
         if not self.updated:
             self.updated = self.date

@@ -367,7 +378,7 @@
             raise
         try:
             p = Post(src, filename=post_fn)
-        except PostParseException as e:
+        except (PostParseException,PostProcessException) as e:
             logger.warning(u"{0} : Skipping this post.".format(e.value))
             continue
         #Exclude some posts

Read and Post Comments
« Previous Page   |   Next Page »