So if you are intersted in working on a open project hosting platform that is written in Python, I encourage you to register and submit a proposal to the Apache Software Foundation for working on the Allura project. You can find the tickets that are part of the Allura GSoC 2013 on the Apache COMDEV Jira.
You can also find useful information about the GSoC on the ASF GSoC FAQ Page and on the Allura GSoC Wiki page.
]]>matplotlib - This library is a python 2D plotting library that creates great figures that you can use to provide a visual representation of many data. You can generate plots, histograms, bar charts, etc. The API for figure creation is very simple and straight forward and I have really enjoyed using it over the last few months.
2. What new programming technique did you learn in 2012
The programming I do is a pretty equal mix of brand new code and extensions to existing code. This year I took the time to learn how to better refactor existing code. That can mean a lot of things to a lot of people, but I specifically focused on refreshing my knowledge of code smells as defined in Refactoring - Fowler (1999) and taking the time to learn how to make better code from the start by re-reading Clean Code - Martin (2008).
Everytime I re-read books such as these, they almost feel completely new. The years that have passed have given me a completely different learning expereince from the exact same text.
3. Which open source project did you contribute to the most in 2012? What did you do?
Allura - To be fair, I work for SourceForge so I get to contribue to Allura as part of my day job. The contributions range from bug fixes to backend updates to tools and extensions.
Pyramid - This is the project I contributed to the most that was not part of my day job. That said, the overall level of contributions wasn't that high, just a few bug fixes, minor JSON renderer update, documentation tweaks, and some Pyramid related blog posts. You can see them here.
4. Which Python blog or website did you read the most in 2012
Reddit, but to be fair, a lot of the stories I went to on Reddit were because they were linked in the Pycoder's Weekly newsletter. So maybe Pycoder's Weekly should take the credit.
5. What are the top things you want to learn in 2013
In 2013 I hope to continue learning more about mobile development, use AngularJS for something other than a trivial demo, and continue my general quest to always be a better developer than I am today. I've also been learning a lot of nutrition and other health sciences and will probably contintue to focusing on learning more about them in 2013.
6. What is the top software, application, or library you wish someone would write in 2013
I think it would be really awesome if there was a recipe application that allowed you to enter in your macronutrient needs for the day (Protein, Fat, Carbs) and spit out a list of recipes, based on a selected meal frequency, that would make you hit all of your requirements.
Want to do your own list? here's how:
Fortunately the latest Pyramid master has new features that allow you to easily define a serialization protocol for custom objects.
We provide you with two approaches, the first one will feel familar if you have used TurboGears. The second allows you to handle the serialization of any object, even those third-party ones you didn't write.
Using the __json__ method is great when you want an easy way to serialize your own objects. All you need to do is define a __json__ method on your class and use the default json renderer on your view. Like this.
class CustomObject(object):
def __init__(self, name, email):
self.name = name
self.email = email
self.timestamp = datetime.utcnow()
def __json__(self, request):
return dict(
name=self.name,
email=self.email,
timestamp=self.timestamp.isoformat(),
)
@view_config(route_name="custom_object", renderer="json")
def custom_object(request):
from objects import CustomObject
results = dict(
count=2,
objects=[
CustomObject('Wayne Witzel III', 'wayne@pieceofpy.com'),
CustomObject('Fake Person', 'fake.person@pieceofpy.com'),
],
)
return results
You can see here, this is taking the non-serializable datetime that is part of your custom object and turning it into a serializable string using the isoformat call. The default json renderer looks for this special __json__ method. Once you have that defined, there is nothing more for us to do. As long as the return of __json__ is serializable, everything is handled for us. Even when returning lists of custom objects, like say the results of a SQL query.
Now if extending the object itself isn't desirable, you can use a custom adapater. This uses a type checking approach, that registers a serialization method for a specific type. It has a little more setup than the __json__ approach above, but is great when dealing with built-in types of third party objects.
First we create a method that knows how to serialize our object.
class ThirdPartyObject(object):
def __init__(self):
self.value = Decimal(0.1)
def third_party_adapter(obj, request):
return dict(
value=str(obj.value),
)
So here, we define a very simple adapter that knows how to deal with our Decimal value. Now in our __init__ we need to tell Pyramid about this custom method.
json_third_party = JSON()
json_third_party.add_adapter(ThirdPartyObject, third_party_adapter)
config.add_renderer('json_third_party', json_third_party)
config.add_route('third_party', '/third_party.json')
Finally we can now tell our view to use our newly registered json_third_party renderer when our view returns.
@view_config(route_name="third_party", renderer="json_third_party")
def third_party(request):
from objects import ThirdPartyObject
results = dict(
count=1,
objects=[
ThirdPartyObject(),
],
)
return results
As you can see if is very easy to configure Pyramid to JSON serialize custom objects even if you aren't the original author or don't want to modify the code of an object.
You can try out this feature by checking out the latest copy of Pyramid master from Github. A working Pyramid demo for this blog post is available here.
Reference: Renderers
]]>As part of creating this integration, I created my own scaffold to hook everything up. You can download and try the scaffold yourself from my SourceForge repository. Below I will outline some of the key areas I had to configure. I highly reccomend you install and create a sample project with the scaffold when you follow along with this blog post.
First in the main init.py of the project, I had to setup some configuration. If you have created a new project using the scaffold this will be done for you. Here are the important snippets from root init.py
# stockpot/__init__.py
from pyramid.events import NewRequest
import stockpot.models as M
def main(global_config, **settings):
# ...
config.begin()
config.scan('stockpot.models')
M.init_mongo(engine=(settings.get('mongo.url'), settings.get('mongo.database')))
config.add_subscriber(close_mongo_db, NewRequest)
# ...
def close_mongo_db(event):
def close(request):
M.DBSession.close_all()
event.request.add_finished_callback(close)
Now lets take a look at what you are getting from the models import (M). We use init mongo which does exactly what you think and then we use DBSession which I used as a name since it is familar to those who are coming from SQLalchemy. I will just show you the entire file.
# stockpot/models/__init__.py
from ming import Session
from ming.datastore import DataStore
from ming.orm import ThreadLocalORMSession
from ming.orm import Mapper
session = Session
DBSession = ThreadLocalORMSession(session)
def init_mongo(engine):
server, database = engine
datastore = DataStore(server, database=database)
session.bind = datastore
Mapper.compile_all()
# Here we just ensure indexes on all our mappers at startup.
for mapper in Mapper.all_mappers():
session.ensure_indexes(mapper.collection)
# Flush changes and close all connections
DBSession.flush()
DBSession.close_all()
from .user import User
Now I want to show you how I did the groupfinder and RequestWithAttribute. Since I imagine most of you looking to use Ming will probably have some type of user model. For the groupfinder and RequestWithAttributes the only special thing I had to do was add an extra step to convert the users id in to a proper bson.ObjectId
from pyramid.decorator import reify
from pyramid.request import Request
from pyramid.security import unauthenticated_userid
import bson
import stockpot.models as M
def groupfinder(userid, request):
userid = bson.ObjectId(userid)
user = M.User.query.get(_id=userid)
return [] if user else None
class RequestWithAttributes(Request):
@reify
def user(self):
userid = unauthenticated_userid(self)
userid = bson.ObjectId(userid)
if userid:
return M.User.query.get(_id=userid)
return None
Finally here is what my mapped User model might look like. This is a sample taken from a real project with some sensitive elements replaced and/or removed. Use it as an example, but not as a good one.
# stockpot/models/user.py
from hashlib import sha1
from datetime import datetime
from string import ascii_letters, digits
from random import choice
from ming import schema as S
from ming.orm import FieldProperty
from ming.orm.declarative import MappedClass
from pyramid.httpexceptions import HTTPForbidden
from stockpot.models import DBSession
# If you change this AFTER a user signed up they will not be able to
# login until they perform a password reset.
SALT = 'supersecretsalt'
CHARS = ascii_letters + digits
MAX_TRIES = 100
class User(MappedClass):
class __mongometa__:
session = DBSession
name = 'users'
custom_indexes = [
dict(fields=('email',), unique=True, sparse=False),
dict(fields=('username',), unique=True, sparse=False),
dict(fields=('identifier',), unique=True, sparse=False),
]
# This should feel familar to anyone coming from SQLa
_id = FieldProperty(S.ObjectId)
username = FieldProperty(str)
email = FieldProperty(str)
password = FieldProperty(str, if_missing=S.Missing)
signup_date = FieldProperty(datetime, if_missing=datetime.utcnow())
# Simple init method
def __init__(self, *args, **kwargs):
self.signup_date = datetime.utcnow().replace(microsecond=0)
self.username = kwargs.get('username')
if kwargs.get('password'):
self.password = User.generate_password(kwargs.get('password'),
str(self.signup_date))
self.email = kwargs.get('email', '{0}@example.com'.format(self.username))
# Update method is a little different than you are used to
# ensure by the time you're calling this everything is validated and safe
def update(self, *args, **kwargs):
for k,v in kwargs.items():
if k == 'password':
v = User.generate_password(v, str(self.signup_date))
setattr(self, k, v)
# Standard authenticate method
@classmethod
def authenticate(cls, login, password):
user = cls.query.find({'$or': [{'username':login}, {'email':login}]}).one()
if user:
password = User.generate_password(password, str(user.signup_date))
if password == user.password:
return user
else:
return None
# Password hashing
@staticmethod
def generate_password(password, salt):
password = sha1(password).hexdigest() + salt
return sha1(password+SALT).hexdigest()
# Username generation, the username generation is very useful if you use social auth
@staticmethod
def random_username(_range=5, prefix=''):
return prefix + ''.join([choice(CHARS) for i in range(_range)])
And there you have it, if you tie all the elements together you get a pretty easy and straightforward Ming integration with Pyramid.
]]>Here is what I came up with, can you make it cleaner? Faster? I am curious to see the different variations people come up with.
pg_date_part_month = sa.func.date_part('month',
sa.func.to_timestamp(Group.register_time))
pg_date_part_year = sa.func.date_part('year',
sa.func.to_timestamp(Group.register_time))
group_month_select = (
db.query(
sa.sql.label('year', pg_date_part_year),
sa.sql.label('month', pg_date_part_month),
sa.sql.label('total', sa.func.count(Group.status))
)
.filter_by(status='A')
.group_by(pg_date_part_year)
.group_by(pg_date_part_month)
.group_by(Group.status)
.order_by(pg_date_part_year)
.order_by(pg_date_part_month)
)
]]>Searching the internet I ran across this blog post by Ralph Bean, which does just that, but at a level that was well beyond what I needed. So taking away some inspiration from Ralph's blog post, I decided to just go with a Custom Type.
from time import mktime
from datetime import datetime
class IntegerDateTime(types.TypeDecorator):
"""Used for working with epoch timestamps.
Converts datetimes into epoch on the way in.
Converts epoch timestamps to datetimes on the way out.
"""
impl = types.INTEGER
def process_bind_param(self, value, dialect):
return mktime(value.timetuple())
def process_result_value(self, value, dialect):
return datetime.fromtimestamp(value)
Then in my reflected table, I just override the column that holds the integer representation of the datetime I want.
group_table = sa.Table('groups', metadata,
sa.Column('register_time', IntegerDateTime),
autoload=True,
include_columns=[
'group_id',
'register_time',
'type'
],
)
Now when we query and begin to use our results, register_time will be a DateTime object making it very easy to do any timedelta arithmetic or string formatting.
]]>I will be candid here, when I first thought of working for Geeknet on the SourceForge team, I was thinking probably the same thing you are ... isn't that only CVS and SVN? That is just for hosting downloads right?
So as part of my initial curiosity of what people currently think of SourceForge versus what is actually happening on the site I did some googling and read up on anything that mentioned SourceForge that was less than 6-months old. I found that it wasn't so much that SourceForge wasn't keeping pace with the other options out there, but that it wasn't marketing the fact that it was keeping pace with the other options out there.
Here are a list of improvements that I didn't know about until I did some research.
When you combine that with one of the best mirror networks out there, live IRC support, and a great community; You get a pretty nice resource. And at a price of FREE.99 (like beer), it was all I needed to know that this was where I wanted to work. Contributing to an long-time open source project that paved the path for other project hosting sites while helping to keep that open spirit alive and well.
]]>
My change is pretty simple, if the date in the YAML header is > than now, throw a PostProcessing exception and continue on with the next post. I added a cronjob that runs blogofile build every hour, so this solution sucks for blogs that take a long time to build or if you want precision scheduling, but my site builds fast and I am ok with an hour delay.
diff -r e370cb5a903f blog/_controllers/blog/post.py
--- a/blog/_controllers/blog/post.py Tue Aug 23 23:16:37 2011 -0400
+++ b/blog/_controllers/blog/post.py Sun Aug 28 19:03:28 2011 -0400
@@ -70,7 +70,14 @@
def __str__(self):
return repr(self.value)
+class PostProcessException(Exception):
+ def __init__(self, value):
+ self.value = value
+
+ def __str__(self):
+ return repr(self.value)
+
class Post(object):
"""
Class to describe a blog post and associated metadata
@@ -179,7 +186,11 @@
self.slug = slug
if not self.date:
- self.date = datetime.datetime.now(pytz.timezone(self.__timezone))
+ self.date = datetime.datetime.now(pytz.timezone(self.__timezone))
+ else:
+ if self.date > datetime.datetime.now(pytz.timezone(self.__timezone)):
+ raise PostProcessException('Post date is in the future.')
+
if not self.updated:
self.updated = self.date
@@ -367,7 +378,7 @@
raise
try:
p = Post(src, filename=post_fn)
- except PostParseException as e:
+ except (PostParseException,PostProcessException) as e:
logger.warning(u"{0} : Skipping this post.".format(e.value))
continue
#Exclude some posts
]]>The purpose of the post is to summarize briefly the steps taken to achieve the desired results of automatically updating the index information when models change. Please browse the source and clone and run the project, looking at the source as a whole will definitely give you a better start to finish understanding than just reading this blog post.
GitHub: sawhoosh
There are few parts that make up the magic of all this, but each part is really simple. First you have the index schema itself. This is setup to hold the unique ID of an item, as well as a pickled version of the uninstantiated class, and an aggregate of values (decided by you) that you want to be searchable for the item. The schema looks like this:
class SawhooshSchema(SchemaClass):
value = TEXT
id = ID(stored=True, unique=True)
cls = ID(stored=True)
In the Whoosh world this basically says store ID and CLS so it comes back with the search results, but not value. value will be what we will search over in order to find our results. We need to tell our models what attributes are important to use for indexing, I've chosen to do this with a whoosh_value property on the model itself.
class Document(Base):
__tablename__ = 'document'
__whoosh_value__ = 'id,title,content,author_id'
id = Column(CHAR(32), primary_key=True, default=new_uuid)
title = Column(String, nullable=False)
content = Column(Text, nullable=False)
author_id = Column(Integer, ForeignKey('author.id'), index=True)
def __str__(self):
return u'Document - Title: {0}'.format(self.title)
This says make the id, title, content, and author_id for document all searchable values. This happens automatically for us because our Base class has a parent class that handles all of the Whoosh work. This parent class, I've called it SawhooshBase, handles all the indexing, reindexing, and deindexing of our models. The class itself looks like this:
class SawhooshBase(object):
# The fields of the class you want to index (make searchable)
__whoosh_value__ = 'attribue,attribute,...'
def index(self, writer):
id = u'{0}'.format(self.id)
cls = u'{0}'.format(pickle.dumps(self.__class__))
value = u' '.join([getattr(self, attr) for attr in self.__whoosh_value__.split(',')])
writer.add_document(id=id, cls=cls, value=value)
def reindex(self, writer):
id = u'{0}'.format(self.id)
writer.delete_by_term('id', self.id)
self.index(writer)
def deindex(self, writer):
id = u'{0}'.format(self.id)
writer.delete_by_term('id', self.id)
Base = declarative_base(cls=SawhooshBase)
As you can see in the index method, we use the whoosh_value to create an aggregate of searchable strings and supply that to the value we defined in our search schema. We pickle the class and also store the ID. This is what later lets us easily take search results and turn them in to object instances.
Those methods above are only ever run by our SQLalchemy after_flush session event, though that is not to say you couldn't run them manually. The callback and event hook looks like this:
def update_indexes(session, flush_context):
writer = WIX.writer()
for i in session.new:
i.index(writer)
for i in session.dirty:
i.reindex(writer)
for i in session.deleted:
i.deindex(writer)
writer.commit()
event.listen(DBSession, 'after_flush', update_indexes)
The WIX object you see being used here is created much the same way you create your DBSession and you can view that in the search.py file of the project. Everything else is pretty straight forward. Call the respective indexer methods for new, dirty, and deleted items in the session.
Now we have our models all whooshified (technical term) and we want to have our users searching and displaying usable results. To start we have a simple search form that when submitted does a GET request to our search method with keywords. Now since the whoosh query parser is already smart enough to pickup things like AND and OR we don't have to do much.
The view code itself is where we use another helper method called results_to_instance (also in search.py). This method takes our search results, unpickles the class, fetches the instance from the DB and places the result in to a list. That method looks like this:
def results_to_instances(request, results):
instances = []
for r in results:
cls = pickle.loads('{0}'.format(r.get('cls')))
id = r.get('id')
instance = request.db.query(cls).get(id)
instances.append(instance)
return instances
Now this is where you see the benefit of pickling the class type with the search result. We have ready to use instances of multiple types from a single search result. We can pass them in to a template to render the results, but since we have a mix of instance types, we need a unified way to render them out without explicitly checking their type or using a series of if attribute checks. I decided I would use str on the model to be a search results safe friendly way to display the object to the user and that I would define a route_name helper method on the model so that I can use request.route_url(object.route_name()) to generate the clickable links in the results. The combination of this can be viewed below:
%if results:
No results found
%endifAnd the view itself uses some things we haven't talked about yet. QueryParser is just part of Whoosh and it is what turns your keywords in to something useful. request.ix is just a shortcut to our WIX object we created, we've added it using a custom Request class (same as we've done with db), you can see that in the security.py file of the project. From there we render the search results html and return it to our AJAX caller to inject in to the page. You can see this is also where we use the results_to_instances method discussed earlier.
@view_config(route_name='search', renderer='json', xhr=True)
def search_ajax(request):
query = request.params.get('keywords')
parser = QueryParser('value', request.ix.schema)
with request.ix.searcher() as searcher:
query = parser.parse(query)
results = searcher.search(query)
search_results_html=render('sawhoosh:templates/search/results.mako',
dict(results=results_to_instances(request, results)),
request=request)
return dict(search_results_html=search_results_html)
There you have it. A search indexer using SQLalchemy on top of Pyramid. I highly recommend you clone the git repo for this project or review the code using the Git website if you are interested in getting a start to finish feel. The project allows you to add Authors and Documents and Edit and Delete them all while updating the local search index stored in the a directory on the file system.
GitHub: sawhoosh
]]>Please use caution when reading this post. A lot of the approach and implementation here is flawed. I am keeping the post up for historical purposes, but I am currently working on a follow up post that has a much better and proper implementation of traversal for SQLalchemy models. The practice of not returning real instances as traversal expects and tightly coupling the models to the traversal method is something that is less than desirable and will lead to more pain than gain long term. That being said, some of the approaches here are a good way to learn about traversal and how one might want to use it with their data model.
When Pyramid was first being developed I was intrigued by the idea that I could create context aware views and use a host of methods to check permissions on those contexts, generate URLs based off those contexts, and auto-magically call the view required based on the context and the requested resource path.
So one of my first experiments with Pyramid was to implement proper resource urls for contexts in a RESTful fashion. Eventually I plan to do this for the entire collection as well, but for now all I need is the context level RESTful interface. The goal of which is to have URLs that go something like this.
This ends up being pretty damn simple with Pyramid and Traversal and for those of you new to traversal or even those who aren't, I highly recommend reading the Much Ado About Traversal chapter in the Pyramid documentation. Also on a side note all of the snippets from this post are part of a real project called Stockpot and the code is freely available via SourceForge.
So first step for me was to design my Root object. This is the really the foundation for traversal and determines what resources it will be able to find and how to interact with them once it finds them. My Root object is simple and looks like this.
def _owned(obj, name, parent):
obj.__name__ = name
obj.__parent__ = parent
return obj
class Root(dict):
__name__ = None
__parent__ = None
#
def __init__(self, request):
dict.__init__(self)
self.request = request
self['user'] = _owned(User, 'user', self)
This is pretty straightforward. We create a user entry point for the first call to getitem and return the User model with a name of user and the Root object as the parent.
For my Root object to really do anything useful our model class needs to do some work so that when the traversal algorithm calls getitem on our User model it actually gets something useful back. I've done this using a base class for my declarative_base call.
class StockpotBase(object):
@classmethod
def __getitem__(cls, k):
try:
result = DBSession().query(cls).filter_by(id=k).one()
result.__parent__ = result
result.__name__ = str(k)
return result
except NoResultFound, e:
raise KeyError
@classmethod
def __len__(cls):
return DBSession().query(cls).count()
@classmethod
def __iter__(cls):
return (x for x in DBSession().query(cls))
Base = declarative_base(cls=StockpotBase)
class User(Base):
__tablename__ = 'users'
__name__ = 'user'
#
def __init__(self, email, password=None, display_name=None):
self.email = email
self.password = password
self.display_name = display_name
#
id = Column(Integer, primary_key=True)
email = Column(String, nullable=False, unique=True)
password = Column(String, nullable=True)
display_name = Column(String, nullable=True)
user_groups = relation(Group, backref='user', secondary=groups)
groups = association_proxy('user_groups', 'name', creator=Group.group_creator)
recipes = relation(Recipe, backref='user')
#
def __str__(self):
return 'User(id={0}, email={1}, groups={2})'.format(self.id, self.email, self.groups)
def __repr__(self):
return self.__str__()
So that is a pretty big chunk of code so let me go through what is happening, it is rather simple. I've created StockpotBase which has the methods our traversal algorithm is going to want. I've used that as the cls for my declarative_base call so that any class that I create that inherits from Base will have all of the proper methods needed.
The getitem itself ensures that the parent is set to the generic user class and the name of the class is set to the primary key. This is important later when we start using resource_url() to generate links for us in our templates, if you consider that the urls will be generated with the pattern of /parent.name/context.name
With the Root object setup and our model "traversal enabled", we can look at how the views for this will be setup. I personally like to use the config.scan('stockpot.views') helper and use the @view_config decorator for my views. I find it cleaner and easier to to have the view_config right with the actually def.
# RESOURCE_URL = /user/id
@view_config(context=User, renderer='user/view.mako')
def get(request):
return dict(user=request.context)
# RESOURCE_URL = /user/id/edit
@view_config(name='edit', context=User, renderer='user/edit.mako')
def edit(request):
return dict(user=request.context)
So here is the default GET view. It allows anyone to use this view, but I will have a blog post about permissions with ACL and traversal later, and it uses the renderer of my user/view.mako template. Then we have the edit view which requires User:edit permissions and uses the edit.mako template. Pretty simple. Next we have the first of the JSON views (they don't have to be JSON).
@view_config(context=User, request_method='PUT', xhr=True, renderer='json')
def put(request):
user = request.context
return dict(method='PUT', user_id=user.id, email=user.email)
And the mako template jQuery for this might look something like this
$$code(lang=javascript, linenums=True) $(document).ready(function() { $('#put').click(function() { $.ajax({ url: '${request.resource_url(user)}', type: 'PUT', context: document.body, dataType: 'json', success: function(data) { console.log(data); alert('done'); } }); }); });
And that is it. You would repeat the same view pattern for request_method POST and request_method DELETE and you would have RESTful API in to your resources/models in a very clean fashion.
When a user visits the resource url a simple series of calls to getitem happens. The Root (/) object is called with 'user'. A User object with the name of 'user' and the parent of Root is returned. The User class has it's getitem called and uses the DBSession to lookup a user based on the key given. For example /user/1 (Root / User / k) would result in '1' being passed to the user objects getitem as the key. If it locates the user, it returns the instance and sets the name and parent. If you don't set the name when you call resource_url with the context, the generated URL would look read /user instead of /user/1.
There is nothing after the 1 so it looks for a generic unnamed view that handles the User context. In our case, our get method. When you add on edit, /user/1/edit it works in the same fashion, but when it tries to call getitem a second time on the User instance it will throw a key error which tells Pyramid that I am looking for a view named edit with the context of User. This traversal works the same way for the JSON calls as well.
I don't like the fact that there are extra DB calls here, but it is a trade off. Even the /user/1/edit has to make two database calls to get the KeyError and review the proper view, but as a side-effect I can do something like /user/1/collection/1 and get the specific item of the collection owned by the user. That extends to edits as well ... /user/1/collection/1/edit. Overall I like how this pattern has evolved in my application, but would appreciate any feedback or suggested improvements to what I've done so far.
]]>