It's been a while since I wrote a blog article, so I'm using this one-liner as an excuse. In case you're new here, I do a lot of Python development. In the world of Python, you need to have a special file in a directory before you can use Python code within that directory. Yeah, yeah... that's not exactly the clearest way to explain things, but it'll have to do.

This special file is called __init__.py. Having this file in a directory that contains Python code turns that directory into what's called a "python package." We like Python packages. They make our lives so much fun!

Anyhoo, I was working on a project last night, and I wanted to create a bunch of placeholder directories that I plan to use later on. I plan on keeping Python code in these directories, so putting the special __init__.py file in them is what I was looking to do. I didn't want to have to create the __init__.py file in each directory manually, or copy/paste the file all over the place, so I investigated a way to do it quickly from the command line.

One of my buddies brought an interesting command to my attention recently: xargs. I had seen it before in various tutorials online, but I never bothered to learn about it. This seemed like as good a time as any, so I started playing. The result of my efforts follows:

find . -type d | xargs -I {} touch {}/__init__.py

What it does is:

• recursively finds (find) all directories (-type d) within the current directory (.)
• pipes (|) each directory to xargs, which makes sure that the __init__.py file exists in each one (touch {}/__init__.py)
• the -I {} tells xargs what to use as a placeholder when considering each directory found by the find command

Turns out that xargs can be used for all sorts of good stuff. My friend brought it up as a way to get rid of those nasty .svn directories on his path to "Mercurial bliss."

find . -name ".svn" -type d | xargs -I {} rm -Rf {}

How beautiful!

I spent a little bit of time last night and this morning trying to find some examples for AES encryption using Python and PyCrypto. To my surprise, I had quite a difficult time finding an example of how to do it! I posted a message on Twitter asking for any solid examples, but people mostly just responded with things I had seen before--the libraries that do the encryption, not examples for how to use the libraries.

It wasn't long after that when I just decided to tackle the problem myself. My solution ended up being pretty simple (which is probably why there weren't any solid examples for me to find). However, out of respect for those out there who might still be looking for a solid example, here is my solution:

#!/usr/bin/env python

from Crypto.Cipher import AES
import base64
import os

# the block size for the cipher object; must be 16 per FIPS-197
BLOCK_SIZE = 16

# the character used for padding--with a block cipher such as AES, the value
# you encrypt must be a multiple of BLOCK_SIZE in length.  This character is
# used to ensure that your value is always a multiple of BLOCK_SIZE
PADDING = '{'

# one-liner to sufficiently pad the text to be encrypted
pad = lambda s: s + (BLOCK_SIZE - len(s) % BLOCK_SIZE) * PADDING

# one-liners to encrypt/encode and decrypt/decode a string
# encrypt with AES, encode with base64
EncodeAES = lambda c, s: base64.b64encode(c.encrypt(pad(s)))
DecodeAES = lambda c, e: c.decrypt(base64.b64decode(e)).rstrip(PADDING)

# generate a random secret key
secret = os.urandom(BLOCK_SIZE)

# create a cipher object using the random secret
cipher = AES.new(secret)

# encode a string
encoded = EncodeAES(cipher, 'password')
print 'Encrypted string:', encoded

# decode the encoded string
decoded = DecodeAES(cipher, encoded)
print 'Decrypted string:', decoded

Edit: thanks to John and Kaso for their suggestions, though John's didn't seem to work for me (?)

Edit 2015.12.14: thanks to Stephen for pointing out that the block size for AES is always 16, and the key size can be 16, 24, or 32. See FIPS-197 for more details.

If you plan to use this script, you'll need to have PyCrypto installed on your computer. I have had a difficult time finding this for Windows in the past, so I will mirror the installer that I found over here: http://jintoreedwine.wordpress.com/2008/07/20/python-25-and-encryption-pycrypto-under-windows/. I haven't tried it on Mac OS X yet, but it should be fairly simple to install it. Same goes for Linux.

The output of the script should always change with each execution thanks to the random secret key. Here's some sample output:

$ python aes_encryption.py
Encrypted string: aPCQ8v9WzLM/JusrJPS19K8uUA/34Xiu/ZR+arzl1oM=
Decrypted string: password

$ python aes_encryption.py
Encrypted string: F0cp4hMk8RXjcww270leHnigH++yqysIyPy8Em/qEbI=
Decrypted string: password

$ python aes_encryption.py
Encrypted string: 7gH2QCIPOxXVBjTXrMmdgU2l7Iku5Lch5jpG9OScGZw=
Decrypted string: password

$ python aes_encryption.py
Encrypted string: oJUq0/XHdmYgC3ILgFgF6Tpuo8ZhoEHN9wmnuYvV58Y=
Decrypted string: password

If the comments in the script aren't explanatory enough, please comment and ask for clarification. I will offer any that I am capable of, and I invite others to do the same.

I'd like to take this opportunity to officially announce my latest little side project: django-ittybitty! Some of you out there might not find this to be a useful application, but I hope others will enjoy it.

Many of you are familiar with the URL-shortening sites like http://tinyurl.com/, http://is.gd/, http://cli.gs/, and whole slew of others. These sites are all fine and dandy, right? Wrong! What happens when those sites have downtime and potential visitors to your site never get to your site because the URL-shortening site is down? You lose traffic. That's not good, in case you were unsure about it.

That is why I made this application. It allows you to have short URLs for any and every page on your Django site. No more need to rely on 3rd party servers to translate short URLs to real URLs on your site. So long as your pony-powered site is up and running, your visitors will be able to use URLs generated by this application to get anywhere on your site. All you need to do to make this work is download and install the application, add a middleware class to your MIDDLEWARE_CLASSES, and then use a simple template tag to generate a short URL for any given page.

django-ittybitty will keep track of the number of times a particular "itty bitty URL" has been used to access your site. I suppose some people will find that useful, but it's hardly a true metric for your "most popular" pages.

The algorithm behind this application is very simple, but it can potentially handle around 18,446,744,073,709,551,615 shortened URLs in 64 characters or fewer, neglecting the 'http://www.....' for your site (good luck getting your database to play well with that many records, much less storing them on a server :)).

For more information, please check out the project pages and enjoy:

• http://code.google.com/p/django-ittybitty/
• http://pypi.python.org/pypi/django-ittybitty

For those who are interested, here are some code samples for how to use django-ittybitty:

{% extends 'base.html' %}
{% load ittybitty_tags %}

{% block content %}
<a href="{% ittybitty_url %}">Link to this page!</a>
{% endblock %}

or:

{% extends 'base.html' %}
{% load ittybitty_tags %}

{% block content %}
{% ittybitty_url as ittybitty %}
<a href="{{ ittybitty.get_shortcut }}">Link to this page!</a>
{% endblock %}

or:

{% extends 'base.html' %}
{% load ittybitty_tags %}

{% block content %}
{% ittybitty_url as ittybitty %}
{% with ittybitty.get_shortcut as short_url %}
<a href="{{ short_url }}">Link to this page!</a>
<a href="{{ short_url }}">Link to this page again!</a>
<a href="{{ short_url }}">Link to this page one more time!</a>
{% endwith %}
{% endblock %}

Enjoy!

I just love the Django community. So many good times. I hope that the rest of you find some form of entertainment in the next part of this article.

This morning I woke up to a humorous post by Eric Walstad on the django-users mailing group. It discussed a story about his 9-year-old daughter who has seen the light with Django. She apparently fully understands what Django is capable of and how amazing it truly is. Here is her version of what Django, embodied as a "critter," can do:

Django is a computer programming critter. He is loyal only to computer programmers and does all their work. He types with the ball on the end of his tail, at the speed of light. He beeps when his work is done and when you take him home, he flies around the house, doing all your chores. He's a helpful little fellow.

That just about sums it all up! Django rocks. We already have Pony Power to get us through the day, but when you put Pony Power and the Django Critter together, this is what you get:

Oh man!! Can you feel it? I sure can. Django is amazing, and anyone who's not using it is missing out.

(disclaimer: the characters in the image above remain the property of their respective owners)

I just had a buddy of mine ask me to write a script that would send an e-mail to you whenever you get an "@reply" on Twitter. I've recently been doing some work on a Twitter application, so I feel relatively comfortable with the python-twitter project to access Twitter. It didn't take very long to come up with this script, and it appears to work fine for us (using a cronjob to run the script periodically).

I thought others on the Internets might enjoy the script as well, so here it is!

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
A simple script to check your Twitter account for @replies and send you an email
if it finds any new ones since the last time it checked.  It was developed using
python-twitter 0.5 and Python 2.5.  It has been tested on Linux only, but it
should work fine on other platforms as well.  This script is intended to be
executed by a cron manager or scheduled task manager.

Copyright (c) 2009, Josh VanderLinden
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

- Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
- Neither the name of the organization nor the names of its contributors may
be used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""

import twitter
import ConfigParser
import os
import sys
from datetime import datetime
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText
from email.Utils import formatdate

# get the user's "home" directory
DIRNAME = os.path.expanduser('~')
CONFIG = os.path.join(DIRNAME, '.twitter_email_replies.conf')
FORMAT = '%a %b %d %H:%M:%S +0000 %Y'
REPLY_TEMPLATE = """%(author)s said: %(text)s
Posted on %(created_at)s
Go to http://twitter.com/home?status=@%(screen_name)s%%20&in_reply_to_status_id=%(id)s&in_reply_to=%(screen_name)s to post a reply
"""

# sections
AUTH = 'credentials'
EXEC = 'exec_info'
EMAIL = 'email_info'

# make the code a bit "cleaner"
O = lambda s: sys.stdout.write(s + '\n')
E = lambda s: sys.stderr.write(s + '\n')
str2dt = lambda s: datetime.strptime(s, FORMAT)

def get_dict(status):
    my_dict = status.AsDict()
    my_dict['screen_name'] = my_dict['user']['screen_name']
    my_dict['author'] = my_dict['user']['name']
    return my_dict

def main():
    O('Reading configuration from %s' % CONFIG)
    parser = ConfigParser.SafeConfigParser()
    config = parser.read(CONFIG)

    # make sure we have the proper sections
    if not parser.has_section(AUTH): parser.add_section(AUTH)
    if not parser.has_section(EMAIL): parser.add_section(EMAIL)
    if not parser.has_section(EXEC): parser.add_section(EXEC)

    try:
        # get some useful settings from the configuration file
        username = parser.get(AUTH, 'username')
        password = parser.get(AUTH, 'password')

        to_address = parser.get(EMAIL, 'to_address')
        from_address = parser.get(EMAIL, 'from_address')
        smtp_server = parser.get(EMAIL, 'smtp_server')
        smtp_user = parser.get(EMAIL, 'smtp_user')
        smtp_pass = parser.get(EMAIL, 'smtp_pass')

        if '' in [username, password, to_address, from_address, smtp_server]:
            raise Exception('Not configured')
    except Exception:
        E('Please configure your credentials and e-mail information in %s!' % CONFIG)

        # create some placeholders in the configuration file to make it easier
        sections = {
            AUTH: ('username', 'password'),
            EMAIL: ('to_address', 'from_address', 'smtp_server', 'smtp_user', 'smtp_pass')
        }

        for section in sections.keys():
            for opt in sections[section]:
                if not parser.has_option(section, opt):
                    parser.set(section, opt, '')
    else:
        # determine the last time we checked for replies
        try:
            last_check = str2dt(parser.get(EXEC, 'last_run'))
        except ConfigParser.NoOptionError:
            last_check = datetime.utcnow()
        last_check_str = last_check.strftime(FORMAT)

        info = 'Fetching updates for %s since %s' % (username,
                                                       last_check_str)
        O(info)

        # attempt to connect to Twitter
        api = twitter.Api(username=username, password=password)

        # not using the `since` parameter for more backward-compatibility
        timeline = api.GetReplies()
        new_replies = []
        for reply in timeline:
            post_time = str2dt(reply.GetCreatedAt())
            if post_time > last_check:
                new_replies.append(reply)

        count = len(new_replies)
        if count:
            # send out an email for this user
            O('Found %i new replies... sending e-mail to %s' % (count, to_address))
            reply_list = '\n\n'.join([REPLY_TEMPLATE % get_dict(r) for r in new_replies])
            is_are = 'is'
            plural = 'y'
            if count != 1:
                is_are = 'are'
                plural = 'ies'

            params = {
                'is_are': is_are,
                'count': count,
                'replies': plural,
                'username': username,
                'reply_list': reply_list,
                'last_check': last_check_str
            }

            text = """There %(is_are)s %(count)i new @repl%(replies)s for %(username)s on Twitter since %(last_check)s:

%(reply_list)s""" % params

            # compose the e-mail
            msg = MIMEMultipart()
            msg['From'] = from_address
            msg['To'] = to_address
            msg['Date'] = formatdate(localtime=True)
            msg['Subject'] = 'New @Replies for %s' % username
            msg.attach(MIMEText(text))

            # try to send the e-mail message out
            email = smtplib.SMTP(smtp_server)
            if smtp_user and smtp_pass:
                email.login(smtp_user, smtp_pass)
            email.sendmail(from_address,
                           to_address,
                           msg.as_string())
            email.close()

        # save the current time so we know where to pick up next time
        parser.set(EXEC, 'last_run', datetime.utcnow().strftime(FORMAT))

    # write the config
    O('Saving settings...')
    out = open(CONFIG, 'wb')
    parser.write(out)
    out.close()

if __name__ == '__main__':
    main()

Feel free to copy this script and modify it to your desires. Also, please comment if you have issues using it.

Yesterday I had one of my coworkers ask me what I thought to be a simple Django question: "in the admin pages im trying to show fields from different tables but it wont let me." I clarified the problem with this chap, and eventually suggested using something like this:

from django.contrib import admin
from project.app.models import AwesomeModel

class AwesomeModelAdmin(admin.ModelAdmin):
    list_display = ('fk_field__fk_attr1', 'fk_field2__fk_attr')

admin.site.register(AwesomeModel, AwesomeModelAdmin)

from django.contrib import admin

def mygetattr(obj, hier):
    """
    Recursively attempts to find attributes across Django relationships.
    """
    if len(hier):
        return mygetattr(getattr(obj, hier[0]), hier[1:])
    return obj

def dynamic_attributes(self, attr, *args, **kwargs):
    """
    Retrieves object attributes.  If an attribute contains '__' in the name,
    and the attribute doesn't exist, this method will attempt to span Django
    model relationships to find the desired attribute.
    """
    try:
        # try to get the attribute the normal way
        return super(admin.ModelAdmin, self).__getattribute__(attr, *args, **kwargs)
    except AttributeError:
        # the attribute doesn't exist for the object.  See if the attribute has
        # two underscores in it (but doesn't begin with them).
        if attr and not attr.startswith('__') and '__' in attr:
            # it does!  make a callable for the attribute
            new_attr = lambda o: mygetattr(o, attr.split('__'))

            # add the new callable to the object's attributes
            setattr(self, attr, new_attr)

            # return the callable
            return new_attr

# override the __getattribute__ method on the admin.ModelAdmin class
admin.ModelAdmin.__getattribute__ = dynamic_attributes

from django.contrib import admin
from project.app.models import AwesomeModel

class AwesomeModelAdmin(admin.ModelAdmin):
    pass

admin.site.register(AwesomeModel, AwesomeModelAdmin)
AwesomeModelAdmin.list_display = ('fk_field__fk_attr1', 'fk_field2__fk_attr')

There are a lot of us out there who build these reusable, pluggable Django applications and share them with the world. Some of the applications are more useful than others. Personally, I think my applications tend to fall in the less useful category, but I like to build and release them nonetheless.

from django.core.urlresolvers import reverse
admin_url = reverse('admin_pendulum_pendulumconfiguration_add')

[<RegexURLPattern admin_pendulum_pendulumconfiguration_changelist ^$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_add ^add/$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_history ^(.+)/history/$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_delete ^(.+)/delete/$>,
 <RegexURLPattern admin_pendulum_pendulumconfiguration_change ^(.+)/$>]

from django.contrib.auth.views import login
from django.contrib.sites.models import Site
from django.core.urlresolvers import reverse
from django.http import HttpResponseRedirect
from django.conf import settings
from pendulum.models import PendulumConfiguration

SITE = Site.objects.get_current()
admin_url = reverse('admin_pendulum_pendulumconfiguration_add')
login_url = getattr(settings, 'LOGIN_URL', '/accounts/login/')

class PendulumMiddleware:
    """
    This middleware ensures that anyone trying to access Pendulum must be
    logged in.  If Pendulum hasn't been configured for the current site, the
    staff users will be redirected to the page to configure it.
    """
    def process_request(self, request):
        try:
            SITE.pendulumconfiguration
        except PendulumConfiguration.DoesNotExist:
            # this will force the user to configure pendulum if they're staff
            if request.user.has_perm('add_pendulumconfiguration') and \
                request.path not in (admin_url, login_url):
                # leave the user a message
                request.user.message_set.create(message='Please configure Pendulum for %s' % SITE)

                return HttpResponseRedirect(admin_url)
        else:
            entry_url = reverse('pendulum-entries')

            if request.path[:len(entry_url)] == entry_url and request.user.is_anonymous():
                if request.POST:
                    return login(request)
                else:
                    return HttpResponseRedirect('%s?next=%s' % (login_url, request.path))

I just read an interesting article on how to manage geeks, and I wholeheartedly endorse it.

A couple of my favorites:

8. Include them in IT related decisions. Never make decisions without consulting geeks. Geeks usually know the technical side of the business better than the manager, so making a technical decision without consulting them is one of the biggest mistakes a leader can make.

10. Remember that geeks are creative workers. Programming and system analysis are creative processes, not an industrial one. Geeks must constantly come up with solutions to new problems and rarely ever solve the same problem twice. Therefore they need leeway and flexibility. Strict dress codes and too much red tape kill all innovation. They also need creative workspace surroundings to avoid "death by cubicle."

Hah. Death by cubicle... that's great. Finally:

Geeks don't like dead weight. If you have any, get rid of it, and your team will be better off. Teams work best when everyone is pulling their weight.

Managers beware!

Pretty much anyone who's been doing any Web development in the last few years probably prefers to use Firefox because of the incredibly powerful extensions it offers. Among the extensions I hear most Web developers complain about not having in other browsers are Web Developer and Firebug. Several people feel that they could get by with another browser (such as Google Chrome) if it only had Firebug.

Well, my friends, the trusty folks who built Firebug actually offer their amazing product for other browsers! It goes by the name of "Firebug Lite." I'm not sure exactly how long this has been around, but the earliest date I can find suggests that it was released to the public in July of 2008.

I happened upon this utility while perusing Django Snippets the other day. A member by the username of jfw posted a middleware which injects the Firebug Lite utility into a response when you're not using Firefox and when your site is in debug mode. I've found it to be quite useful. I hope you all do too!!

This article is more about basic Python than anything else, but I hope it will help some newbies out there get further along faster. I should note that I learned Python by learning Django, so I don't exactly have the best foundation in Python. I think it's getting better though :)

Anyway, there are many occasions when I find myself creating applications that have custom settings which can be defined in a Django project's settings.py file. Sometimes I want to force the developer to define the setting (such as when a third-party API key is required). Other times I want to have sensible defaults which can easily be overridden by the developer if they desire. I struggled for a while to find an elegant solution for providing such default values for my custom settings.

I first tried something like this:

1
2
3
4
5
6

from django.conf import settings

try:
    MY_CUSTOM_SETTING = settings.MY_CUSTOM_SETTING
except AttributeError:
    MY_CUSTOM_SETTING = 'default value'

That seemed to work well enough, but it just felt dirty to me. After a while of doing thing that way, I started doing it like this:

1
2
3
4
5
6

from django.conf import settings

if hasattr(settings, 'MY_CUSTOM_SETTING'):
    MY_CUSTOM_SETTING = settings.MY_CUSTOM_SETTING
else:
    MY_CUSTOM_SETTING = 'default value'

Despite being essentially the same, it felt slightly better than the try..except method. I think I stuck with this method for a bit longer than the first method I mentioned. Finally, I found out that Python's built-in getattr function allows you to specify a default value if it fails to retrieve the value of the attribute you want:

1
2
3

from django.conf import settings

MY_CUSTOM_SETTING = getattr(settings, 'MY_CUSTOM_SETTING', 'default value')

So far, this is the most elegant and efficient method I've found for allowing custom settings with default values. Perhaps there is a better way, and I'd be delighted if someone would enlighten me and the rest of the readers.