I've just tagged Gaebar Beta 3 (version 0.3) in GitHub.

This is a bugfix release, with changes almost entirely submitted by or suggested by the community.

Thanks go to gumptionthomas, Martin Kleppmann, Tony Andrews, Rafal Jonca, and, of course, Guido van Rossum, for your patches and feedback.

Changes in Beta 3

• Merged gumptionthomas's pull request (commit)
  • Fixed unicode and escape string issues with key_names
  • Properly reference InternalError as db.InternalError
• Applied patch by Martin Kleppmann (commit)
• Backup doesn't blow up when there are non-backup files/folders in the backups folder. (e.g., if your project uses SVN)
• Updated the readme.txt based on Guido's feedback (commit)
• Added note to readme.txt regarding getting more detailed exceptions (thanks to Tony Andrews; commit).
• Applied patch by Rafal Jonca (commit):
  • Key names may now contain unicode
  • ReferenceProperties can now be created with keys of parents, not just model instances
  • Now using global datastore's delete, put, and get methods instead of model's

See the original announcement for a screencast and general information on Gaebar.

Please do keep your feedback coming and, of course, if you have pull requests, patches, or suggestions, please send them my way (and thank you!)

I just posted my slides from last night's talk at the London Django User Group on SlideShare (and you can view them above).

A big thank you to Rob and the DJUGL team for organizing the event and kindly inviting me over to speak. I had a great time and it seems like the audience enjoyed it too! :)

I'm happy to announce that there's quite an important update to Gaebar that brings with it some essential bug fixes and should help shave quite a bit of time off of your restores.

I've also created a new project called gaebar-gaed-skeleton on GitHub that gives you an empty Google App Engine Django (Helper) project with Gaebar pre-installed to start you off when creating your own applications.

Here's a list of changes in Beta 2:

• Fixed: All required properties, including references should work correctly now. Required properties would raise an exception during the restore process in Beta 1.
• Refactored: We are no longer using actual entity references (which was unnecessary) but keys to create reference properties. This means that creation order of references doesn’t matter any more (thanks to Pete Koomen who explained all this to me in an email ages ago; it only just sunk in, Pete!)
• New feature: Due to above changes, the restore process is now one-pass, not two. This means that restores should now take considerably less time.
• Refactored: I moved the functional tests out to their own GitHub module since they were being shared by both the gaebar-gaed and gaebar-aep test apps. This will make maintaining them much easier.

I want to thank Jonathan Ricketson for alerting me in the comments earlier today that the Beta 1 release was not handling required properties correctly (thanks, Jonathan). I also want to thank Thomas Bohmbach, Jr. for providing feedback and helping me test.

You can download Gaebar Beta 2 and the other Gaebar projects from my GitHub account.

If you'd like to learn more about Gaebar, please see my original announcement, which contains a screencast showing Gaebar (albeit Beta 1) in use.

As always, I value your bug reports, comments, and suggestions so please feel free to leave a comment and let me know of your experiences with Gaebar.

Update: Beta 2 Released! See the Gaebar Beta 2 announcement.

Here's my Christmas present for the Google App Engine community: Google App Engine Backup and Restore (or Gaebar, for short).

Gaebar is an easy-to-use, standalone Django application that you can plug in to your existing Google App Engine Django or app-engine-patch-based Django applications on Google App Engine to give them datastore backup and restore functionality.

For a quick overview of Gaebar, watch the screencast, above. For the impatient, links to the project pages where you can download Gaebar follow.

Downloads

Gaebar is hosted on GitHub. You can either download archives or clone the repository (or install Gaebar as a git submodule) via git. Alongside the Gaebar project itself are two sample applications — one built on Google App Engine Django and the other on app-engine-patch — that contain the Gaebar functional test suite. The functional test suite tests every datatype supported by Google App Engine as well as references, Expandos, and ancestor relationships.

Please make sure you read the readme files after downloading the projects for installation and usage instructions.

• Gaebar: http://github.com/aral/gaebar/tree/master
• Gaebar-gaed: http://github.com/aral/gaebar-gaed/tree/master
• Gaebar-aep: http://github.com/aral/gaebar-aep/tree/master

What you can do with Gaebar

• Backup your deployment application's datastore for safekeeping.
• Restore that backup on your local development server for testing with real data during development.
• Restore that backup to a different Google App Engine application and use that application as a staging application much in the same way you would use a staging server in traditional development.

(You can, of course, also backup and restore your local development datastore as well as your staging application, etc.)

I've got a huge datastore, will Gaebar work for me?

Congratulations on the impressive size of your datastore! The answer should be "yes!" The largest datastore I've tested it with is the <head> web conference datastore. The latest backup had 18, 969 rows from 14 models backed up into 225 code shards.

What's a staging application?

A staging application is a new Google App Engine concept made possible by Gaebar. Basically, if your application is myapp.appspot.com, you can use a separate application (say, myapp-staging.appspot.com) in the same way as you would use a staging server in traditional development.

Your staging application can let you try out new features and test with real data without having your users see your changes until you are ready to deploy to your main application.

In fact, it's a perfect staging environment since it is identical to your deployment environment. For an example of this, see the screencast.

How Gaebar works

Gaebar backs up the data in your datastore to Python code. It restores your data by running the generated Python code.

Since a backup is a long running process, and since Google App Engine doesn't support long-running processes, Gaebar fakes a long running process by breaking up the backup and restore processes into bite-sized chunks and repeatedly hitting the server via Ajax calls.

By default, Gaebar backs up 5 rows at a time to avoid the short term CPU and 10-second call duration quotas and splits the generated code into code shards of approx. 300KB to avoid the 1MB limit on objects. You can change these defaults in the views.py file if your app has higher quotas and you want faster backups and restores.

Once a backup of a remote server is complete, Gaebar automatically hits your local development server. From there on, the local development server makes a series of calls to the remote server to download the backup files (code shards) from the remote server. Once the backup is complete, you will see a new backup folder in gaebar/backups with the contents of your backup.

Here's an example of some generated backup code from the Google App Engine Django test application (updated for the upcoming Beta 2 release):

import pickle
from google.appengine.api.datastore import datastore_types
from app1.models import *
from app2.models import *
 
def row_0(app_name):
	existing_entity = Profile.get(datastore_types.Key.from_path('Profile', 1, _app=app_name))
	if existing_entity:
		existing_entity.delete()
	profile_0 = Profile(key_name="id1", friends = pickle.loads('(lp0\n.'), in_relationship_with = pickle.loads('N.'), full_name = pickle.loads('VPaul Booth\np0\n.'))
	profile_0.put()
 
def row_1(app_name):
	existing_entity = Profile.get(datastore_types.Key.from_path('Profile', 2, _app=app_name))
	if existing_entity:
		existing_entity.delete()
	profile_1 = Profile(key_name="id2", full_name = pickle.loads('VAral Balkan\np0\n.'), friends = [datastore_types.Key.from_path('Profile', u'stephalicious', _app=app_name), datastore_types.Key.from_path('Profile', 'id1', _app=app_name)], in_relationship_with = datastore_types.Key.from_path('Profile', u'stephalicious', _app=app_name))
	profile_1.put()

To restore, you simply deploy your application, along with the backup folder, to your deployment environment and hit the Restore button in Gaebar. (If you have a large datastore both the backup and restore processes will take a long time, especially when restoring to a local development server.)

The restore process simply calls each of the generated row functions and each row function restores a single row into the datastore.

A note on the screencast

When mentioning how to install Gaebar, I left out that you also need to add the URL mapping for Gaebar to your application's urls.py. That, along with the rest of the installation instructions are in the readme.txt file, which I highly recommend that you peruse.

Have your say!

As in all things, my approach to blog posts is that they should evolve over time and your feedback is invaluable in achieving this by helping me fix factual errors, fill in details, and expand the original post.

What do you think of Gaebar? Have you run into any issues that need fixing? Do you have other suggestions on how to improve it? Or do you, perhaps, have a patch to send me that adds Webapp support or some other feature? Leave me a comment and let me know!

Gaebar is a Naklab™ production released under GNU GPL v3 and sponsored by the <head> web conference.

To create a new Django application for Google App Engine using Google App Engine Helper, start by exporting the latest Google App Engine Helper:

svn export http://google-app-engine-django.googlecode.com/svn/trunk/ ~/myapp

To create a django.zip file to use with the latest Google App Engine:

1. Export the latest Django trunk

   svn export http://code.djangoproject.com/svn/django/trunk/ django

2. Follow steps #2 and #3 in Dan's instructions under the "Archiving Django 1.0" section of his article
3. Add contrib/auth:

   zip -r django.zip django/contrib/__init__.py \
   	                  django/contrib/auth

4. Add contrib/sessions:

   zip -r django.zip django/contrib/__init__.py \
   	                  django/contrib/sessions

5. Copy the django.zip file into the root folder of your new app:

   cp django.zip ~/myapp

6. Uncomment the sessions middleware or auth middleware lines in settings.py file in your app so that it looks like:

   MIDDLEWARE_CLASSES = (
   	    'django.middleware.common.CommonMiddleware',
   	    'django.contrib.sessions.middleware.SessionMiddleware',
   	    'django.contrib.auth.middleware.AuthenticationMiddleware',
   	#    'django.middleware.doc.XViewMiddleware',
   	)

That's it! To test it out:

./manage.py runserver

Hit http://localhost:8000/ and you should see the default Django welcome page.

Update: Don't make the mistake of adding django.contrib.auth or django.contrib.sessions to your INSTALLED_APPS, even though you are including those apps in the django.zip file. If you do, ./manage.py runserver will work but ./manage.py shell and ./manage.py flush will return errors.

You can build cross-platform GUI applications in Python using a library called PyQt, a set of Python bindings for Nokia's Qt application framework. Using PyQt, you can create Python applications that look and feel like native applications regardless of whether they happen to be running on Linux, OS X, or Windows.

The application files themselves, however, do not look like native executables; they are simply Python files and require Python, PyQt, and other dependencies to be installed to run. Needless to say, you cannot distribute your applications in this manner unless your target audience is entirely comprised of alpha geeks. To provide a usable user experience that is consistent with native applications, you can bundle PyQt applications into native executables for each of the platforms you're supporting. When deploying on OS X, for example, you can bundle your PyQt application into a standalone OS X application bundle. This process, which involves the use of the excellent py2app application, comes with several gotchas that I hadn't bargained on.

Specifically, you cannot build standalone application bundles with the system python in OS X using py2app. If you want to build standalone PyQt apps, I recommend setting up Python, PyQt, etc., under MacPorts.

If you're comfortable with Terminal and don't need to be spoon fed, here's a quick overview of how to get your system in shape to compile standalone OS X application bundles from PyQt apps using py2app on Leopard 10.5.5 (which happens to be the only operating system version I've tested this on; this may work on other versions of OS X but YMMV).

(If you need step-by-step instructions, see the Macports installation details section, below).

Setting up MacPorts

1. Install MacPorts. You will be installing Python, PyQt, py2app and everything else you need under MacPorts and using that environment for everything.
2. Install the following portfiles: python25, py25-macholib-devel, py25-sip, py25-pyqt4, py25-py2app-devel, py25-pyqt, python_select.

Note that I am installing the -devel version of py25-py2app, as well as py25-macholib-devel on purpose. I couldn't get it to work with the stable versions. You may have to deactivate py25-macholib if it gets installed automatically as a dependency for py25-py2app-devel (this is what I did, but I'm assuming that installing py25-macholib-devel before py25-py2app-devel should mean that you won't have to).

Deploying a PyQt application as a Mac OS X application bundle

Once you've successfully installed everything under MacPorts, change your environment to use the version of Python in Macports via python_select:

sudo python_select python25

Once you're done, create a setup.py file for py2app. Something along the lines of the following example:

from setuptools import setup
 
APP = ['myapp.pyw']
OPTIONS = {'argv_emulation': True, 'includes': ['sip', 'PyQt4._qt']}
 
setup(
    app=APP,
    options={'py2app': OPTIONS},
    setup_requires=['py2app'],
)

Note that the includes array contains sip and PyQt4._qt. Those two entries are essential if you're bundling a PyQt app.

Finally, issue the following command to create a standalone OS X application bundle from your application:

python setup.py py2app

That's it. Double click on your app, which you can find in the dist folder and it should run.

If you run into any problems, see the My Macports setup section, below, for an exact dump of my Macports portfiles that you can use to compare (obtained using port installed).

Read on if you want to the full story and details of how I got my system up and running with MacPorts to deploy OS X app bundles using PyQt and py2app. I'm including this with as much information (error messages, etc., as possible so that others may find this post if they're similarly stuck.) Or, if you're not interested, you can jump to the Reference section.

The gory details

My initial attempt at creating a standalone PyQt app with py2app resulted in the app crashing on launch.

The error message:

Fatal Python error: Interpreter not initialized (version mismatch?)

Judging by the error message, and emboldened by a forum exchange between Derrick Hendricks and Christopher Baker I guessed that the conflict was due to my having two versions of Python installed (the System python and a 2.5 through MacPorts). So I removed MacPorts completely, ran py2app again and the app launched but with a different error:

ImportError: '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload/PyQt4/QtCore.so' not found

Searching for a solution, I stumbled upon a thread where someone who ran into the same issue suggested adding the following line to __boot__.py in the Contents/Resources folder of the application bundle:

sys.path = [os.path.join(os.environ['RESOURCEPATH'], 'lib', 'python2.5', 'lib-dynload')] + sys.path

Unfortunately, doing that caused the app to crash at launch.

It was only when I actually read the output from py2app that I noticed that it appeared to be creating a semi-standalone app instead of the standalone app that I asked it to create. In retrospect, I should have realized this from the very first error since I should not have gotten a version mismatch error on the Python interpreter if it had been correctly embedded in the app bundle.

The message in the py2app log read:

creating . . ./bdist.macosx-10.5-i386/python2.5-semi_standalone/app/collect/PyQt4

A Google search later, I had a thread where someone else was having the same problem and I found a definitive answer from Bob Ippolito, one of my Python heroes and the author of py2app, stating the following:

[py2app] refuses to build a standalone bundle out of the system framework.

OK, that's something that should be in the docs page on H1! (I couldn't find any mention of it in the docs at all.)

This makes sense, as David Warde-Farley states in a forum posting:

First of all, if you want to build bundle .app's that are redistributable, it's a good idea not to use the system Python, especially if you want to support Tiger (10.4). There are things in Apple's Python (i.e. DTrace support) that I imagine won't play nicely on an older OS.

At this point, I knew that I had to install a separate Python and decided to use MacPorts. It did take a while to install everything but that approach paid off in the end.

Reference

My Macports setup

Here's a full list of my MacPorts setup (which I set up specifically to compile PyQt apps with py2app, so there aren't any superfluous packages in there) which you can compare to yours if something isn't working:

• autoconf @2.63_0 (active)
• automake @1.10.1_0 (active)
• dbus @1.2.4_2 (active)
• expat @2.0.1_0 (active)
• gettext @0.17_3 (active)
• gperf @3.0.3_0 (active)
• help2man @1.36.4_1 (active)
• jpeg @6b_3 (active)
• lcms @1.17_0 (active)
• libiconv @1.12_0 (active)
• libmng @1.0.10_0 (active)
• libpng @1.2.33_0 (active)
• libtool @1.5.26_0 (active)
• m4 @1.4.12_1 (active)
• ncurses @5.7_0 (active)
• ncursesw @5.7_0 (active)
• openssl @0.9.8i_0 (active)
• p5-locale-gettext @1.05_0 (active)
• perl5 @5.8.8_0+darwin_9
• perl5.8 @5.8.8_3+darwin_9 (active)
• pkgconfig @0.23_1 (active)
• py25-altgraph @0.6.7_0 (active)
• py25-bdist_mpkg @0.4.3_0 (active)
• py25-hashlib @2.5.2_0 (active)
• py25-macholib @1.1_0
• py25-macholib-devel @1.2_0 (active)
• py25-modulegraph-devel @0.7.2_0 (active)
• py25-py2app-devel @0.4.2_0 (active)
• py25-pyqt4 @4.4.3_0 (active)
• py25-setuptools @0.6c9_0 (active)
• py25-sip @4.7.6_0 (active)
• py25-zlib @2.5.2_0 (active)
• python25 @2.5.2_5+darwin_9 (active)
• python_select @0.1_4+darwin_9 (active)
• qt4-mac @4.4.3_0+dbus+docs (active)
• tiff @3.8.2_2+darwin_9+macosx (active)
• zlib @1.2.3_2 (active)

Macports installation details

Here are step-by-step instructions on how I got things working:

1. Install MacPorts
2. sudo port install python25
3. sudo port install py25-macholib-devel
4. sudo port install py25-sip
5. sudo port install py25-pyqt4
6. sudo port install py25-py2app-devel
7. sudo port -v install py25-pyqt
8. sudo port install python_select

Note that at one point during the PyQt installation, MacPorts aborted with:

Target org.macports.activate returned: Image error: /opt/local/bin/a2p is being used by the active perl5.8 port. Please deactivate this port first, or use the -f flag to force the activation.

It appears that there were two versions of PERL installed. I simply restarted the installation (sudo port install py25-pyqt) and, after waiting a long while, and having to restart it once more, it completed successfully in installing Qt and PyQt.

(Note that the PyQt installation will take seemingly forever so it might be an idea to (a) start it in verbose mode, as shown above, so you know it's still doing something and hasn't hung, and (b) set up a growlnotify message to let you know when the process is complete.)

Python Launcher is used to launch Python apps by double-clicking on them in the Finder, and is thus useful when testing GUI Python apps.

I wasted a bit of time trying to track down where exactly I had done something wrong in configuring my system so I hope this simple post keeps someone else from the same fate! :)

Based on the issue ticket at python.org, the bug appears to be fixed but I'm not sure which version it will be/was released in (I'm running Python 2.5.1 and experiencing the issue).

Related links:

• PythonLauncher not working correctly on OS X 10.5/Leopard
• Topic : Python Launcher Broken
• Python launcher not working on Mac after Leopard upgrade?
• http://www.macintouch.com/readerreports/leopard/topic4381.html (search for "Python Launcher" in the page).

I started out by downloading the source for the shell app via a link on a post on the PyAMF blog. Unfortunately, the link was to the wrong version of the shell app (I've since informed the PyAMF team and Nick's fixed the link.)

Seeing that the example used sessions, I decided to port the site from Google App Engine Helper to app-engine-patch, which has support for sessions in Google App Engine. Porting to app-engine-patch meant that I had to port the site to Django 1.0 also. I've chronicled my experience with this previously.

Today, I pinged the PyAMF folks on IRC and realized that there was a Google App Engine-specific version available. Unfortunately, that didn't work out of the box either due to a clash between its sessions implementation and the sessions implementation in app-engine-patch. That, however, was simple enough to fix.

If you're running app-enginge-patch and want to run the PyAMF shell, refactor the gateway.py module to rename the Sessions class to ShellSessions and Bob's your uncle. (It might be an idea for the PyAMF team to modify the example so that it works out of the box with app-engine-shell.)

Also, make sure that you protect the shell by only allowing admin access to it.

I can't begin to express how empowering it is to have shell access to your app on Google App Engine and I have no idea how I got by without it for so long. A big thank-you to the PyAMF team for making the shell app and releasing it. Great job, guys!

I would love to see a standalone Django app of the shell that you can simply plug in to your existing apps. Unfortunately, this is currently not possible since the app has static files and you need to add static folders manually to your application's app.yaml file. I've filed an ECR on this -- star it if you'd like that feature.

Update: Scratch that, I keep running into the 1MB limit for datastructures while trying to run queries, etc., through the shell, making it practically useless.

For one thing django.core.validators, which I was using, was removed. The fix: copy it back from the earlier version.

Not sure if it was a bad practice or not but I was also doing dictionary lookups on the request object and that's been removed now.

The errors you will get are:

• 'WSGIRequest' object is not iterable
• 'WSGIRequest' object is unsubscriptable

If you're in the same boat, here's the regular expressions you need to find/replace in TextMate:

Fix 'WSGIRequest' object is not iterable:

Find:

' in request(?!\.)

Replace:

' in request.REQUEST

'WSGIRequest' object is unsubscriptable:

Find:

request\['

Replace:

request.REQUEST['

Also, I switched from using Google App Engine Helper for Django to app-engine-patch. The main reason? So I could have the sessions middleware and run the PyAMF Shell. (And, of course, it's good to be running Django 1.0 now.)

App-engine-patch supports Django's mail feature so I stopped using google.appengine.api.mail.EmailMessage and replaced it with django.core.mail.EmailMessage. One thing to note if you do this is that you need to change the to properties on your EmailMessage instances to be lists, not strings, lest you end up trying to email each letter in the email address separately.

(Also, on my local server, I keep getting a Broken Pipe (error 32) since I upgraded to Django 1.0. I set fail_silently to True on my send() calls and that's fixed it. I don't have any issues on the deployment server. Looking into that one but it's a hairy one to track down.)

Of course, since I hadn't considered images in session descriptions, this had the side-effect of breaking the layout of the sessions page.

(In case you're wondering, yes, this is the way I like to work. Instead of over-engineering things, I like to see how people actually use stuff and then evolve them to meet their needs.)

So tonight I wrote a bit of code to massage and tame how images in session descriptions are displayed and I thought I'd share it with you in case it helps anyone else. (Another, more complicated way to go about things would have been to grab the images using urlfetch, store them in the datastore, and resize them via the image API -- but that would have been overkill for my needs.)

# Copyright (c) 2008 Aral Balkan, Singularity Web Conference
# http://www.singularity08.com
# Released under the open source MIT license.
 
from markdown import Markdown
 
image_tag_width_re = r'(?P<img><img.*?width=")(?P<width>\d*?)"'
image_tag_height_re = r'(?P<img><img.*?height=")(?P<height>\d*?)"'
image_tag_re = r'(<img)(.*?)>'
image_tag_src_re = r'<img.*?src="(.*?)"'
image_tag_alt_re = r'<img.*?alt="(.*?)"'
 
image_tag_width_rc = re.compile(image_tag_width_re)
image_tag_height_rc = re.compile(image_tag_height_re)
image_tag_rc = re.compile(image_tag_re)
image_tag_src_rc = re.compile(image_tag_src_re)
image_tag_alt_rc = re.compile(image_tag_alt_re)
 
IMAGE_SAFE_WIDTH = 160.0
 
def massage_images(html):
	"""Helper: Alters dimensions of any images in the passed HTML to make them safe for the site's design."""
	image_tag_widths = image_tag_width_rc.findall(html)
	image_tag_heights = image_tag_height_rc.findall(html)
	image_tag_srcs = image_tag_src_rc.findall(html)
	image_tag_alts = image_tag_alt_rc.findall(html)
 
	for i in range(len(image_tag_widths)):
		# Reduce the width of any found images to 160px so as not to break the layout
		original_width = int(image_tag_widths[i][1])
 
		maintain_aspect_ratio = True
		try:
			original_height = int(image_tag_heights[i][1])
		except IndexError:
			# Mismatched width/height pairs on image tags. We won't be
			# able to maintain aspect ratio.
			maintain_aspect_ratio = False
 
		if maintain_aspect_ratio:
 
			aspect_ratio = float(original_width)/float(original_height)
			new_height = int(IMAGE_SAFE_WIDTH/aspect_ratio)
 
			# Substitute the new height
			html = image_tag_height_rc.sub(r'\g<img>'+repr(new_height)+r'"', html)
			logging.info(html)
 
		# Substitute the new width
		html = image_tag_width_rc.sub(r'\g<img>'+str(int(IMAGE_SAFE_WIDTH))+'"', html)
 
		# Add float:left and slight margin so that text flows around the image
		html = image_tag_rc.sub(r'\1 style="float:left; margin-right:.5em;" \2>', html)
 
		# Finally, add a link to the original image if people want to see it larger
		html = image_tag_rc.sub(r'<a href="'+ image_tag_srcs[i] + '" title="'+image_tag_alts[i]+r'">\1\2></a>', html)
 
	return html
 

There are a couple of basic but helpful regular expressions in there and you might find the snippet useful if you want to manipulate image tags generated from user submitted content.

Oh, and before I forget, Dr. Woohoo is going to be talking about Generating Artwork at the Singularity Web Conference. Check out his bio and session and the other sessions at the conference.

(You can find out more about Dr.Woohoo on his web site and take a look at his latest book, Color Visualizations: Exploring the Circle, vol 02.)

If you haven't booked your ticket for Singularity yet, hurry, as the $99 early bird discount ends at the end of this month.