I woke up this morning to some great news in my inbox from Pete Koomen, one of the engineers on the App Engine team who was kindly helping me out with the issues I was having last year and whom I've been liaising with since regarding App Engine:

Subject: I think you might enjoy this

http://googleappengine.blogspot.com/2009/02/skys-almost-limit-high-cpu-is-no-more.html

Thanks for all the feedback and patience, Aral — it really made it clear to us how much of a pain point this was for developers, and I'm thrilled to see this limit erased!

Basically, the crippling high CPU quotas that I've been bitching high and low about have been entirely removed, file size limits have been raised to 10MB from 1MB, and the response deadline quota has been raised to 30 seconds from 10.

Woot! Thanks for the heads up and thanks for listening, Pete, Paul, Guido! You guys rock! :)

I've just tagged Gaebar Beta 3 (version 0.3) in GitHub.

This is a bugfix release, with changes almost entirely submitted by or suggested by the community.

Thanks go to gumptionthomas, Martin Kleppmann, Tony Andrews, Rafal Jonca, and, of course, Guido van Rossum, for your patches and feedback.

Changes in Beta 3

• Merged gumptionthomas's pull request (commit)
  • Fixed unicode and escape string issues with key_names
  • Properly reference InternalError as db.InternalError
• Applied patch by Martin Kleppmann (commit)
• Backup doesn't blow up when there are non-backup files/folders in the backups folder. (e.g., if your project uses SVN)
• Updated the readme.txt based on Guido's feedback (commit)
• Added note to readme.txt regarding getting more detailed exceptions (thanks to Tony Andrews; commit).
• Applied patch by Rafal Jonca (commit):
  • Key names may now contain unicode
  • ReferenceProperties can now be created with keys of parents, not just model instances
  • Now using global datastore's delete, put, and get methods instead of model's

See the original announcement for a screencast and general information on Gaebar.

Please do keep your feedback coming and, of course, if you have pull requests, patches, or suggestions, please send them my way (and thank you!)

I just posted my slides from last night's talk at the London Django User Group on SlideShare (and you can view them above).

A big thank you to Rob and the DJUGL team for organizing the event and kindly inviting me over to speak. I had a great time and it seems like the audience enjoyed it too! :)

I'm happy to announce that there's quite an important update to Gaebar that brings with it some essential bug fixes and should help shave quite a bit of time off of your restores.

I've also created a new project called gaebar-gaed-skeleton on GitHub that gives you an empty Google App Engine Django (Helper) project with Gaebar pre-installed to start you off when creating your own applications.

Here's a list of changes in Beta 2:

• Fixed: All required properties, including references should work correctly now. Required properties would raise an exception during the restore process in Beta 1.
• Refactored: We are no longer using actual entity references (which was unnecessary) but keys to create reference properties. This means that creation order of references doesn’t matter any more (thanks to Pete Koomen who explained all this to me in an email ages ago; it only just sunk in, Pete!)
• New feature: Due to above changes, the restore process is now one-pass, not two. This means that restores should now take considerably less time.
• Refactored: I moved the functional tests out to their own GitHub module since they were being shared by both the gaebar-gaed and gaebar-aep test apps. This will make maintaining them much easier.

I want to thank Jonathan Ricketson for alerting me in the comments earlier today that the Beta 1 release was not handling required properties correctly (thanks, Jonathan). I also want to thank Thomas Bohmbach, Jr. for providing feedback and helping me test.

You can download Gaebar Beta 2 and the other Gaebar projects from my GitHub account.

If you'd like to learn more about Gaebar, please see my original announcement, which contains a screencast showing Gaebar (albeit Beta 1) in use.

As always, I value your bug reports, comments, and suggestions so please feel free to leave a comment and let me know of your experiences with Gaebar.

Update: Beta 2 Released! See the Gaebar Beta 2 announcement.

Here's my Christmas present for the Google App Engine community: Google App Engine Backup and Restore (or Gaebar, for short).

Gaebar is an easy-to-use, standalone Django application that you can plug in to your existing Google App Engine Django or app-engine-patch-based Django applications on Google App Engine to give them datastore backup and restore functionality.

For a quick overview of Gaebar, watch the screencast, above. For the impatient, links to the project pages where you can download Gaebar follow.

Downloads

Gaebar is hosted on GitHub. You can either download archives or clone the repository (or install Gaebar as a git submodule) via git. Alongside the Gaebar project itself are two sample applications — one built on Google App Engine Django and the other on app-engine-patch — that contain the Gaebar functional test suite. The functional test suite tests every datatype supported by Google App Engine as well as references, Expandos, and ancestor relationships.

Please make sure you read the readme files after downloading the projects for installation and usage instructions.

• Gaebar: http://github.com/aral/gaebar/tree/master
• Gaebar-gaed: http://github.com/aral/gaebar-gaed/tree/master
• Gaebar-aep: http://github.com/aral/gaebar-aep/tree/master

What you can do with Gaebar

• Backup your deployment application's datastore for safekeeping.
• Restore that backup on your local development server for testing with real data during development.
• Restore that backup to a different Google App Engine application and use that application as a staging application much in the same way you would use a staging server in traditional development.

(You can, of course, also backup and restore your local development datastore as well as your staging application, etc.)

I've got a huge datastore, will Gaebar work for me?

Congratulations on the impressive size of your datastore! The answer should be "yes!" The largest datastore I've tested it with is the <head> web conference datastore. The latest backup had 18, 969 rows from 14 models backed up into 225 code shards.

What's a staging application?

A staging application is a new Google App Engine concept made possible by Gaebar. Basically, if your application is myapp.appspot.com, you can use a separate application (say, myapp-staging.appspot.com) in the same way as you would use a staging server in traditional development.

Your staging application can let you try out new features and test with real data without having your users see your changes until you are ready to deploy to your main application.

In fact, it's a perfect staging environment since it is identical to your deployment environment. For an example of this, see the screencast.

How Gaebar works

Gaebar backs up the data in your datastore to Python code. It restores your data by running the generated Python code.

Since a backup is a long running process, and since Google App Engine doesn't support long-running processes, Gaebar fakes a long running process by breaking up the backup and restore processes into bite-sized chunks and repeatedly hitting the server via Ajax calls.

By default, Gaebar backs up 5 rows at a time to avoid the short term CPU and 10-second call duration quotas and splits the generated code into code shards of approx. 300KB to avoid the 1MB limit on objects. You can change these defaults in the views.py file if your app has higher quotas and you want faster backups and restores.

Once a backup of a remote server is complete, Gaebar automatically hits your local development server. From there on, the local development server makes a series of calls to the remote server to download the backup files (code shards) from the remote server. Once the backup is complete, you will see a new backup folder in gaebar/backups with the contents of your backup.

Here's an example of some generated backup code from the Google App Engine Django test application (updated for the upcoming Beta 2 release):

import pickle
from google.appengine.api.datastore import datastore_types
from app1.models import *
from app2.models import *
 
def row_0(app_name):
	existing_entity = Profile.get(datastore_types.Key.from_path('Profile', 1, _app=app_name))
	if existing_entity:
		existing_entity.delete()
	profile_0 = Profile(key_name="id1", friends = pickle.loads('(lp0\n.'), in_relationship_with = pickle.loads('N.'), full_name = pickle.loads('VPaul Booth\np0\n.'))
	profile_0.put()
 
def row_1(app_name):
	existing_entity = Profile.get(datastore_types.Key.from_path('Profile', 2, _app=app_name))
	if existing_entity:
		existing_entity.delete()
	profile_1 = Profile(key_name="id2", full_name = pickle.loads('VAral Balkan\np0\n.'), friends = [datastore_types.Key.from_path('Profile', u'stephalicious', _app=app_name), datastore_types.Key.from_path('Profile', 'id1', _app=app_name)], in_relationship_with = datastore_types.Key.from_path('Profile', u'stephalicious', _app=app_name))
	profile_1.put()

To restore, you simply deploy your application, along with the backup folder, to your deployment environment and hit the Restore button in Gaebar. (If you have a large datastore both the backup and restore processes will take a long time, especially when restoring to a local development server.)

The restore process simply calls each of the generated row functions and each row function restores a single row into the datastore.

A note on the screencast

When mentioning how to install Gaebar, I left out that you also need to add the URL mapping for Gaebar to your application's urls.py. That, along with the rest of the installation instructions are in the readme.txt file, which I highly recommend that you peruse.

Have your say!

As in all things, my approach to blog posts is that they should evolve over time and your feedback is invaluable in achieving this by helping me fix factual errors, fill in details, and expand the original post.

What do you think of Gaebar? Have you run into any issues that need fixing? Do you have other suggestions on how to improve it? Or do you, perhaps, have a patch to send me that adds Webapp support or some other feature? Leave me a comment and let me know!

Gaebar is a Naklabâ„¢ production released under GNU GPL v3 and sponsored by the <head> web conference.

To create a new Django application for Google App Engine using Google App Engine Helper, start by exporting the latest Google App Engine Helper:

svn export http://google-app-engine-django.googlecode.com/svn/trunk/ ~/myapp

To create a django.zip file to use with the latest Google App Engine:

1. Export the latest Django trunk

   svn export http://code.djangoproject.com/svn/django/trunk/ django

2. Follow steps #2 and #3 in Dan's instructions under the "Archiving Django 1.0" section of his article
3. Add contrib/auth:

   zip -r django.zip django/contrib/__init__.py \
   	                  django/contrib/auth

4. Add contrib/sessions:

   zip -r django.zip django/contrib/__init__.py \
   	                  django/contrib/sessions

5. Copy the django.zip file into the root folder of your new app:

   cp django.zip ~/myapp

6. Uncomment the sessions middleware or auth middleware lines in settings.py file in your app so that it looks like:

   MIDDLEWARE_CLASSES = (
   	    'django.middleware.common.CommonMiddleware',
   	    'django.contrib.sessions.middleware.SessionMiddleware',
   	    'django.contrib.auth.middleware.AuthenticationMiddleware',
   	#    'django.middleware.doc.XViewMiddleware',
   	)

That's it! To test it out:

./manage.py runserver

Hit http://localhost:8000/ and you should see the default Django welcome page.

Update: Don't make the mistake of adding django.contrib.auth or django.contrib.sessions to your INSTALLED_APPS, even though you are including those apps in the django.zip file. If you do, ./manage.py runserver will work but ./manage.py shell and ./manage.py flush will return errors.

I traced the issue to the _RetrieveURL method in /usr/local/google_appengine/google/appengine/api/urlfetch_stub.py and implemented a quick hack, adding the following at line 152:

# Aral - hack: for some reason urlfetch doesn't like localhost
# so we translate it to 127.0.0.1. This is for local testing only.
if host == 'localhost:8080':
  host = '127.0.0.1:8080'

This basically allows me to hit one instance of the local dev server from another for testing some functionality I need for the Google App Engine backup/restore solution I'm working on. Without this hack, I can't hit localhost with urlfetch at all.

I'm still not sure why this is happening. (Could appenginepatch be messing things up — it was working under AppEngineHelper... hmm?)

I'm seeing a lot of misinformation out there that Google App Engine has SSL support. It doesn't.

If you read Google's announcement from last month carefully, you will see that it is titled "Announcing HTTPS support for appspot.com", not "for Google App Engine" .

"What's the difference?", you may be wondering. It's quite big.

What google supports is SSL for subdomains for appspot.com. So, if you're happy with hosting your application only from your appspot.com subdomain, then you can use SSL in your application. For example, I have several simple open source apps that I host on Google App Engine, like opencountrycodes and isvat. Since I'm not worried about having custom domains for these, I host them on appspot.com subdomains (e.g., opencountrycodes.appspot.com). If I wanted to use HTTPS on these apps, I could and you could hit, for example, https://opencountrycodes.appspot.com to get to the secure version of the site.

The issue arises when you want to use your own domain for your App Engine application. The <head> conference web site, for example, resides at http://www.headconference.com. It also has a subdomain on appspot.com but I don't publicize that. Currently, there is no way I can use SSL on the Head Conference web site as Google App Engine only supports SSL for appspot.com subdomains (so, if the headconference subdomain on appspot.com was headconference.appspot.com, I could use SSL on that subdomain).

"So", I can hear you ask, "why don't you just use headconference.com for everything and headconference.appspot.com for the SSL connections?"

The answer: sessions and usability.

Take this example:

I want to use PayPal to process a payment. My user hits www.headconference.com, signs in, and starts the checkout process. I forward the user to PayPal and pass the secure URL of my notification script to PayPal (e.g., https://headconference.appspot.com/my/notification/handler/). When the user returns from PayPal, they are no longer on the headconference.com site, their session is no longer valid, and they need to sign in again.

In other words, this is not a usable solution for custom domains.

The problem is that implementing SSL on Google App Engine is not a trivial solution. One possible methodx that has been suggested several times in the forums is to use the Server Name Indication (SLI) extension to Transport Layer Security (TLS).

So what is the current state of SSL on Google App Engine? It's still not supported, with the exception of applications that are hosted only at appspot.com subdomains. In other words — given that most real-world applications, whether commercial or not, will want to be hosted under their own domains — SSL is still not supported for real-world applications on Google App Engine.

Google today released Google App Engine SDK 1.1.6 (announcement, download) which, among other things, brings with it datastore keys that you can sort and filter on.

What this means is that you can now do batch operations on data.

So it's time for me to go back to my Google App Engine backup solution, which broke after Google introduced artificial limits on query offsets, and update it to use sortable keys.

With 1.1.6, the SDK also makes inroads into emulating deployment environment quotas by implementing URLFetch call timeouts after 5 seconds.

Good work, guys! Here's hoping that the next thing you're working on is map/reduce and long-running processes.

Last week, we made a little history when over 70 speakers and 500 attendees from around the world congregated on the Interwebs for the <head> web conference and, in doing so, took part in creating the world's first virtual web conference. Last week, thanks to you, we proved that there is a new model for conferences that is engaging, environmentally-friendly, and affordable.

Voilà, l'âge de la Conférence 2.0 commence

Presenters spoke from home, from hotels, from our local conference hubs in London, Brighton, and San Francisco, and, in one case, from a car parked on a street-side in downtown New York City.

I'm overwhelmed by the wonderful reviews I'm reading from attendees and speakers alike, as well as the lovely feedback we've been receiving directly. I'd like to share some of this with you here, starting with this gem from my friend Jeremy Keith, who opened the conference with his excellent talk on The Long Web:

It spanned three days and as many continents. It was a preposterously ambitions undertaking and, incredibly, it worked! (Read the full post.)

One of the things that I loved about the conference was finding out more about the things I had heard of, but knew nothing about (both Richard Rutter and Ann McKeekin's talks come to mind) . . . Thank you again for a wonderful conference, I'm looking forward to next year. (Jonathan Ferguson, attendee.)

I had great time interacting with the crowd by telling my stories, answering questions, giving away books and hopefully instilling some inspiration to the one hundred in my apartment. (Christopher Schmitt, speaker; read the full post.)

I really enjoyed the conference sessions last weekend. Congratulations on a great event! (Stuart Giles, attendee.)

The Head Conference was an ambitious, timely, and much needed creative exploration of the potential for "green" conferencing using Adobe Connect Pro, Second Life and local conference hubs in various cities. (Tish Shute, read the full post.)

Great job with the Head Conference. I had a blast . . . Was surprised how interactive and social and online conference could be. Already looking forward to next year. (Mark Grossnickle, attendee.)

All by all, it turned out to be an amazing experience, that not only made you realize again why you love internet, but also contributed to your sense of commitment to the . . . community. I guess the best thing I like about <head> is that you can come together, interact and learn from the cracks, sitting in your lazy chair, no matter where you are! (Jeff Winder, attendee; read the full post.)

(I won't go on so this post doesn't turn into a testimonials page — suffice to say we got a heap more of lovely reviews on the web, on twitter, and in emails — thank you all so much for sharing them with us. We're wonderfully happy by the response, and we're listening to your feedback so that we can evolve the conference and make it even better for next year.)

Undoubtedly, following the success of <head>, we are going to see more conferences embrace virtual aspects and I can only assume that we will see more virtual conferences popping up. <head>, of course, will not be standing still. We have learned tons by staging this conference and I have very definite plans for where I want to see <head> go next as we evolve and continue to explore the concept of Conference 2.0.

In closing, I want to once again thank our wonderful speakers, sponsors, micro-sponsors, and, of course, our attendees for breathing life into the conference and making <head> a success — thank you all for your support and here's to an even better <head> 2009!

Photos

• Flickr tag head2008
• Flickr tag headconference
• Flickr group: head08

Related posts from speakers

• Jeremy Keith - </head>
• Yahoo! Developer Network Blog - Moving on with the style of conferences: <head> was a great success
• Christopher Schmitt - Head Conference, The Virtual Conference
• Chris Heilmann - Seven reasons why web development is running around in circles, The art and pain of teaching JavaScript - my talk at <head>
• Seb Lee-Delisle - Reprezenting the Flash crew at the <head> conference.
• Henny Swan - Have you got the X Factor? Building websites for mobile and disabled users
• Joshua Hirsch - <head> conference

Related posts from attendees

• Jeff Winder - <head> 2008: The Future Is Here
• Tish Shute - Doing Something Useful With Virtual Worlds, Tim O'Reilly: Instrumenting the World
• Tom Morris - Taking the hat off my head, On not banging one's head against a brick wall
• Mark Grossnickle - Head conference wrap-up: Day 1, Day 2, Day 3

(If you have a blog post or review that I missed, please leave me a link to it in the comments — thanks!)