Archive for the 'django' Category

Screencast demonstrating the first Google App Engine data export solution (full backup and restore)

Published on August 12, 2008 in Open Source, Singularity, django and python. 5 Comments

This is a quick screencast to show you the data export solution I've created for Google App Engine that lets you backup your application's datastore and restore it either locally on your development machine or on the same Google App Engine application on the deployment environment or on a different Google App Engine application (which you can use as a staging environment).

That's right, this solves one of the big criticisms I've seen levelled at Google App Engine, which is that there is no way to export your data from your applications.

With this solution, not only can you export your data but you can also easily restore it.

I am going to be sharing the solution with you in the coming days as a separate open source project in the form of a Django application that you can plug into your own applications as part of the Singularity Web Conference Open Initiatives program (stay tuned for updates on that front).

In the meanwhile, watch the screencast to see how the backup and restore solution works.

View the full-resolution screencast on screencast.com

New Google App Engine Helper for Django released

Published on August 11, 2008 in django, google app engine and python. 0 Comments

A new version of the Google App Engine Helper for Django has been released.

Release notes (Wed 6 August 2008) follow:

This is the last version of the Google App Engine Helper for Django that will
support Django 0.96. Future development of the helper will be targetted for the
upcoming 1.0 release of Django.

  • Improved SDK detection on Windows by looking at both the PATH variable that
    may be set by the installer and using the win32api module (if available) to
    look for the SDK in the default Program Files location.
  • Replaced the startapp command with a version that installs an App Engine

    Compatible application skeleton. Patch contributed by Andi Albrecht.

  • Changed the default runserver port to 8000 to match standard Django
    behaviour. Path contributed by Waldemar Kornewald
  • Email server settings from the Django settings file are provided to the App
    Engine Mail API. Patch contributed by Waldemar Kornewald
  • Added support for the Django memcache cache backend. Patch contributed by
    Jonca Rafal.
  • Added support for the Django session middle with db and cache backends for
    Django 1.0alpha only. Patches contributed by Jonca Rafal and Waldemar
    Kornewald
  • Moved the Django compatible login_required decorator to the standard Django
    location. Patch contributed by Andi Albrecht
  • Replaced the Django ModelForm class with the App Engine ModelForm class
  • Added a repr implementation for the BaseModel class
  • Many minor improvements to increase robustness and avoid errors if portions
    of Django are not present.

Via App Engine Guy/Mattias Johansson.

Coming soon: backup and restore the data in your Google App Engine applications

Published on August 5, 2008 in Tools and Utilities, django, google app engine and python. 0 Comments

Google app Engine Backup Restore Datastore

Today, I made a full backup of the data on the Singularity web site on Google App Engine and restored it on my local machine running the SDK.

If you're not familiar with Google App Engine, you may be thinking, "so what?" Big deal, Aral, I can do backups with a click of the button in PhpMyAdmin. Unfortunately, though, there isn't currently a publicly available data export feature for Google App Engine, much less a solution to backup and restore your data easily. (One of the top criticisms aimed at Google App Engine is that you cannot backup/download your data.)

As far as I know, this is the first time a backup and restore has been done on Google App Engine (though Google engineers may have successfully tested their own solutions internally.)

My solution works by backing up the datastore incrementally into Python code (I ran into every possible limit on App Engine while developing this, as those of you following my Twitters today will have witnessed).

Yep, that's right, the backups are stored as Python code. "What about restores?", I hear you ask. Well, a backup is pretty useless if you cannot restore it. Restoring the backups is as easy as running the generated python code (either all at once, on the local SDK, or incrementally on the deployment environment.)

The four use cases I see for this are as follows:

  1. Backup you data (data safety)
  2. Backup your data and restore to the local SDK (local testing with real data)
  3. Backup your data and restore to a different App Engine instance (staging server)
  4. Backup your data and restore to your live application instance (data recovery)

I've already successfully handled use case 1 and I'm currently almost done with a generic restore feature that should correctly handle use cases 2-4.

When I've got restores working properly for the Singularity app, I'm going to decouple these handlers from the app and create a separate Django app that you can include in your own projects to give you backup and restore functionality in Google App Engine.

I will be releasing this as an open source project as part of Singularity's Open Source Initiative, alongside OpenCountryCodes, The European VAT Number Validation API, and The GAE SWF Project.

I've also got plans to make the whole process entirely seamless but my priority is working on the Singularity web conference so don't expect a very polished solution immediately. I will be working on this as my "20% project" of sorts, though, as it is essential to have a solid backup/restore solution for commercial apps on Google App Engine.

To read more about this, including some of the challenges, check out the following thread on the forums: Datastore backup solution (almost ready).

Based on forum postings, it would appear that Google is also working on the data export issue and I'm talking with Pete Koomen at the moment regarding my solution. Check out Pete's thread on backups here: Feedback on data exporter design?

Finally, one of the challenges in working with real data on the local SDK was that the SDK would grind to a halt after you populated a certain number of rows in the local datastore; see performance issue with SDK datastore with large volumne (>1000 rows). Thankfully, Baptiste Lepilleur created a very timely patch (see Issue 390) to work around the problem. This means that restoring a backup to your local machine will not take forever (it's still not blisteringly fast -- taking 0.1 sec per put but at least the duration does not increase linearly like it used to).

I plan to get restores fully working tomorrow (along with my other, more social responsibilities for Singularity and Pistach.io). Follow me on Twitter or keep an eye on the blog and/or the relevant thread on the forums for updates.

We're close to having a working data backup and restore solution on Google App Engine! :)

The 1MB hard limit in Google App Engine

Published on August 4, 2008 in django, google app engine and python. 5 Comments

There isn't much documentation on what exactly is affected by the 1MB limit in Google App Engine so here's my effort at documenting this based on my empirical findings:

  • You cannot have a data structure (e.g., variable) larger than 1MB in size or you'll get a MemoryError.
  • You cannot return a response that is larger than 1MB in size or you'll get an error similar to: HTTP response was too large: 3457738. The limit is: 1048576. (I got that error when trying to circumvent the 1MB MemoryError issue by returning a generator for my HttpResponse).
  • Model instances cannot be larger than 1MB in size or you'll get a RequestTooLarge error.

New runserver options for manage.py in Google App Engine Helper for Django

Published on July 12, 2008 in django, google app engine and python. 0 Comments

Update: Just as I was writing/releasing my patch, Matt Brown apparently committed r38 (diff) to the Google App Engine Helper for Django source tree which resolves the same issue -- and much more simply and elegantly than my solution, by using Django's own settings file. Use Matt's solution instead of my patch.

Original post follows:

The Google App Engine Helper for Django is an excellent library if you want to build Django apps on Google App Engine. I've been using it without any hassles on the new Singularity Web Conference site. One thing that it currently lacks, however, is a way to specify dev_appserver options (like smtp_host and smtp_port, for example) when you start the server.

So I got into the lazy habit of using dev_appserver.py to start the server when I needed to test email functionality and using ./manage.py runserver at other times.

Don't do this! :)

It started tripping me up because the two use different datastore instances. So I would do dumb things like run the server using dev_appserver.py and then start a shell with ./manage.py shell and wonder why certain entities were not in the datastore. Doh! :)

Instead, I'd recommend that you use manage.py for everything.

The problem then remains, how do you pass options to manage.py? Well, I just bit the bullet and added that functionality to the Google App Engine Helper for Django.

Here's the patch (runserver.patch.zip, 1kb).

After you apply it to your appengine_django folder, you will be able to pass the smtp_host and smtp_port options to the server just like you can with dev_appserver.py.

For example:

./manage.py runserver --smtp_host=localhost --smtp_port=5000 192.168.1.2:8080

I've only set those two options up because those are what I need at the moment but the code is flexible. Just add more valid dev_appserver.py options to the valid_args tuple in runserver.py and they'll automatically be proxied to the dev_appserver instance.

Winning at the shell game: iPython on Google App Engine

Published on June 22, 2008 in Development, Tools and Utilities, django, google app engine and python. 1 Comment

iPython is an awesome extended Python shell that gives you goodies like tab completion for instances, history tracing (so you can easily copy interactive sessions as doctests), etc. And, if you install it, your Django project on Google App Engine will automatically start using it instead of the regular python shell when you use ./manage.py shell.

To install iPython on OS X Tiger (yes, my Leopard discs are still safely in their box since I downgraded and I don't see any reason to bring them back out yet), I followed the following steps:

  1. Download the latest iPython from the iPython distributions page (ipython-0.8.4.tar.gz)
  2. Untar it, cd into the folder
  3. As per the instructions on the iPython download page: 
    python setup.py build
sudo python setup.py install
  4. To test it out on my Google App Engine/Django project, from my project folder: ./manage.py shell

(Note: The docs mention that you need to have readline installed on Mac OS X in order to use some of the features like tab completion and syntax highlighting. It just worked out of the box for me on OS X Tiger 10.4.11 -- I'm not sure if I had installed readline at some point or whether it was just there. Check out these instructions if you're having trouble.)

Once you have it installed, try out the cool code completion:

from my_app import models
models.

Press ⇥, and you'll see a list of all your models. models.my_model. ⇥ will show you the properties for that model and so on.

To create doctests, simply enter your test instructions in the shell and then type hist -n to get a dump of your history without line numbers that you can copy and paste into your doctest.

You can press ⌃ P and ⌃ ⇧ P to interactively bring up the previous and next commands in the history. If you've typed a bit of the command before doing this, it will filter to show you only those commands from your history that match the text you've entered.

You can also access the system shell without leaving iPython by preceding system calls with an exclamation mark. !ls, for example, will show you a listing of the current working directory.

And there's much more you can do that you can read about on the iPython documentation (or just type ? in the iPython shell itself and browse the docs interactively).

Check out iPython, it's yummy!

I found out about iPython from an excellent blog post by AkH on useful tips and good practices for Django projects. Thanks, dude!

Conditionally displaying sIFR text for different languages using content negotiation with Django

Published on June 20, 2008 in Development, django and python. 0 Comments

I'm using sIFR on the new Singularity teaser site (not least because Mark Wubben is speaking at the conference, mind you) and ran into an issue today with extended characters (such as extended Western characters, Chinese, Japanese, etc.) not displaying properly as they weren't embedded into the Flash text field.

Getting extended Western characters working is not too difficult as you can embed most of them in the sIFR SWF without increasing the size of the SWF too much. The current size of my GillSans SWF currently on the site is 32KB and includes the following character sets from Flash (note that some of these contain overlaps):

  • Uppercase
  • Lowercase
  • Numerals
  • Punctuation
  • Basic Latin
  • Latin I
  • Latin Extended A
  • Latin Extended B
  • Greek
  • Cyrillic
  • Armenian

Embedding fonts for other sets such as Hebrew, however, balloons the size of the SWF to unacceptable sizes.

The workaround I implemented was to use content negotiation to switch sIFR off for various languages. The code (which you can put inside your request handler or in a decorator or middleware method):

use_sifr = True
if 'HTTP_ACCEPT_LANGUAGE' in request.META:
	language = request.META['HTTP_ACCEPT_LANGUAGE']
	remove_sifr_for = ('ar', 'zh', 'he', 'ja', 'el', 'ko', 'pa', 'th')
	for lang_test in remove_sifr_for:
		if lang_test in language:
			use_sifr = False
context['use_sifr'] = use_sifr
 

Then, based on the setting of the use_sifr template variable, I conditionally include the JS for sIFR: 

{% if use_sifr %}
	<script type="text/javascript" src="/js/sifr.js"></script>
	<script type="text/javascript" src="/js/sifr-config.js"></script>
{% endif %}

You can easily test this out in Firefox through the Preferences panel (⌘ ,) → General → Languages → Choose... and adding one of the languages for which sIFR is switched off (such as Chinese).

“Google App Engine: To Django or to webapp?” Revisted

Published on May 30, 2008 in Development, django, google app engine and python. 6 Comments

When I was first starting out with Google App Engine (GAE), I wrote a short post detailing my thoughts on whether to use Django or Google's own webapp framework for GAE projects. In that post, I concluded that "there isn't a compelling reason to use Django at the moment with GAE".

Since then, I've re-evaluated my decision and, a few weeks ago, I ported the Singularity web app to Django from webapp.

Here are the main factors that contributed to my decision:

  • I wanted to use features in the latest Django trunk (0.97+) and the Google App Engine SDK ships with 0.96. I also did not want to be tied in to updates from Google for the Django framework.
  • Google released the Google App Engine Helper for Django which makes it very simple to set of a GAE Django app.
  • And, finally, looking through the source of Rietveld, I saw that Guido van Rossum is using Django for his Google App Engine app. And heck, I think he knows a thing or two that I may not.

Going with Django means that you can take advantage of more existing functionality (Django middleware, etc.) and, should you want to move away from Google App Engine in the future, there will be less code to rewrite. (Though, realistically, if you're doing anything with data, you're pretty much tied in to the DataStore at the moment until there is a suitable, scalable alternative that you can port to without completely refactoring your data layer.)

So my new take on this is that it makes a lot of sense to build Google App Engine applications using Django instead of pure webapp.

To get started, check out the Google App Engine Helper for Django.

Bad Behavior has blocked 0 access attempts in the last 7 days.