Unfortunately, Google App Engine's urlfetch api caches requests and doesn't give you a means not to cache the results or to set the cache duration (an issue that has been discussed several times on the forum too.)

For us, this means that there's an unknown delay between when you record your video and when it shows up on the video page, even though it is added to the Qik RSS immediately after it is recorded.

There is an open issue for this. Please star! Thanks!

Update: Thanks to argladd86 on the App Engine issue tracker, this problem has now been solved!

The solution involves using the max-age property, as shown below:

fetch_headers = {'Cache-Control':'no-cache,max-age=0', 'Pragma':'no-cache'}
qik_rss_result = urlfetch.fetch("http://qik.com/event/rss/397/ltheadgt-web-conference", None, urlfetch.GET, fetch_headers)

When you do that, you lose the ability to refer to your pipes using the yahooapis.com domain and thus you cannot load your data from Flash. There doesn't seem to be a way to revert to non-customized URLs either once you've made the switch.

The problem is that the yahooapis.com domain contains the necessary crossdomain.xml file but yahoo.com doesn't. Normally, what you do is replace the yahoo.com URL you get for your pipe with yahooapis.com to use it in Flash (why not just give the yahooapis.com URL as default and save Flash developers a bunch of confusion?)

Once you've customized your domain, however, that substitution results in a 404.

Here's hoping the Yahoo! Pipes team fixes this soon.

In the meanwhile, I guess my only recourse is to set up a new Yahoo! account and use that for my pipe.

Traceback (most recent call last):
  File "/Users/aral/projects/headconference/trunk/shell/gateway.py", line 218, in evalCode
    session.put()
AttributeError: 'NoneType' object has no attribute 'put'

…while using the PyAMF Shell after upgrading your Google App Engine SDK (to the latest 1.1.5, for example) the fix is really quite simple: delete the cookies in your browser.

Google updated the App Engine SDK to version 1.1.5 today (release notes).

Among the new features, the one that really caught my eye was: "Sped up the datastore stub."

I didn't notice a speed up in the initial start-up time of the dev server with a large datastore, but it does seem that my app is more responsive. I need to test further and get some real data though.

One thing I've noticed is that the mean time between SDK releases has been decreasing quite considerably as of late. I hope that means that development on App Engine is speeding up and that we'll start seeing new feature implementations on the deployment environment also.

(And no, Java support is most definitely not one of them.)

Rather than writing a lengthy diatribe, I am going to succinctly list the main showstoppers that Google needs to fix to make Google App Engine work:

1MB limit on data structures

Although this is hinted at in various bits of the documentation, it is not stated plainly anywhere so let me state it for the record:

Google App Engine does not support any data structure that is larger than 1MB in size.

This includes files. So you can't host that 1.2MB PDF you want to offer for download as part of your site.

The limit affects blobs, text, and any other fields in the datastore, as well as variables in Python (so you cannot get around the limitation by breaking things up into smaller pieces in the datastore and stitching them together later in code).

The only way to get around this limit is to stitch the data structure together on the client by making several Ajax calls. The number of use cases where this is useful, of course, is severely limited.

Among other things, the 1MB limit on data structures makes it nigh on impossible to run reports and makes having an administrative shell on the deployment environment pretty much useless as serialized results of queries with hundreds of items will quickly hit the limit.

1,000 item limit on query offsets

The datastore has always had a 1,000 item limit on query results. I don't have a problem with this.

However, they recently also introduced a 1,000 item limit on offsets. This means that you can at most get the 2,000th entity for a given kind.

Again this isn't documented anywhere.

Unless you have been keeping a sortable field (like a numeric index that you were manually saving) in your datastore, this new limitation effectively locks you out of your data beyond the 2,000th item for each kind.

The fact that such a radical change was introduced without any forewarning or even an announcement is, in and of itself, troubling.

Currently, the only way to get around this limitation is to make sure that you have a sortable field that you can limit your queries on. I would highly recommend saving a numeric key for every entity. This does mean, however, that you will be doing two datastore writes for every entity (once to save the record and get its numeric id, and another to write the numeric id into a separate field that you can sort on). This is both more cumbersome and will no doubt raise the risk of your running into another Google App Engine limit: The short-term high CPU quota.

(From what I've heard, the Google team is working on adding sortable keys in a future update.)

The short-term high CPU quota

Not only are your calls limited to returning in 10 seconds but if you actually try to do anything too stressful within those calls, you will quickly hit the short-term high CPU quota limit.

Thankfully, Google has raised this limit (and the other limits) for my app but plain vanilla apps are not as lucky. In my tests, I've found that the high CPU limit can be randomly triggered even in calls that return within a second. To tell you the truth, I don't actually know what causes these. I memcache nearly everything and try my hardest not to stress the system out and yet I routinely see the high CPU warnings in my logs for even the most mundane calls.

(I've also seen strange high CPU errors in my logs informing me that I am 1.0x above the high CPU limit which makes no sense at all.)

The short-term high CPU quotas must be removed. Not only that but Google must review how it handles quotas in general.

Quotas in general

When was the last time you saw an "over quota" error on one of your favorite web applications?

Google's handing of quotas is a major step backwards to the days of Geocities and "this user has used too much bandwidth" errors. Probably acceptable if you're hosting pictures of Little Timmy and Sally Jo's Summer Camp Adventure on Geocities, not so acceptable if you're hosting your next big web app on what you thought was Google's infinitely-scalable Cloud solution.

Simply put, they couldn't have come up with a worse PR campaign for Google App Engine if they had hired Steve Ballmer to handle the job.

Think about it:

You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the "intelligent throttling" "feature" in Google App Engine which freaks out and shuts down your app with an "Over Quota" error -- effectively making the "Over Quota" message the first impression most of your audience has of Google App Engine.

Not good.

Especially not good when your unique selling point is that your system can scale.

We don't care that it can scale. We care that it does scale. And that it scales when you need it the most.

But that's not the worst bit.

Admin? What's that?

Currently Google App Engine does only one half of what a web application needs (and does that only half well). As it stands today, Google App Engine is a highly scalable request/response system that is tuned to handle lots of tiny calls.

A typical web application, however, needs more than that.

While concentrating on making applications scalable, Google App Engine entirely ignores a crucial use case of any web application: administration. The administrative features of your web application may not be consumer-facing but they are just as important. They may include features for running reports or mailing all of your users. Key, essential tasks for any modern web application.

With Google App Engine today, if you have more than a couple of thousand members/records in your datastore, you can forget about running any sort of admin task.

A total lack of long running processes, coupled with the 1MB limit on data structures and the 1,000 limit on query offsets means that you cannot run reports or backups. (Not that there is a data backup system currently available for Google App Engine -- I have one that I wrote myself which used to work but is currently crippled due to the 1,000 item limit on offsets.)

Similarly, unless you knew to plan ahead and create a sortable key to query on, you will find yourself locked out reaching certain records in your datastore (for example, not being able to email all of your members).

These are core showstoppers for anyone considering building a real-world application on Google App Engine and I can only hope that they are at the top of the engineering team's list of new features.

25% ready for prime-time

As things stand, Google App Engine is about 25% ready for prime time. Once quotas are handled properly and the 1MB limit removed, it will be about 50% ready. The other 50% has nothing to do with scalability and everything to do with everything else that a typical web application needs. Specifically, long running processes and the ability to run reports, aggregate data, and perform operations on large data sets.

In effect, Google App Engine is entirely missing a separate mode of operation and this glaring omission must be addressed before Google App Engine can be deemed a serious web application platform.

Priorities and showstoppers

I really hope that Google is not working on adding support for other languages to Google App Engine, as they mentioned that they were at the Google Developer Day in London, when there are such fundamentally crippling issues with the platform that must be addressed first. Adding support for other languages to Google App Engine today is like sewing a new set of drapes for a house that doesn't have any walls yet.

In summary, before Google App Engine can be used for real web applications, the following issues have to be addressed:

• 1MB limit on data structures must be removed.
• Quotas must be handled optimistically, without crippling applications at the very moment that they should be benefitting from The Cloud. The "intelligent throttling" "feature" and short-term CPU quotas must be removed.
• The 1,000 item limit for offsets in queries must be removed.
• Support for long-running processes must be added.

It may just be that Google has to implement two modes of operation for each application on Google App Engine: one that is request/response-only and scales (i.e., basically what we have today, with better quota handling) and a separate admin mode that has long-running processes and isn't crippled by data structure size limits and short-term high CPU quotas.

As things stand today, running a real-world application on Google App Engine is a complete nightmare because the system completely ignores the essential administration-related use cases for web applications that we take for granted on other platforms.

Update: Several people have brought it to my attention that I forgot to mention another major showstopper for most applications, which is the lack of SSL. I ran into this early on, did my research, and found that PayPal was the only viable e-commerce solution at the time if your application needs to receive notification callbacks on purchases (ironically, Google Checkout requires SSL for its notifications API).

I started out by downloading the source for the shell app via a link on a post on the PyAMF blog. Unfortunately, the link was to the wrong version of the shell app (I've since informed the PyAMF team and Nick's fixed the link.)

Seeing that the example used sessions, I decided to port the site from Google App Engine Helper to app-engine-patch, which has support for sessions in Google App Engine. Porting to app-engine-patch meant that I had to port the site to Django 1.0 also. I've chronicled my experience with this previously.

Today, I pinged the PyAMF folks on IRC and realized that there was a Google App Engine-specific version available. Unfortunately, that didn't work out of the box either due to a clash between its sessions implementation and the sessions implementation in app-engine-patch. That, however, was simple enough to fix.

If you're running app-enginge-patch and want to run the PyAMF shell, refactor the gateway.py module to rename the Sessions class to ShellSessions and Bob's your uncle. (It might be an idea for the PyAMF team to modify the example so that it works out of the box with app-engine-shell.)

Also, make sure that you protect the shell by only allowing admin access to it.

I can't begin to express how empowering it is to have shell access to your app on Google App Engine and I have no idea how I got by without it for so long. A big thank-you to the PyAMF team for making the shell app and releasing it. Great job, guys!

I would love to see a standalone Django app of the shell that you can simply plug in to your existing apps. Unfortunately, this is currently not possible since the app has static files and you need to add static folders manually to your application's app.yaml file. I've filed an ECR on this -- star it if you'd like that feature.

Update: Scratch that, I keep running into the 1MB limit for datastructures while trying to run queries, etc., through the shell, making it practically useless.

For one thing django.core.validators, which I was using, was removed. The fix: copy it back from the earlier version.

Not sure if it was a bad practice or not but I was also doing dictionary lookups on the request object and that's been removed now.

The errors you will get are:

• 'WSGIRequest' object is not iterable
• 'WSGIRequest' object is unsubscriptable

If you're in the same boat, here's the regular expressions you need to find/replace in TextMate:

Fix 'WSGIRequest' object is not iterable:

Find:

' in request(?!\.)

Replace:

' in request.REQUEST

'WSGIRequest' object is unsubscriptable:

Find:

request\['

Replace:

request.REQUEST['

Also, I switched from using Google App Engine Helper for Django to app-engine-patch. The main reason? So I could have the sessions middleware and run the PyAMF Shell. (And, of course, it's good to be running Django 1.0 now.)

App-engine-patch supports Django's mail feature so I stopped using google.appengine.api.mail.EmailMessage and replaced it with django.core.mail.EmailMessage. One thing to note if you do this is that you need to change the to properties on your EmailMessage instances to be lists, not strings, lest you end up trying to email each letter in the email address separately.

(Also, on my local server, I keep getting a Broken Pipe (error 32) since I upgraded to Django 1.0. I set fail_silently to True on my send() calls and that's fixed it. I don't have any issues on the deployment server. Looking into that one but it's a hairy one to track down.)

Today is your last chance to take advantage of the $99 early-bird discount for the <head> web conference.

Starting October 1st, tickets will be available at the regular price of $149.

Don't miss the early-bird discount and get your ticket now!

A quick reminder that the session I'm presenting tomorrow at FOTB has changed. My new session is called Grab the low-hanging fruit: or 5 rules for Hedonistic Creatives.

The session may have changed but, lest you worry, it's still taking place at the awesome time of 9am. So, once you've left tonight's party at Audio, head straight down to the Dome for some after-party fun!

Hoping to see at least three of you there tomorrow morning (and yes, that's you, Mom, Dad, and Stephanie!)

Don't forget to ask me for <head> moo stickers when you see me at FOTB this week!

Not at FOTB? You can order your own in just two clicks thanks to the magic of the moo API

(Oh yeah, and the $99 <head> early-bird discount ends in just two days so get your ticket if you haven't already!)