Archive for the 'google app engine' Category

Winning at the shell game: iPython on Google App Engine

iPython is an awesome extended Python shell that gives you goodies like tab completion for instances, history tracing (so you can easily copy interactive sessions as doctests), etc. And, if you install it, your Django project on Google App Engine will automatically start using it instead of the regular python shell when you use ./manage.py shell.

To install iPython on OS X Tiger (yes, my Leopard discs are still safely in their box since I downgraded and I don't see any reason to bring them back out yet), I followed the following steps:

  1. Download the latest iPython from the iPython distributions page (ipython-0.8.4.tar.gz)
  2. Untar it, cd into the folder
  3. As per the instructions on the iPython download page:
    python setup.py build
    sudo python setup.py install
  4. To test it out on my Google App Engine/Django project, from my project folder: ./manage.py shell

(Note: The docs mention that you need to have readline installed on Mac OS X in order to use some of the features like tab completion and syntax highlighting. It just worked out of the box for me on OS X Tiger 10.4.11 -- I'm not sure if I had installed readline at some point or whether it was just there. Check out these instructions if you're having trouble.)

Once you have it installed, try out the cool code completion:

from my_app import models
models.

Press ⇥, and you'll see a list of all your models. models.my_model. ⇥ will show you the properties for that model and so on.

To create doctests, simply enter your test instructions in the shell and then type hist -n to get a dump of your history without line numbers that you can copy and paste into your doctest.

You can press ⌃ P and ⌃ ⇧ P to interactively bring up the previous and next commands in the history. If you've typed a bit of the command before doing this, it will filter to show you only those commands from your history that match the text you've entered.

You can also access the system shell without leaving iPython by preceding system calls with an exclamation mark. !ls, for example, will show you a listing of the current working directory.

And there's much more you can do that you can read about on the iPython documentation (or just type ? in the iPython shell itself and browse the docs interactively).

Check out iPython, it's yummy!

I found out about iPython from an excellent blog post by AkH on useful tips and good practices for Django projects. Thanks, dude!

Don’t call your Django application “site” in Google App Engine

This one had me scratching my head for a while today:

I created a Django application called "site" in my project, and although everything was working perfectly locally with the Google App Engine SDK and Google App Engine Django Helper but when I deployed, I got an ImproperlyConfigured error.

Changing the name of the application fixed the issue.

I'm off to file a bug on this... done: Issue 487.

Update: Please star Issue 404 to bring OpenID to Google App Engine

Issue 404 is the key to getting OpenID working on Google App Engine but currently it only has 10 stars.

Please star Issue 404 to get OpenID support on Google App Engine.

(Please add a star, _not_ a "me too" comment which will get emailed to everyone who has starred the issue and may cause other people to unstar it.)

Ryan Barrett from the Google App Engine team has built both OpenID provider and OpenID consumer libraries (open source) that you can reach at http://openid-provider.appspot.com/ and http://openid-consumer.appspot.com/ (uses version 2.1.1 of JanRain's OpenID library and is compatible with OpenID 2.0 providers).

Once this bug is fixed, those existing libraries will just work and we will have OpenID support on Google App Engine.

Please star Issue 404 so OpenID doesn't stay 404 on Google App Engine.

Running doctests from TextMate for Google App Engine modules

Developer emptor: I just lost a couple of hours to this: make sure you disable the Google App Engine doctest import in your apps when you're done testing a module lest you encounter _weird_ errors. I started having the login URL returned by users.create_login_url() being returned incorrectly when I forgot to remove the doctest import. It started forwarding to https://www.google.com/accounts/Login?continue=. Check out my forum post on it here.

I love Python's doctests. Basically, you test out your functions in the interactive shell and copy the results into the comments for a function. That's it! So simple.

Example:

def http_request(self, url, data, method=urlfetch.POST):
	"""
	Makes an API call to Triggermail and returns the response.
 
	>>> client = TriggerMail()
	>>> client.http_request('email', {'email':'EMAIL_REMOVED'}, urlfetch.GET)
	{u'blacklist': u'0', u'templates': {u'test2': 0}, u'verified': u'0', u'vars': {u'first_name': u'Aral', u'last_name': u'Balkan'}, u'optout': u'0'}
	"""

To run the doctests, you just need a main method in your module that looks like this:

if __name__ == "__main__":
	import doctest
	doctest.testmod()

And, if you're working with TextMate, you can run the current script and its doctests by pressing ⌘ R.

Sweet!

However, when working with Google App Engine, this doesn't work out of the box.

If you try it, you'll get an error similar to the following:

ImportError: No module named google.appengine.api

This is because the local GAE environment isn't set up properly. The same goes when trying to test your apps from the Python interactive shell.

(If you're using Django for your app, you're in luck, all you have to do is ./manage.py shell and you're up and running with an interactive shell that's configured for your GAE project.)

Thankfully, Duncan over at the GAE forums went to the trouble of finding out exactly which imports are necessary to get you up and running.

His code listing actually goes beyond setting up the environment to finding your modules and running the tests. For my purposes, I just want to be able to hit ⌘ R in TextMate and run the tests for my current module while developing it, so I took the top bit of his code and put it into a module called gae_doctests.py.

It looks like this:

# To enable doctests to run from TextMate, import this module
# (Use only when testing, then comment out.)
# From: http://groups.google.com/group/google-appengine/browse_thread/thread/fa81f6abd95aa8b9/efed988b302aafb4?lnk=gst&q=duncan+doctests#efed988b302aafb4
 
import sys
import os
sys.path = sys.path + ['/usr/local/google_appengine', '/usr/local/google_appengine/lib/django', '/usr/local/google_appengine/lib/webob', '/usr/local/google_appengine/lib/yaml/lib', '/usr/local/google_appengine/google/appengine','/Users/aral/singularity/']
 
from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub
from google.appengine.api import mail_stub
from google.appengine.api import urlfetch_stub
from google.appengine.api import user_service_stub
 
APP_ID = u'test_app'
AUTH_DOMAIN = 'gmail.com'
LOGGED_IN_USER = 't... {at} example(.)com'  # set to '' for no logged in user
 
# Start with a fresh api proxy.
apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
 
# Use a fresh stub datastore.
stub = datastore_file_stub.DatastoreFileStub(APP_ID, '/dev/null', '/dev/null')
apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub)
 
# Use a fresh stub UserService.
apiproxy_stub_map.apiproxy.RegisterStub('user',
user_service_stub.UserServiceStub())
os.environ['AUTH_DOMAIN'] = AUTH_DOMAIN
os.environ['USER_EMAIL'] = LOGGED_IN_USER
 
# Use a fresh urlfetch stub.
apiproxy_stub_map.apiproxy.RegisterStub(
    'urlfetch', urlfetch_stub.URLFetchServiceStub())
 
# Use a fresh mail stub.
apiproxy_stub_map.apiproxy.RegisterStub(
  'mail', mail_stub.MailServiceStub())
 

(Either replace the /usr/local/ bit with the actual path to your GAE install or use Duncans code which is neater -- I was lazy and copied the contents of the sys.path list from the Django interactive shell.)

To use it, simply:

import gae_doctests

And hit ⌘ R in TextMate. Sweet!

Once I'm done hacking away on a module, I simply comment out the import.

Could not embed video.

OpenID on Google App Engine (and how you can help make it happen!)

Update: Ryan just updated me that the issue we should be starring for this is Issue 404 not Issue 17.

You can almost but not quite create an OpenID consumer on Google App Engine today using python-openid.

Unfortunately, redirects to external OpenID providers are failing and Google currently does not think that this is a high priority to fix (this is where you can help, read on.)

Ryan from the Google App Engine team had this to say:

We do intercept some redirects for security reasons, which breaks apps like openid consumers and providers. This is why http://openid-provider.appspot.com/ currently fails on many consumers.

It's not a high priority, but we do plan to look into this eventually. Feel free to file an issue on the issue tracker and star it; if it gets enough votes, that will help convince us to prioritize the fix!

Ryan even has an open source OpenID consumer example* that would be working if it wasn't for this bug.

Read this thread on the forums for all the juicy details.

How you can help

There is already an issue where this is being discussed: Issue 17 - OpenID Support Issue 404 (When urlfetch.fetch is used to retrieve redirected web page, the new URL location is not given).

Please _star_ this issue to show your support and join the conversation on the forum.


* svn checkout http://google-app-engine-samples.googlecode.com/svn/trunk/ google-app-engine-samples-read-only to get the OpenID consumer example as well as the other Google App Engine examples.

Google App Engine not blocking PayPal?

Update: Marzia from the Google App Engine team responded to the post. It is a bug (and, according to yet another update on the forums, this has now been fixed).

Here's Marzia's comment:

This is a bug, and we have located the problem. There was an error in our anti-phishing protections that was blocking some specific URL domains from being fetched using the URLFetch service. This was an oversight on our part, and these specific domain restrictions will be removed in the next few days.

Great news, Marzia, thanks! :) It's definitely going to make my life easier to be able to reach PayPal from Google App Engine.

Original post follows:

You know that I am very optimistic and supportive of Google App Engine but something I read today has made me a little worried.

It appears that Google App Engine is deliberately blocking PayPal URLs. At first people thought this was due to a technical glitch, but, according to Petko D. Petkov on the Google App Engine forum, the calls succeed when using a forwarding URL (like TinyURL).

To quote Petko's post:

Apparently Google blocks URLs to paypal but with a bit of creativity we can bypass this restriction. . .

Requests to:

https://www.paypal.com/cgi-bin/webscr
https://www.sandbox.paypal.com/cgi-bin/webscr

are blocked . . . in order to bypass them we need to change the paypal URLs to something different. For example, we can use tinyurl. . .

http://tinyurl.com/3ro7da

which is actually

https://www.sandbox.paypal.com/cgi-bin/webscr

If we send the post verification to that URL, we bypass the restriction

If this is true, I can't see how this move demonstrates good faith on Google's part. It doesn't seem to gel with their "do no evil" policy or, as Petko states, their championing of the open web.

So far, Google has been completely silent on this issue.

I hope that changes. And I hope this is a bug.

At the very least, someone from Google needs to explain to us why they are blocking PayPal URLs.

If this is true, it sets a dangerous precedent that should give any developer considering Google App Engine pause: Will Google use its position of power to dictate which services and web sites your applications will be able to access? We need a clear policy statement on this.

So, Google, what do you say?

“Google App Engine: To Django or to webapp?” Revisted

When I was first starting out with Google App Engine (GAE), I wrote a short post detailing my thoughts on whether to use Django or Google's own webapp framework for GAE projects. In that post, I concluded that "there isn't a compelling reason to use Django at the moment with GAE".

Since then, I've re-evaluated my decision and, a few weeks ago, I ported the Singularity web app to Django from webapp.

Here are the main factors that contributed to my decision:

  • I wanted to use features in the latest Django trunk (0.97+) and the Google App Engine SDK ships with 0.96. I also did not want to be tied in to updates from Google for the Django framework.
  • Google released the Google App Engine Helper for Django which makes it very simple to set of a GAE Django app.
  • And, finally, looking through the source of Rietveld, I saw that Guido van Rossum is using Django for his Google App Engine app. And heck, I think he knows a thing or two that I may not.

Going with Django means that you can take advantage of more existing functionality (Django middleware, etc.) and, should you want to move away from Google App Engine in the future, there will be less code to rewrite. (Though, realistically, if you're doing anything with data, you're pretty much tied in to the DataStore at the moment until there is a suitable, scalable alternative that you can port to without completely refactoring your data layer.)

So my new take on this is that it makes a lot of sense to build Google App Engine applications using Django instead of pure webapp.

To get started, check out the Google App Engine Helper for Django.

Google App Engine SDK 1.0.2

Google released version 1.0.2 of the Google App Engine SDK two days ago. If you're using Google App Engine, you should definitely upgrade.

You can find a list of fixed issues in the release notes, including the template cache issue that I was running into.

Also, it's not in the release notes but I just confirmed that the app.yaml skip_files issue has also been fixed. So you don't need the patch any longer.

I also tested the latest version of The GAE SWF Project and it works out-of-the-box with version 1.0.2.

Picture this!

In version 1.53, you can now add a photo to your profile at The GAE SWF Project Flex sample (and see other user's profiles).

It was a bitch to get FileReference working as I wanted it to (look ma, no sessions!), but it was well worth the effort. The sample uses an authentication-token based system instead of the less secure hack of tacking the session ID on to the request URL method (yuck!)

The example demostrates File Upload via File Reference, outputting an image via Python/webapp and both sending and returning images as ByteArrays from webapp.

The PhotoCropper component lets you upload photos, crop them, and rotate them. I started building it by expanding on Arpit Mathur's ScrollImage component (thanks, Arpit!) :) It's not complete yet but it is functional. Specifically, I don't like how it zooms and a future update will handle that. If you notice any issues, please do let me know in the comments so I can tackle them.

The updated ScrollImage class includes the following features:

  • Correctly scrolls both zoomed in (<1) and zoomed out (>1) content.
  • Correctly frames scrolled content so that no whitespace shows.
  • Supports the loading of images via the load() method.
  • Applies smoothening to the bitmap when zoomed in/out (not when at 100%).
  • Implements "natural" click and drag scrolling (reversed).
  • Image does not get stuck to the sides when user keeps scrolling after an edge has been reached.
  • Component displays the hand cursor when over the image.
  • Allows rotation of the image.
  • Provides a ByteArray snapshot of the image.

I also made use of Joel Connect's FlexMatrixTransformer class (thanks, Joel!) :)

Remember that The GAE SWF Project is completely open source so you can grab the source code and use it to build your own applications (have fun!)

Getting started is as easy as 1,2,3:

  1. Download Google App Engine and install it.
  2. Download The GAE SWF Project and unzip it.
  3. Navigate to the folder you unzipped it to in Step 2, and start the Development Web Server by typing ./start in Terminal. Go to http://localhost:8080 to see your local version of The GAE SWF Project and start hacking!

(The above instructions are for OS X, please adapt as necessary for your platform.)

The features I'm spiking out and sharing with you in The GAE SWF Project are the ones I'm using to build the web application for the Singularity Web Conference. Expect some updates on that front in the coming days. I am going to release the Singularity app and open ticket sales as soon as the core features are in place.

Also expect some more cool Singularity-related speaker and sponsor updates this week :)

Oops, I broke Google?

Update: They're on it! :) In the meanwhile, the suggested workaround is to add debug=True to your template.render() methods (pass it as the third argument, i.e., after the context).

Update: As of 2.30pm (when I last checked), the app appears to be working again. I've made no changes whatsoever.

If you hit the Flash or Flex client on The GAE SWF Project right now, you will be greeted with a page long traceback.

I've just opened issue 273 on the Google App Engine Issue Tracker to track this and added this comment on the forums.

The problem is that the error never occurs locally when developing and testing with the Development Web Server and randomly starts happening on appspot (the app will work well for stretches of time and then start failing 100% of the time.)

The last time this happened, issuing an appcfg.py update . fixed the issue but I'm not going to do that now in hopes that seeing the error will make it easier for Google's engineers to fix it.






Bad Behavior has blocked 0 access attempts in the last 7 days.