Sucking from the pipe...
Yahoo! Pipes, the excellent visual data mashup creator, has an unfortunate blockage problem. Specifically, it can pull in data from numerous sources, but the flow stops at the end of the pipe, waiting to be sucked out and consumed instead of automatically flowing out to its final destination.
When you create a Yahoo! Pipe, you can choose to consume sources such as RSS feeds, CSVs, or even regular web pages. You can then sort, filter, and generally mash up this data to create your own output feed.
Unfortunately, this is where the flow in Yahoo! Pipes stops. The output feed you create is made available in various formats, including RSS, JSON, and PHP. All formats that you need to pull in to consume.
... versus letting the pipe flow.
What Yahoo! Pipes could do is go one step further and use HTTP and the concept of web hooks to push the data out to custom endpoints. In other words, it could keep the data flowing.
In other words, once I have consumed my various data sources, mashed them up, and created an output feed, I should be able to push that to an HTTP endpoint on my web application. As the data changes (based on the frequency that Yahoo! updates the pipe), my application gets a notification and the latest data is pushed to it, removing the need for me to implement polling in my application and thus greatly simplifying things at the application level.
If you'd like to learn more about web hooks, Jeff Lindsay, recently gave a talk introducing web hooks at the <head> web conference in a presentation titled Web Hooks and the Programmable World of Tomorrow (<head> 2008 attendees can watch the video of Jeff's talk here).
The Yahoo! Pipes could flow to exciting new places with web hooks article by Aral Balkan, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial 2.0 UK: England License.
You can get half of this today using the WebService module in Pipes you can send the output of your Pipe to an endpoint and even further transform it using your own code. Here is an example using Google App Engine:
http://www.javarants.com/2008/04/13/using-google-app-engine-to-extend-yahoo-pipes/
If you want that to periodically call you back when the feed is updated, I suggest that you subscribe to the Pipe in an online RSS reader that will poll for you like My Yahoo, Google Reader or Bloglines. You won’t get deltas because there is no way to track which items a particular endpoint had previously received but it should be easy to track using the GUID in the pipe items.
yeah i was thinking about this too actually it seems like logical next step from pipes… it would be sooo easy to build powerful content agrigators…
i would love to see yahoo put a bit more work into pipes… it seems that since it was released it kind of stayed that way (perhaps i have not kept close enough tabs on it).
googles mashup offerings are not cool at all…
my 2cents..
Hey Aral,
Sorry, I couldn’t find a way to contact you on your site, so I am posting this here. I am a big fan of your blog as a fellow App Engine developer and have subscribed. However, no offense, but I am uninterested in the large volume of Qik.com posts a day that show up in your feed, along with your awesome posts.
Is there any way to filter just the regular blog posts without the Qik content?
aral you are big. thanks
Boy am I late to this. Yes! I thought this exact thought Aral! But the cool thing is that using something like Gnip or possibly something like RssFwd+Switchub, you can get just the functionality you’re talking about. Obviously it’s nicer to have less pieces involved, but whatever works!
Hey, nice tips. Perhaps I’ll buy a glass of beer to that man from that chat who told me to visit your site :)
+1 (at least since AppJet is dead)