I haven’t been blogging much about technology stuff, but lately I’ve been playing with some neat new technology that I think is worth writing about. There are a couple of pieces involved, so I’ll break this up over a couple of posts, but they all center around mashups, the blending of multiple data sources or services on the internet to make a new tool. There are a variety of new tools out there to make this sort of thing really easy to do. First up is Yahoo! Pipes.

Yahoo! Pipes (freely available, but you’ll have to have a Yahoo! account to use it) is a visual tool and is based on a “data flow” model, where you draw out diagrams showing how the data flows from a source, through operations that change it and finally out to its destination. You start off with a main workspace in which you can draw out your pipe. To the left of the workspace is a menu of different building blocks: these can be “sources” for the data (ways to pull data off of various websites or files), “operators” to transform the data in some way, some blocks for user inputs so you can let users of your pipe configure it, and then some special purpose building blocks to handle specific data types (numbers, dates, etc.). You create a pipe by dragging a source block (you can have more than one) onto your workspace, configuring it, and then drawing a line from the block to a special “Pipe Output” block at the bottom of the workspace. If you need to transform the data in some way, you would insert the correct operator blocks in between.

A Simple Pipe

I’ll make this more concrete with a specific example. I love web comics and read my favorites every day. I would love to be able to have a single list of all the most recent updates to my favorite web comics so I can see at a glance when there is something new to read. Fortunately, many of the web comics I read are pretty savvy and have RSS feeds. Wouldn’t it be convenient if I could mash all those RSS feeds together and have just one to look at? This is easy to do in Pipes. Here are the step-by-step instructions (assuming you’ve got a Yahoo! account and have gone to the Pipes page):

  1. Click “Create Pipe.”
  2. When the workspace opens up, click the “Sources” label in the left channel menu and drag “Fetch Feed” to the workspace. You should get a small rectangular block in the workspace. You’ll see a plus sign labeled “URL,” and a text box with a minus sign. There is also a funny little bloop at the bottom of the block, which is a connection point. The “Fetch Feed” block takes the RSS data it finds at the URL you provide and makes it available to the pipe on the connection point on the bottom of the block. You can give “Fetch Feed” more than one URL, in which case it mixes the data all up in a bucket and makes all the results available on that connection point.
  3. Paste the URL for the first RSS feed you want to use in the text box.
  4. Click the plus sign, and a new text box should appear.
  5. Paste the URL for the next RSS feed in the new (empty) text box.
  6. Repeat #4 and #5 for as many feeds as you like.
  7. Drag a line from the connection point on the bottom of the “Fetch Feed” block to the connection point on the top of the “Pipe Output” block at the bottom of the workspace (the “Pipe Output” block is a special block that indicates what the results of the pipe will be). You should see a curvy connection form between the two… this means that the output of the “Fetch Feed” block will now go to the output of the entire pipe. You should also see some activity happen and a list of items (the results of running the pipe) should appear in a text area below the grid of the workspace. The results should look like the picture below (although you don’t need to use the RSS URLs I did).
  8. If everything looks right, go ahead and save your pipe. You can then look at it in the “My Pipes” page and see what it looks like when you run it for real. Also, from the pipe’s page in “My Pipes,” you can subscribe to the pipe data in a variety of RSS readers, or get the RSS URL to use in some other reader of your choice.

Basic Comics Pipe

A Filtered Pipe

This is great and everything, but it shows every comic in all the feeds… in some cases even ones that are pretty old. Wouldn’t it be nice if we could filter things a bit and just show the most recent items… say, only those posted within the past seven days? Well, we can quite easily by using an operator:

  1. Click the connection point on the top of the “Pipe Output” block.
  2. Click the scissors icon that appears on the connection to break the connection between the “Fetch Feed” block and the pipe output.
  3. Click the “Operators” label in the menu.
  4. Drag “Filter” to the workspace a little bit below your “Fetch Feed” block. A new block labeled “Filter” will appear with a lot of options. Operators are blocks that take feed data as input, make some change, and send the results out for further processing. In this case, the “Filter” operator takes in feed data (on its top connection point) and throws away items that match certain criteria you specify. What’s left over is made available for the next stage on the bottom connection point.
  5. Drag a connection from the bottom of the “Fetch Feed” block to the top of the “Filter” block.
  6. Set the options on the Filter block so you “Block items that match all of the following rules.”
  7. Select “item.pubDate” in the first drop-down box of the rule (this is the property of the feed item we’re going to check… in this case, the date the item was published).
  8. Select “is before” as the test to make.
  9. Type “1 week ago” into the text box. (The “Filter” block will now look at each item in the source RSS feeds, and throw out any item which has a publication date (pubDate) that is older than seven days ago). Isn’t it nice that the “Filter” block understands relative date phrases (like “1 week ago”)?
  10. Drag a connection from the bottom of the “Filter” block to the top of the “Pipe Output” block.
  11. You should be able to refresh the test data at the bottom of the screen and save the updated feed. It should look like the picture below. You should see that the number of items in the results is lower than before… at the very least, nothing in the results should be more than a week old.

Recent Comics

If you’d like, you can see my version of this pipe on my Yahoo! Pipes page. It’s called “Favorite Comics.”

There’s a great deal more to Yahoo! Pipes than what I’ve written here; this is just a very basic example. If you’re interested, you can explore and get a feel for the possibilities. But, look at what we’ve done. In the span of just a few minutes, we’ve created a new RSS feed that brings together a variety of our favorite RSS feeds and filters out items we don’t want (those that are too old). In other words, we have pulled together data from a variety of sources, tailored it to our specific desires, and then bundled it up so we can use it. We’ve just scratched the surface however.

Next time: so, now that you have a feed of your own, how can you use it?