Entrepreneurial Research Tools: Build Your Own Feed Mashup With Yahoo Pipes

Sunday, July 29, 2007 at 9:30pm by Site Administrator

Wondering how to create the 100 Daily Must-Reads for Entrepreneurs monster mashup feed? Functionally-speaking, I mashed up the RSS feeds of all 100 blogs and sorted the results reverse-chronologically by date. In reality, because someone else put the list together, I had to start from the raw HTML page and go from there. Yahoo! Pipes was my tool of choice.

If you have not worked with Yahoo Pipes before, I recommend you check out the Pipes tutorials at Tubetorial first. You don’t have to be a programmer to use Pipes. In fact, that’s what’s so great about it as a quick research tool.

Productivity Blogs Feed Mashup

Now, since productivity is an important part of successful entrepreneuring, I thought I’d supplement 100 daily must-reads feed mashup with another one, using Zen Habits’ Top 50 Productivity Blogs list. Some of the blogs in that list are in the Bootstrapper list, but not all. If you feel like subscribing to the Zen Habits list en masse, you’re in luck. I’ve taken pretty much the same Yahoo Pipe used in the 100 blogs mashup and created a 50 Productivity blogs feed mashup. On top of this, I’ve also produced a screencast video tutorial showing you how to use Yahoo! Pipes to do your own similar feed mashups.

For those of you not interested in the tutorial, here are some Pipes you can run, or the corresponding feeds you can subscribe to. Keep in mind that you can clone all of the Pipes and tweak them to your liking. If you want to view the tutorials, skip this list.

  1. Zen habits -50 productivity blogs -full list:

    • Yahoo Pipe results/ feed.
    • This Pipe takes all 50 blogs and mashes up their RSS feeds, filters them for unique items, sorts them by freshest items first (reverse chronological order).
  2. Same mashup, truncated to 200 items:
    • Yahoo Pipe results/ feed.
    • This is the same as the last Pipe, but the number of mashup feed items is limited to 200 items.
  3. Searchable, truncatable feed mashup:
    • Yahoo Pipe results.
    • Once again, the same Pipe, but users can enter a search term. They can also enter the desired number of items in the resulting mashup feed. Because Pipes auto-generates the RSS feed URL based on a pipe’s id and any user-entered parameters, there is no single feed. Run the pipe with your preferred search term and number of items, then copy the resulting RSS feed URL.

How-to Build Your Own RSS Feed Mashup in Yahoo Pipes

The rest of this article is a video screencast tutorial on how to build a feed mashup using Yahoo! Pipes. We’re starting with a pre-built .CSV file of the 50 productivity blogs that Zen Habits listed. You do not have to be a programmer to use Pipes, but it takes a bit of getting used to.

Process
The mashup process goes like this:

  1. Convert the HTML page list of blogs to a list of blog urls.
  2. Save the url list as a CSV (Comma Separated Value) file:
    • Insert an arbitrary field name for the urls. I used “blogurl” in the very first row of the file.
    • Save the file with a .CSV extension.
    • Upload the file to a web server so that it can be access online.
  3. Yahoo Pipes:
    1. Import the CSV file.
    2. For each blog url in the CSV list, auto-discover any RSS feed urls.
    3. Filter out any unnecessary feed urls. (Some blogs have multiple feed formats. We just need one.)
    4. For each feed url in the resulting shortlist, grab the actual feed and mash them all together.
    5. Filter out any duplicate items by title.
    6. Sort the mashup feed so that the freshest items come first.
  4. Run the Pipe and copy the auto-generated RSS url into your favorite feed reader.

Videos
Each of the three Pipes for the productivity blogs list are accessible to the public. Provided you have a Yahoo! Mail account, you can clone any Pipe, tweak and save it. If you want to see how the above three Pipes were built, you can watch the narrated screencast video below. I’m using the SplashCast Media‘s SplashCast player. Once you click the Start button, the first video will play. There are three videos, and the player will automatically move to the next one. If you want to skip forwards, hover your mouse cursor over the player, and a menu bar will drop down. Click on the “forward” icon to move to the next video. Total playing time for all three videos is approximately 20 minutes.


Add bootstrapper-productivity blogs-feedmash to your page



If you enjoyed this article, please bookmark it at del.icio.us »

AddThis Social Bookmark Button

2 Comments »

  1. Yet another great Pipes tutorial Raj. I think you’ve cornered the market on these. Do you have any suggestions how to convert opml to csv file? An easy way to do that would open up this feedmash to a whole range of possibilities.

    Comment by Shanta — August 1, 2007 @ 11:19 am

  2. Hi Shanta, thanks for the compliment.

    Re converting OPML to CSV. That’s a great question. Short of a manual hack, you’d need a script, which is now on my to-do list.

    However, if you just want to feed in an OPML to a Yahoo Pipe, you can use the new “Fetch Data” module and specify the URL of your OPML file. Here’s an example Pipe that someone else wrote:

    Comment by Raj Dash — August 1, 2007 @ 1:48 pm

RSS feed for comments on this post.

-->

Leave a comment