Mash-ups with YQL

Nov7
  • Share
…and why Corizon mashups suck.

Difficulty: ★★☆☆☆
It was because of working with Corizon that I became more interested in creating Mash-ups*.
During the last 2 years, I’ve seen a lot of mash-up tools, From the easy (”Lego-like”) Popfly from Microsoft which is recently discontinued, to large handy API’s or serverside Java frameworks; such as the nice but heavy SiteMesh framework.
But it was the Corizon mash-up toolkit that I started to realize, coding mash-ups can have it’s ups and downs.

In my opinion creating a mash-up should be easy & fast. Not buggy and heavy (running 2 or 3 Tomcat servers in the background *kuch*). Also it’s an unwritten rule that for a mash-up you should not first destroy and rebuild it’s components before you gonna mix them up. This means you can only mash-up content what’s in your maintenance and when one of your components changes, your mash-up is broken.
It’s not a wonder that I became extremely positive about YQL, when I first saw this working.
And now, after Fronteers 2009, where I had a presentation and talk with Chris Heilmann from Yahoo.

With YQL (Yahoo Query Language), you can make the same mash-ups, client or serverside and you don’t need thousands lines of code.
Why you should use YQL for a mash-up?

  • You won’t use your own bandwidth, but the bandwidth from Yahoo.
  • It’s easy, just the basic SQL knowledge is required.
  • It’s lightweight.
  • It’s fast.

But like every Mashup tool, there are also downside’s. So it’s good to consider:

Why you shouldn’t use YQL for a mash-up?

  • When one of the including components is slow or has timeouts, you result object will be null.
Mash-ups with YQL

Ok enough with the theory. Let’s practice with a simple YQL mash-up. As you might know, I like videogames. So I want to have a website with the release dates of games on my website, which I can watch on YouTube. Fokzine Games (http://games.fok.nl/overzicht.php?action=release) has a nice game release list.

  • For using this tutorial, you will need a Yahoo Developer Account, which you can create for free.
  • Unless you like to write full XPath lines without any help,
    you’ll also need Mozilla Firefox browser with the Firebug plugin installed.
  1. Register your Yahoo Dev account.
    Go to the below URL, register your Yahoo account and login
    to the YQL console: http://developer.yahoo.com/yql
  2. Query the content.
    YQL is quite similar to SQL. To query content (the videogame
    release list) from an external website, I will select all the HTML nodes from an URL. With Xpath I can point to the part of the website I will need. (Now you’ll need Firebug, inspect the website, till you’ll reach the table tag of the release list. – In top of your Firebug screen, you will see the full Xpath line.
    – Just right click, and “Copy Xpath”. This will be your YQL query for selecting the table tag on Fok.nl Game Release Page:

    select * from html where url ='http://games.fok.nl/overzicht.php?action=release' and xpath ='//*[@id="itemtext"]'
//or select the tr tag on Fok.nl Game Release page
//(note that I removed “tbody” from the xpath line to make it work)
select * from html where url ='http://games.fok.nl/overzicht.php?action=release' and xpath ='/html/body/div[4]/table/tr/td[2]/div[2]/div/table/tr'

    Hit enter (or press “Test”) and you can switch between formatted view or tree view to see the result. Result can be in XML of JSON.
    Also note that you can copy the REST URL Query, from the console:

    http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%20%3D%20′http%3A%2F%2Fgames.fok.nl%2Foverzicht.php%3Faction%3Drelease’%20and%20xpath%20%3D%20′%2Fhtml%2Fbody%2Fdiv%5B4%5D%2Ftable%2Ftr%2Ftd%5B2%5D%2Fdiv%5B2%5D%2Fdiv%2Ftable%2Ftr’&format=xml

    YQL Console

  3. How the retrieved XML looks like:

    <query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng"yahoo:count="50" yahoo:created="2009-09-25T12:14:12Z"yahoo:lang="en-US" yahoo:updated="2009-09-25T12:14:12Z"
yahoo:uri="http://query.yahooapis.com/v1/yql?q=select+*+from+html+where+url+%3D+%27http%3A%2F%2Fgames.fok.nl%2Foverzicht.php%3Faction%3Drelease%27+and+xpath+%3D+%27%2Fhtml%2Fbody%2Fdiv%5B4%5D%2Ftable%2Ftr%2Ftd%5B2%5D%2Fdiv%5B2%5D%2Fdiv%2Ftable%2Ftr%27">
<diagnostics>
<publiclyCallable>true</publiclyCallable>
<url execution-time="263" proxy="DEFAULT"><![CDATA[http://games.fok.nl/overzicht.php?action=release]]></url>
<user-time>272</user-time>
<service-time>263</service-time>
<build-version>3130</build-version>
</diagnostics>
<results>
 <tr bgcolor="#E0E0E0">
  <td>
   <img alt="PC" src="http://images.fok.nl/ads/games/IB_PC.png"/>
  </td>
  <td>
   <p>29-09-2009</p>
  </td>
  <td>
   <a class="invisiblelink" href="gameinfo.php?gameid=9985">Tales of Monkey Island episode 3</a>
  </td>
 </tr>
  ..
</results>
</query>

    How your JSON (Javascript object looks like:

    releaseList({
 "query":{
 "count":"50",
 "created":"2009-09-25T12:10:17Z",
 "lang":"en-US",
 "updated":"2009-09-25T12:10:17Z",
 "uri":"http://query.yahooapis.com/v1/yql?q=select+*+from+html+where+url+%3D+%27http%3A%2F%2Fgames.fok.nl%2Foverzicht.php%3Faction%3Drelease%27+and+xpath+%3D+%27%2Fhtml%2Fbody%2Fdiv%5B4%5D%2Ftable%2Ftr%2Ftd%5B2%5D%2Fdiv%5B2%5D%2Fdiv%2Ftable%2Ftr%27",
 "diagnostics":{
  "publiclyCallable":"true",
  "url":{
   "execution-time":"28",
   "proxy":"DEFAULT",
   "content":"http://games.fok.nl/overzicht.php?action=release"
  },
 "user-time":"36",
 "service-time":"28",
 "build-version":"3130"
 },
"results":{
 "tr":[{
 "bgcolor":"#E0E0E0",
 "td":[.. ]
 }
..
 ]
}}});
  4. Use the YQL widget.
    Create a new HTML page. In this page, you’ll include the YQL JS Widget:
    http://github.com/sh1mmer/dotnet-yql-tutorial/blob/master/yql_js_widget.js
    This is a pre-made Javascript widget. If you check out the links on the end of this article, you can check out other implementations as well. (Such as Dojo-YQL or JQuery YQL).
    For this article, we will use vanilla Javascript.
  5. Create the Mash-up.
    The only think what you’ll need now is the logic to implement the previous made YQL query, to have the YQL results in your Document Object Model (DOM).
    With the Firebug plug-in, you can check the “Net” tab, to see if the correct object is retrieved. You can walk through the results, and implement it in your Javascript to do fancy things with it.
    6. <html>
<head>
<title>YQL - Game Release</title>
</head>
<body>
<script type="text/javascript" src="yql-js-widget.js" ></script>
<script type="text/javascript">
var config = {};
var format = '<li class="{td[0].img.alt}">{td[1].p} - <a href="http://games.fok.nl/{td[2].a.href}" target="_blank">{td[2].a.content}</a> - <a href="http://www.youtube.com/results?search_query={td[2].a.content}" target="_blank">Watch on YouTube</a></li>';
var gameQuery = "select * from html where url = 'http://games.fok.nl/overzicht.php?action=release' and xpath = '/html/body/div[4]/table/tr/td[2]/div[2]/div/table/tr'";
yqlWidget.push(gameQuery, config, format, "widgetContainer");
yqlWidget.render();
</script><ul id="widgetContainer"></ul>
</body>
</html>

    Well that’s it. See how easy it is to create a nice Mash-up? As long as you can reach the DOM, you can mash-up everything. Below my article, I’ve written some nice links. If you liked my article or you would like to discuss with me about YQL or Mash-up technologies; feel free to add me on LinkedIn or Twitter.

    * In webdevelopment, a mashup is a web page or application that combines
    data or functionality from two or more external sources to create a new
    service. An example of a Mash-up is a Google Map combined with Wikipedia
    town information and a (accuweather.com) Weather indicator.

, / , / , / , /

  » Post your comment, there are no comments yet. »
Posted in YUI Library, javascript, opensource |

Leave a comment