Recently, I found out that Google has discontinued their support for their SOAP API in favor of an AJAX interface. In non-Acronymian, this means that the Google Search API is now useless for anything except very boring, browser-based, Google-branded web search applications. This is grave news, my friends. With the SOAP API, developers could execute queries from nearly any programming language and get amazingly detailed results in a generic, granular format. With the AJAX interface, developers are limited to JavaScript, and the results are returned in pre-formatted HTML. It looks like some hackers have taken it upon themselves to unobsfucate the Google AJAX API in order to bring the granularity back to the API, but some say that this would be a blatant violation of the terms of service that you agree to in order to get your Google API Key, which would results in your application being blocked. Even if these hackers succeed, and even if someone writes an interface to convert the AJAX response into a more generic format, and even if they somehow do it all without violating the TOS, it is going to be significantly more difficult to use what was once the greatest API on the web. What a mess. Just the idea of purposefully obfuscating JavaScript makes me pound my fist vigorously on this table in disgust. It’s a perversion of nature! I’d like to say that this stuff will all sort itself out, but Google’s decision to scale back their generosity has brought them dangerously close to violating their corporate motto: “Don’t Be Evil”. And once Google goes evil, it’s all down hill from there.
From Google’s point of view, I can understand that the SOAP API did little for their corporate interests. People (like me) used the search behind the scenes in their applications without giving any kind of credit to Google. I mangled their results and used them for my own purposes without giving the user the “Google Experience”. I never show the user any AdWords banners, which is the bread and butter of the Google Empire. It is likely that their TOS mandates that you give credit to Google if you use the API, but I never read the TOS, and no one ever stopped me from using my application. It was there, I used it. The new AJAX API fixes these “problems” by limiting the usefulness of the result format, thereby preserving the Google Experience. In the end, they are a corporation, and if their prerogative is to strictly define what people can and can’t do with their service, of course it is their right. It’s just shocking to see a company change their tune so dramatically.
But if I were Google, I definitely would have handled it differently. Is the data that Google gains from tracking queries not enough to justify the relatively small number of queries coming from the SOAP API? Couldn’t they find a way to make it worth their while? Perhaps by providing an API that retains the level of granularity, but returns a click-through message to Google when one of the results of a query is used? There are hundreds of ways that the SOAP service could be modified in order to make it more useful for Google, but instead they took the easy way out. And for that, I chide Google. Bad Google!
I don’t have any hard evidence to back this up, but I suspect that having an open API only brings good things to a company. All you need to do is look at the ProgrammableWeb Matrix to see that people jump at the opportunity to put time, effort and creativity into making applications that use open APIs. These projects ultimately bring more attention to the respective companies, and more attention == more hits, which == more revenue2. For this reason, I believe that Google has made a huge mistake and ruined the best API on the web in the process. You can’t even really call it an API any more, unless you also call the Flickr badge creator an API. I worry now that Yahoo and MSN will follow suit, which they usually do.
But despite this blunder, Google is still one of the most generous companies out there when it comes to sharing their information, even if they are becoming a little misguided in how they share it. Companies like AllMusic go to great lengths to limit access to their data, and this seems to be the norm. Luckily, in the case of AllMusic, it wasn’t difficult to penetrate their defenses in order to screen-scrape the data I wanted, but other services have more thorough ways of obfuscating their data.
I mentioned in an earlier entry that one theme of my work with net art and Switchboard is the idea of taking back the information on the web from the increasingly commercial, closed interests. And although it causes a tear to form at the corner of my right eye to say so, Google is now on track to become one of these companies. I created Switchboard with the goal of making the web what it was originally meant to be: an open platform for sharing information. Basically, Switchboard is (or should be) a web hacking tool for artists. Of course it is not my place to tell Google how they should run their business. But the fact is, once something goes on the web, it is (TOS be damned) fair game to anyone who can figure out how to use it. This doesn’t fit so neatly into an economy based on scarcity, but it’s time to force people (and companies) to adapt. And for some reason, “figuring out how to use it” is something that I really enjoy doing, which is why I made Switchboard.
But the forces of evil are stronger than I alone. Even if I came up with the most robust XQuery to scrape data from some misguided data-hoarding website, eventually the data hoarders will catch on and break my code. For this reason, I have been trying to think of ways to make screen-scraping and other methods of data retrieval more accessible. Not just to hackers and script kiddies, but to anyone out there that wants to access the information on the Web in their own terms, to use for practical or artistic purposes. To this end, I have finally given a name to an idea that I have been playing with for a while, and that name is P3TS0RZ, which is leet-speak for Petz, the UbiSoft cyber-pet game.
P3Tz0rZ are cyber-pets that you train to do your bidding. From birth, they are mischievous hacking cyber pets. They open browsers and try to brute-force web forms, download pictures from the web and files from P2P networks, intercepting instant messages that it hears on the network, and basically does everything it can with the things that it probably shouldn’t be getting into – much like a real pet. As the owner, you must decide how to train your pet. Do you want your pet to be a spy and tell you want is going on on your network? Do you want it to download certain files from a P2P network? Or maybe something as simple as collecting weather reports from a website, or barking when there is a bid on an eBay auction that you are watching.
As you can see, the scope of this project is still pretty huge, but it is still in the early research phase, so it will definitely be trimmed down with time. More on t
his very soon.
Footnote: Google’s decision to convert their Search API to JavaScript actually gave me some great ideas for Switchboard. If JavaScript is the flavor of things to come, then I better hop on the bandwagon now and start coming up with some ideas. I am currently investigating the Rhino Javascript engine and considering a live-coding interface for Switchboard.
Fotnote 2: The wrapper from an 8-pack of Bounty paper towels just flew over the 12-story building across from where I am sitting. It looked completely comfortable in flight, and seemed to have no intention of landing any time in the near future. A balloon is not hot on its trail.
P3tz0rZ
Recently, I found out that Google has discontinued their support for their SOAP API in favor of an AJAX interface. In non-Acronymian, this means that the Google Search API is now useless for anything except very boring, browser-based, Google-branded web search applications. This is grave news, my friends. With the SOAP API, developers could execute queries from nearly any programming language and get amazingly detailed results in a generic, granular format. With the AJAX interface, developers are limited to JavaScript, and the results are returned in pre-formatted HTML. It looks like some hackers have taken it upon themselves to unobsfucate the Google AJAX API in order to bring the granularity back to the API, but some say that this would be a blatant violation of the terms of service that you agree to in order to get your Google API Key, which would results in your application being blocked. Even if these hackers succeed, and even if someone writes an interface to convert the AJAX response into a more generic format, and even if they somehow do it all without violating the TOS, it is going to be significantly more difficult to use what was once the greatest API on the web. What a mess. Just the idea of purposefully obfuscating JavaScript makes me pound my fist vigorously on this table in disgust. It’s a perversion of nature! I’d like to say that this stuff will all sort itself out, but Google’s decision to scale back their generosity has brought them dangerously close to violating their corporate motto: “Don’t Be Evil”. And once Google goes evil, it’s all down hill from there.
From Google’s point of view, I can understand that the SOAP API did little for their corporate interests. People (like me) used the search behind the scenes in their applications without giving any kind of credit to Google. I mangled their results and used them for my own purposes without giving the user the “Google Experience”. I never show the user any AdWords banners, which is the bread and butter of the Google Empire. It is likely that their TOS mandates that you give credit to Google if you use the API, but I never read the TOS, and no one ever stopped me from using my application. It was there, I used it. The new AJAX API fixes these “problems” by limiting the usefulness of the result format, thereby preserving the Google Experience. In the end, they are a corporation, and if their prerogative is to strictly define what people can and can’t do with their service, of course it is their right. It’s just shocking to see a company change their tune so dramatically.
But if I were Google, I definitely would have handled it differently. Is the data that Google gains from tracking queries not enough to justify the relatively small number of queries coming from the SOAP API? Couldn’t they find a way to make it worth their while? Perhaps by providing an API that retains the level of granularity, but returns a click-through message to Google when one of the results of a query is used? There are hundreds of ways that the SOAP service could be modified in order to make it more useful for Google, but instead they took the easy way out. And for that, I chide Google. Bad Google!
I don’t have any hard evidence to back this up, but I suspect that having an open API only brings good things to a company. All you need to do is look at the ProgrammableWeb Matrix to see that people jump at the opportunity to put time, effort and creativity into making applications that use open APIs. These projects ultimately bring more attention to the respective companies, and more attention == more hits, which == more revenue2. For this reason, I believe that Google has made a huge mistake and ruined the best API on the web in the process. You can’t even really call it an API any more, unless you also call the Flickr badge creator an API. I worry now that Yahoo and MSN will follow suit, which they usually do.
But despite this blunder, Google is still one of the most generous companies out there when it comes to sharing their information, even if they are becoming a little misguided in how they share it. Companies like AllMusic go to great lengths to limit access to their data, and this seems to be the norm. Luckily, in the case of AllMusic, it wasn’t difficult to penetrate their defenses in order to screen-scrape the data I wanted, but other services have more thorough ways of obfuscating their data.
I mentioned in an earlier entry that one theme of my work with net art and Switchboard is the idea of taking back the information on the web from the increasingly commercial, closed interests. And although it causes a tear to form at the corner of my right eye to say so, Google is now on track to become one of these companies. I created Switchboard with the goal of making the web what it was originally meant to be: an open platform for sharing information. Basically, Switchboard is (or should be) a web hacking tool for artists. Of course it is not my place to tell Google how they should run their business. But the fact is, once something goes on the web, it is (TOS be damned) fair game to anyone who can figure out how to use it. This doesn’t fit so neatly into an economy based on scarcity, but it’s time to force people (and companies) to adapt. And for some reason, “figuring out how to use it” is something that I really enjoy doing, which is why I made Switchboard.
P3Tz0rZ are cyber-pets that you train to do your bidding. From birth, they are mischievous hacking cyber pets. They open browsers and try to brute-force web forms, download pictures from the web and files from P2P networks, intercepting instant messages that it hears on the network, and basically does everything it can with the things that it probably shouldn’t be getting into – much like a real pet. As the owner, you must decide how to train your pet. Do you want your pet to be a spy and tell you want is going on on your network? Do you want it to download certain files from a P2P network? Or maybe something as simple as collecting weather reports from a website, or barking when there is a bid on an eBay auction that you are watching.
As you can see, the scope of this project is still pretty huge, but it is still in the early research phase, so it will definitely be trimmed down with time. More on t
his very soon.
Footnote: Google’s decision to convert their Search API to JavaScript actually gave me some great ideas for Switchboard. If JavaScript is the flavor of things to come, then I better hop on the bandwagon now and start coming up with some ideas. I am currently investigating the Rhino Javascript engine and considering a live-coding interface for Switchboard.
Fotnote 2: The wrapper from an 8-pack of Bounty paper towels just flew over the 12-story building across from where I am sitting. It looked completely comfortable in flight, and seemed to have no intention of landing any time in the near future. A balloon is not hot on its trail.