Twitter has been providing a steady stream of updates over XMPP in past. This firehose was being used by many applications on internet. It was a pretty useful stream of updates, but managing it was a infrastructural nightmare.Twitter’s frequent downtimes have been very well documented, part due to blogosphere’s soft corner for twitter, and part due to its own inability to address the problem in time. There have been many after effects of these frequent downtimes and the enormous buzz thy used to create in media. The remedial plan included many cutbacks, one of which was limited access of firehose to developers. Many startups had to close shops due to this differential treatment.

The lucky few having access to firehose are google, bing adn others, which is understandable as they bring traffic to twitter. But all developers deserve a fair chance to access the api and build apps.

Considering that all who have access to firehose are considerably bigger and technologically capable entities, they should further distribute the firehose using their own resources. Although XMPP has scale issues, but distribution can be a chain where an entity is responsible to distribute to n other entities, and all these entities further distribute it to others. In short, whomsoever uses the firehose, distributes it further through XMPP. This will help to get the burden off the twitter infrastructure and at the same time provide access to multiple developers.

This infrastructure sharing paradigm has worked wonders in past in open source community by setting up mirrors at different universities and corporations. They used to serve static files through ftp/http. This has been a proven distribution model and can be further evolved to serve semantic content. XMPP proxies are not tough to setup. XMPP is already a well known distributed platform and the piece of software to facilitate such registration and distribution is not very tough to build.

Twitter has already announced its will to share the firehose with developers again in early 2010. By enforcing the sharing constraints, it can easily start an example, which might fuel content sharing in future.

Video startup boxee is promoting its apis. I saw a mail in my inbox a couple of days back regarding developer challenge over building apps for boxee. A quick run through apis told me that the apps are dead simple to build and are primarily content driven. In short it is an effort to put more content on boxee dashboard for users to consume. I use it sometimes on my ubuntu box.

Here is an idea for someone who can devote some time for build an boxee app. Many universities are hosting their channels on youtube. I myself am a great fan of Prof. Leiserson’s algorithm lectures available here. It would nice to have some of those on boxee. Good education is not a previlege, but should be a fundamental right to all. These lectures will definitely benefit many to whom this sort of quality is not available.

In case anyone has some bandwidth to develop, i am ready to assist

Twitter has been arguably most cloned startsups in recent times. It has the reputation of a startup that has been cloned in most languages across the globe. Amazingly simple idea, and outstanding execution have been drivers behind its success. Simplicity has helped it evolve in multi dimensions as well. However, an important factor behind twitters growth has been the APIs, which have been used by hackers to build multifaceted, useful and interesting applications. You can easily estimate a coarse number of “powered by twitter” apps by the fact that twitter api receives more than 10X traffic than twitter website (as told by twitter co-founder Biz Stone to ReadWriteTalk earlier this month. Click here for the interview transcript). Certainly, APIs have helped twitter to reach every corner of web in form of various interesting applications, and the list is increasing day by day rapidly. No wonder, it has become developer’s darling.

Twitter API scores for its simplicity, ease of use, diversity, multiple data formats, and above all simple REST architecture. Since twitter was much simpler service with its veins powered by text messages, the existing RSS feeds laid a good foundation for first API set. The API is as simple as firing HTTP requests and receiving responses in any of the four formats, plain old XML, JSON, RSS and Atom. Out of these formats, RSS and Atom are most easy to use as they already have matured parsers on almost every platform being used today. But since RSS and Atom have a predefined DTD, it has a limited footprint and does not capture much information. XML and JSON provide a better and complete view, but you need to take help of third party abstractions in order to use them efficiently. For example, twitter request for obtaining public timeline in XML is http://twitter.com/statuses/public_timeline.xml. A typical response looks like this

<status>
<created_at>Thu Sep 27 18:42:18 +0000 2007</created_at>
<id>297192592</id>
<text>i just want to know what they are doing at all times</text>
<source>im</source>
<truncated>false</truncated>
<user>
<id>8697452</id>
<name>bluecat34</name>
<screen_name>bluecat34</screen_name>
<location />
<description/> <profile_image_url>http://s3.amazonaws.com/twitter_production/profile_images/ 24378882/ mommy_marissa_normal.jpg</profile_image_url>
<url />
<protected>false</protected>
</user>
</status>

The same response in RSS, ATOM and JSON is like

RSS

<item>
<title>cc_chapman: scheduled to slide into PME around 5 – can’t wait</title>
<description>cc_chapman: scheduled to slide into PME around 5 – can’t wait</description>
<pubDate>Thu, 27 Sep 2007 18:52:26 +0000</pubDate>
 <guid>http://twitter.com/cc_chapman/statuses/297208942</guid>
 <link>http://twitter.com/cc_chapman/statuses/297208942</link>
</item>

ATOM

<entry>
   <title>petestone: longest day ever</title>
   <content type=”html”>petestone: longest day ever</content>
 <id>tag:twitter.com,2007-09-27T18:53:17+00:00:http://twitter.com/petestone/statuses/297210002</id>
   <published>2007-09-27T18:53:17+00:00</published>
   <updated>2007-09-27T18:53:17+00:00</updated>   <linktype=”text/html” href=”http://twitter.com/petestone/statuses/297210002” rel=”alternate”/>
</entry>

JSON

{“user”:{“name”:”Jessica”,”profile_image_url”:”http:\/\/s3.amazonaws.com\/twitter_production\/profile_images\/23719302\/JeffDIcon_normal.jpg”,”description”:”",”location”:”",”url”:null,”screen_name”: “MissJess”,”id”:8610542,”protected”:false},”truncated”:false,”text”:”@limespinkles: good luck, Bestie!!”,”created_at”:”Thu Sep 27 18:54:20 +0000 2007″,”source”:”web”,”id”:297211822}

You must have realized that information conveyed by different formats varies. JSON and XML give additional info, whereas RSS and ATOM just provides the messages. Now it is upto you on which format to use. My recommendation for browser based apps is JSON, as it nicely works with javascript, and XML, in case you wish to use it on server side. RSS and ATOM are not information rich, so should be use only when your requirements are fulfilled with mere message and timestamp.

REST powered API is easy to understand and all you need is a browser to play around with it. Although Twitter seems curl territory, but i feel your browser is a good place to begin with. The APIs are nicely documented on twitter wiki and twitter fan wiki. In case you are a “flashy” developer, then start using their actionscript libraries as mentioned on twitter blog. In case you speak Java, Python, .NET, Runy on Rails, php or perl, and find HTTP/XML/JSON/ATOM/RSS too basic/cryptic/unfriendly/, then there is enough juice for you to get started with twitter APIs here.

There are lots of “Powered by Twitter” apps. TwitterGram, and TwitterEarth  are significantly better ones. The most recent interesting app is twiddeo. You can feel twitter power when you go through a really loooong list of twitter apps on twitter fan wiki. No wonder how 10x traffic emerged.

Twitter is a slowly maturing into a robust communication platform. It provides a scalable message delivery system which can be utilized in number of ways other than sending statuses. Many apps utilize twitter for email/blog notifications(EmailTwitter), website monitoring(MoniTwitter), server monitoring(ServerMojo), news delivery (BBC News, Manchester United News). Some pundits even talk of twitter as a publishing platform.

Twitter API is a useful tool to have. It has built a mature and versatile message delivery network over a period of time, which can be utilized nicely provided it is integrated with other useful services. It can deliver weather forecasts, stock quotes, new arrivals in a store, public address messages, school notices etc. Time will tell where it will head, but this is for sure that it will continue to power better and useful apps each day.

Each new day is a dawn and dusk for umpteen startups. If you are are regular on blogosphere feeds, you’ll simply bypass subject lines like, “We have launched a new social network for ___ (fill any random keyword)”, “we have launched a new search engine that searches ______(again a random keyword)”. Most of these are passed without any attention. A few are interesting to pay  a onetime visit, a very few more than once. Many of the new startups spend a considerable amount of time in reinventing the wheels, and then they start focusing on the real thing.

Lets take an example of a search engine. Each search engine does many common things. They crawl web, invest efforts in stripping html tags from webpages, and then index these pages. After spending considerable efforts in this process, they start building “the” thing. And these steps are mandatory for any search engine. Each one of these employ some of the smartest engineers on this planet. Individually, first few steps in building a search engines are same, and they all end up repeating it again and again. They pickup some open source crawler, and tweak it heavily to make it production ready, spend vast time crawling the web, monitoring their crawler and making it sure that it is running successfully. Compare it with a scenario, where a startup simply just crawls and crawls and crawls, and then builds a platform around it, and opens the API’s for other folks. Now you have arguably, a good crawl(assuming their smart engineers did a good job). Over time, they can even open the interfaces to improve their crawl by implementing submission model. They concentrate on one thing, and make it better. Monetizing their crawls is not a big deal if they can prove that they are crawling too much and too often. Zillion entrepreneurs will be interested in paying them moolah for API’s to access their crawls. Adding icing on the cake, if they provide html-less pages, it increases their value manifold. A good number can use their APIs to build nice apps with great ease. It serves a two fold optimization. First, it helps in concentrating on real problem, and secondly, IT HELPS IN CONCENTRATING ON REAL PROBLEM RATHER THAN REINVENTING THE WHEEL.  It lowers the entry barrier for a new idea that has dependency on crawls, and can make many ideas feasible which currently are due to these gigantic dependencies. Such analogy can be applied to various other aspects of any startup theme.

Established search engines have built much intelligence over time. As goes an adage in hindi which means “knowledge increases as you share it”. The quality of APIs from these bigshots is still not appreciable as far as knowledge delivered through their API’s is concerned. They sit on a big knowledge bank, which has potential to transform the human race, if they open it up.

Take another example of mobile devices. People are still searching for holy grail of mobile applications. Starting from scratch, you get a nice idea. You work overnight pondering over it. Finally, you choose J2ME, and develop it. You show it to your friend, and he confirms it being the next big thing. Then you decide to launch it. But wait! You suddenly realize that it does not work on nokia, sony ericsson, and blackberry. Then you take help of a porting company and get it ported over to many platforms. Porting guys take a reference build and deliver zillion builds you need to launch your app. Then it does not work on BREW phones. etc. And this process takes a cycle for every minor change and is a management disaster. This is true for “EVERY” mobile app. There have been enterprenuers who have cashed in on porting automation business. There is a better way to solve this problem. You take pains in developing a platform, say an appmanager. Your appmanager abstracts basic functions of the underlying mobile device. Through porting you make your appmanager visible on all. Then you expose appmanager API’s to developers. Since it has lowered the entry barrier, people can develop better mobile apps for your platforms real fast. You provide them a reach on most mobile devices. The moment I upload an application on your platform, all your users are notified about it. Certainly you will see more flux than earlier. Sun did a great job with publishing J2ME JSR(Java Specification Report). But the problem with JSR’s and RFC’s is that they are mere recommendations. While adopting them, vendors weak them according to their needs, and thus create a big porting problem altogether.

Facebook realized this are doing great with apps, since they launched their platform. In fact, facebook is a platform within a platform. A web browser is perhaps the most successful example of Apilatform philosophy. Imagine if a browser were not there, and you had to implement a mail client like gmail. You would have to make a separately app for windows, linux, mac, unix . Within vista, you would have to port it to vista, 2k, xp. Service packs make it worse. Firefox makes all this transparent to you. You can implement gmail on firefox, without worrying about diversity factor.

Where does facebook fit in? Facebook’s platform is second generation platform. You can term it as a semantic platform. Browser was to make things work, facebook is to make things better by injecting intelligence. They have a strong user base. Their platform gives me a two fold advantage. I can concentrate on what they call as “business logic”, and secondly, instantly i have a good crowd to attract. Take it as performing in a stadium full of folks, versus performing on a road cross, hoping people will pay attention. Again, since facebook’s platform is pretty restricted, so the use case depends on idea to idea about how feasible it is. i hope they open it further some day.

There is tonnes of intelligence within walled gardens of big corporations. Sharing it would help in constructing a better and usable internet. A person might not be good in creating text mining algorithms, but is an usability expert and might come with a better way to view google news. Open platforms will help in preventing people reinventing the wheel and concentrating on better apps, improving app quality, and reducing the effort to enter the market.