walmart pharmacy morton il passing urine drug test
Home + News + Speaking + Connect + About

Derivative businesses and scraping

December 31, 2007 @ 6:40am

Boing Boing led me to a Wired story called Should Web Giants Let Startups Use the Information They Have About You?

It started off with a story about a guy who was scraping Craigslist data who got a cease and desist letter. Even though he was formating their listings into a more usable format for his readers, he was showing Google ads with them and his spiders were slowing the Craiglist "page-loading times to a crawl". Whether that was accurate or not, I agree with Craigslist. The use was against their terms of service. At Weblogs, Inc. we gave out full blog posts for free in RSS to people who wanted them that way, but we didn't allow anyone to show them on another web site and run ads against them.

Then it touches on the ecosystem of giant companies who have data -- like maps, photos, headlines or stats -- letting smaller companies use the data to build new derivative services:

There's an awkward dance going on, an unregulated give-and-take of information for which the rules are still being worked out.

We talked about this kind of thing at AOL. When you're a little startup, the costs of the extra bandwidth and server resources needed to support massive syndication can put you out of business. But for Internet giants like AOL, Google and Yahoo providing a service like that is effectively free.

The most interesting part of the article to me is the focus on companies that build their businesses on top of someone else's data or someone else's platform -- like all of the companies who exist solely to make Facebook apps. That whole trend blows my mind. I can still remember when the Mac OS didn't have a clock in the menu bar. There was a guy who wrote a shareware app which added one. A few versions of the Mac OS later, Apple added their own and made his little extension -- and his shareware revenue stream -- history.

To me, every single Facebook app development company is the web 2.0 equivalent of that old Mac shareware menu clock.

The article is probably sticking to my brain because I'm looking at being a provider of comic book data, feeds and APIs and imagining what derivative businesses -- and headaches like bot warfare and cease and desist letters -- are going to come along with that role.

Newer: Banning live-blogging from sports? Good luck.

Older: Putting Netscape out of its misery