Vertical Search Engines and Copyright

An anonymous reader writes "I am a big fan of Oodle, the online classifieds aggregator. I was disheartened when Craigslist announced that they would block Oodle from their site in late 2005 (old link), as I find their service very handy. I came across this page at the site of an aggregator of freelance job openings that summarizes the arguments around the legality of meta search engines and mashup-like sites and I found myself wondering if Oodle could have avoided the ban. There is an interesting argument there that seems to undermine copyright claims of user-generated content compilations. Are mashups legal? How does this affect sites like Digg or YouTube?"

Content Aggregation and Mashups

[blaster151]

In content aggregation lies all of my excitement about the future of the web (if people are allowed to continue being innovative and aren't prevented by heel-dragging by legal departments).

I don't even care if the aggregation happens server-side or browser-side. I want to be able to view a book product page on Amazon and click a "place local library hold" button. I want to be able to view my LiveJournal Friends page and have a superimposed queue and "recently watched" displays for those folks who are also my Netflix friends. Or current weather reports for those friends' locations. Fun stuff. I want to be able to stumble across an old news story and have a "there are 117 comments when this story was posted to Slashdot five months ago" notification.

There is so much potential here for crossover - and it's all data that already exists! Crosslinking through simple knowledge of "which person on one service is which person on another service" - and "which product on one service is which product on another service" - would open so many doors. I hope legal departments don't keep preemptively closing them. To me, this is what would excite me if it were true about "Web 2.0" - beyond just simple pretty, AJAX-enabled user interfaces. Although those are cool, too.

mashup's

[jshriverWVU]

I've wondered how mashups would survive. Back in the 90's companies where suing others for just linking to their site. Let alone blatant copying of data feeds. It's a tricky situation.

If it's a site that is funded strictly from ads, then they have a lot to lose by others ripping their content. But at the same time mashups are a wonderful way of getting a lot of similar info together so it's a convenience to the end user.

I am making something similar

[Safiire Arrowny]

I am making something similar to create notifications for posts on craigslist right now. It is written in Ruby, and it basically enters the sections you specify on craigslist, and downloads and stores the last 100 postings into an Sqlite3 database.

Then, as a human might do if he were obsessive, checks the section indexes for updates say every 10 minutes and incrementally stores new posts.

The data in sqlite is then indexed by the ferret search engine library, so that it can perform searches on the post content and uses gtk2's libnotify to pop up a notification bubble if it has found anything you previously said you were interested in.

I have not gotten banned in any way from craigslist, and I don't expect to be, since beyond the initial download of the sections, it behaves no different than an obsessive human who might be looking at 10 pages every X minutes. With this, I would be necessarily one of the first people to notice anything on the site that I'm interested in.

I will probably release this on my site for everyone. I'm aware it's against the terms of service to completely mirror the entire site, but does this count as mirroring? Can it be deemed similar to greping your firefox cache, or personal mirroring and indexing?

I know I'm sure as hell going to use it, that's why I made it, but it is an awkward feeling that if I give it away for free and people liked it, that I could get into some kind of trouble.

First Hand Experience

[w0lver]

I have experience at two companies that did site aggregation. First, with a company that did travel deals but searching other sites and the next was a job site that did the same. Searching and presenting a summary with link to the real live content is legal. Taking the content and re-purposing even with credit is illegal. So as an example, with a travel sight, searching all the airlines, Expedia and so on, and displaying links with prices is valid. However, showing the flights and prices without links and then booking it in the background never displaying the site, illegal. We had a number of companies that tried to sue us, we send over legal opinions and case history on the topic, the suits would disappear. However, we did have a few sites that blacklisted our IPs, tried to break our scraper, and post nasty things about us on other sites.