Facebook Crawler Speaks Back

Last week we ran a story about Facebook suing to get a crawled dataset offline. This week we have a bit of a response written by Pete Warden, the guy who actually did the crawling. He followed robots.txt, and then Facebook's lawyers went after him. It's actually a quite interesting little tale and worth your time.

Pretty naive

[elrous0]

Did this guy really think he could just give away the data that Facebook sells (or intends to sell) to third parties and NOT have them sue him for it? It's no secret that the business model of most of the social sites and big search engines factor in the massive amounts of data they collect on users as a major corporate asset, to be used internally for data mining and also sold (supposedly after being anonymized) to advertisers and other third parties. It takes a babe in the woods to think he can just waltz in and take that away with a "But your robot.txt didn't say I *couldn't* do it" defense, without expecting a big legal fight.

Is the guy in the right? Probably. Would he have a case? Probably. Does either of those facts matter if he doesn't have the big $ needed to hire lawyers and fight through several courts? Nope.

Re:Pretty naive

[Anonymous Coward]

If he's in his right, but not having as much money as a big cooperation means he'll lose anyway, then your U.S. court system is broken. Please fix it.

Arachnophobia

[mfh]

I might be alone here but spiders revolt me to a point where I simply respect them and leave them alone.

But that said, Google operates a spider, pretty much. So we have to look at any potential spider on the internet like we might look at Google. If he followed the Robots.txt as Facebook set it up and he didn't try to misunderstand it, then there isn't anything they can do. Although, I'm pretty sure the Facebook EULA says you can't spider them so he's SOL anyway if that's the case. This should be a long and drawn out case unless there is a settlement.

Facebook is ripe. People put up EVERYTHING about themselves on there. I never accept a friend request unless I know the person and I offer a challenge question often. If it's not responded to adequately, I simply ignore them. But in the end there isn't much you can do. If you put it on Facebook -- consider it public, like if it was in the phone book.

obviously this is abusive

[circletimessquare]

this is what the guy should do:

1. engage the lawsuit

the downside is financial exposure. so incorporate your work in such a way that it can't hit your personal finances. the upside is massive exposure. you will achieve some level of fame: the guy who finally gave the robots.txt convention a legal status quo. this will help you professionally, as well as make your life story

2. whine to google

you are completely right that google shouldn't have to get permission every time it wants to crawl the site. therefore GET GOOGLE TO DEFEND YOU

Re:Ballsy.

[hansamurai]

Not really ballsy considering he didn't actually let Facebook's challenge of "The only legal way to access any web site with a crawler was to obtain prior written permission" go to court. Maybe he should have gone to the EFF for help as the repercussions of a judge actually deciding in Facebook's favor would have been devastating to the web.

Re:Pretty naive

[elrous0]

Sure, I'll just go to Congress and explain to them that they should pass a law that will be universally opposed by the corporations that give them millions in campaign contributions every year--because it's the right thing to do.

Re:Pretty naive

[elrous0]

The only way to "score a jackpot" in a case like this is to have it declared a civil rights case (meaning the losing party has to pay the lawyer's fees of the winner), and that doesn't seem very likely here.

Ooo, deja vu

[lxt]

It's sort of ironic that Facebook is trying to stop someone crawling public profiles on their site, because that's exactly what Mark Zuckerberg did while he was at Harvard (I was a grad student in the CS department at the time).

Pre-Facebook, Zuckerberg created a site that let Harvard students compare each other, a bit like Hot or Not. Obviously nobody was going to go to a site that wasn't populated with their classmates, so he basically crawled the websites of the various residential houses that put their students info online (but behind passwords and auth) and copied it into his own site.

He actually got into a fair bit of trouble for this, and ended up being sent to Harvard's ad-board for discipline (I think he got put on probation, but I'm not entirely sure).

The key difference here is that this guy actually did everything by the book and followed robots.txt, whereas Mark Zuckerberg didn't.

Re:obviously this is abusive

[elrous0]

Most lawyers work for money. It's nice to think that the little guy in the right can take on the big guy and wins in court. But real life isn't a movie. Most of the time the little guy fighting a case like this ends up broke, whether he wins or loses. It's also nice to think that he could just go to the EFF and get a lawyer for free, but something tells me it's not that simple (I suspect the EFF is already swamped with what few lawyers they have).

Re:Pretty naive

[elrous0]

A good start would be a "loser pays [pointoflaw.com]" system similar to what they have in much of Europe. It gives people who legitimately have a strong case a chance to find a lawyer, and discourages frivolous lawsuits and lawsuits aimed only at intimidation (so-called "SLAP" lawsuits).

Re:Pretty naive

[Lumpy]

WE have the best court system money can buy!

Re:Pretty naive

[Lumpy]

Nope I haven't gave up, I'm hoping for an uprising. problem is that 99.997% of all Americans are placated with their cable tv. Fat dumb and happy is the American way. Almost nobody here will even inconvenience themselves for "freedom" Then we have these "tea party" idiots. loudmouths simply looking for 10 minutes of fame who really have no desire to protect freedom.

Re:Pretty naive

[Lumpy]

It aint their data, it's the owners data. they are simply hijacking ownership.

Re:Pretty naive

[Archangel Michael]

Want to fix the ELECTION laws, while not breaking the First Amendment Rights to Free Speech? It is really quite simple. One simple rule.

Only People (persons, not legal entities)who are eligible to vote can donate to political campaigns.

This doesn't deny corporations from running ads, they just have to do it on their own, and out in the open where everyone can see who they are telling people to vote for. They have to buy their own ads to tell people to vote for Harry Reid or Mitch McConnell.

This also goes for Unions and all other organized groups. Make them buy their own ads for their own causes.

Simple rule, clear, concise, straightforward and solves all sort of problems with current campaign laws, without any bias towards or against anyone.

AND that is why it won't ever be implemented.

And I'm sure that there is someone that is going to be upset because their favorite group won't be able to donate money to a candidate/campaign while at the same time restricting anyone that might oppose them (it) from doing likewise at the same time.

Re:Pretty naive

[kalirion]

And then if your lawyer loses the case, you get to pay for the company's team of 20 $1000/hr lawyers?

Re:Pretty naive

[rilian4]

Interesting. You want people to fight for their freedoms but when someone does, all of a sudden they're a bunch of idiots. Not saying I agree w/ everything the Tea Party stands for but at least they're willing to to stand for it in public and fight for it. If they don't get loud, no one listens. Most of them probably have no desire for fame or fortune. They simply want their freedom. If you really want people to stand up for their rights, be prepared for the consequences.

Re:obviously this is abusive

[Nadaka]

Welcome to the new corporate feudal system where the top 2% of the people own half the resources and the bottom half of the people own 1% of the resources (resources includes the law in this case). Do not offend the the corporate liege lords, for they have unlimited legal irresponsibility and a virtually unlimited supply of lawyers and judges in their pocket.

Re:Pretty naive

[TheRaven64]

How does that stop a company paying a bonus to every employee that donates to a particular party, or simply giving its CEO a big bonus that he then (after tax) gives to a political party?

Re:Pretty naive

[vxice]

no, your solution is a band-aid that only covers up the real problem. Voters that are swayed by fancy campaign ads are the real problem. If everyone in this country took their civic duty seriously and researched from reliable sources while completely discounting campaign ads that would fix the real problem. Prohibiting corporations from making contributions is protecting the electorate from themselves since they can't seem to do it. And as long as the electorate doesn't have to it wont and will never learn to.

Re:Pretty naive

[mcgrew]

Only People (persons, not legal entities)who are eligible to vote can donate to political campaigns.

I'd go one farther -- you shouldn't be able to donate to any candidate you're not eligible to vote for. Bill Gates should not be able to donate to Pat Quinn's re-election campaign. Why should someone who doesn't have the right to vote for a candidate be able to buy his way into office?

I'd also add that you should not be able to "contribute" to more than one candidate in any given race, on the grounds that that's briberey plain and simple.

Where did the idea that money equals speech come from, anyway? Money is NOT speech. But like you say, good luck ever getting that implimented in our plutocratic pseudo-republic.

Re:Pretty naive

[mcgrew]

You can speak all you want, post blogs, bitch on slashdot, just not be able to donate. Donating is NOT speech and should not be covered under the 1st amendment.

Re:Pretty naive

[FlashBIOS]

[...] where were they when Bush and the Republican congress took Clinton's balanced budget and ran up the biggest deficit in history? [...] Where was their cry to vote against incumbents when the Republican held the majority and were running up the debt?

So because they are late you're going to discount their argument? Doing the right thing in spite of it of it being timely is still better than not doing the right thing because you think you may have taken too long.

Re:Pretty naive

[Blakey Rat]

The real problem is that people are being influenced by ads. You shouldn't base your vote on how noisy somebody is, that's terrible and that's what we should be focusing on.

In an ideal world, McDonalds spending tons of money on political ads would be useless because people wouldn't pay attention to them.