My Crowdsourced Follow-Up About Crowdsourcing

Slashdot regular contributor Bennett Haselton writes "In my last article, I proposed an algorithm that Facebook could use to handle abuse complaints, which would make it difficult for co-ordinated mobs to get unpopular content removed by filing complaints all at once. I offered a total of $100 for the best reader suggestions on how to improve the idea, or why they thought it wouldn't work. Read their suggestions and decide what value I got for my infotainment dollar."

In my last article, I proposed an algorithm that Facebook could use to handle abuse complaints, which would scale to a large number of users while also making it difficult for co-ordinated mobs to get unpopular content removed by filing complaints all at once. I offered a total of $100 to readers sending in the best suggestions for improvements, or alternative algorithms, or fatal flaws in the whole idea that would require starting from scratch. As the suggestions were coming in, Facebook obligingly kept the issue in the news by removing a photo of two men kissing from a user's profile, sending a form letter to the user that they had violated Facebook's prohibition on "nudity, or any kind of graphic or sexually suggestive content". (It would be a cheap shot to say that a photo of a man and a woman kissing probably would not have been removed; in truth, probably just about anything will get removed from Facebook automatically if enough users file complaints against it, which is the problem for unpopular but legal content.)

How would these complaints have been handled under my proposed algorithm? The gist of my idea was that any users could sign up to be voluntary reviewers of "abuse complaints" filed against public content on Facebook. Once Facebook had built up a roster of tens of thousands of reviewers, new abuse complaints would be handled as follows. When a complaint (or some threshold of complaints) is filed against a piece of content, a random group of, say, 100 users could be selected from the entire population of eligible reviewers, and Facebook would send them a request to "vote" on whether that content violated the Terms of Service. If the number of "Yes" votes exceeded some threshold, the content would be removed (or at least, put in a high-priority queue for a Facebook employee to determine if the content really did warrant removal). The main benefit of this algorithm is that would be much harder for co-ordinated mobs to "game the system", because in order to swing the vote, they would have to comprise a significant fraction of the 100 randomly selected reviewers, and to achieve that, the mob members would have to comprise a significant fraction of the entire reviewer population. This would be prohibitively difficult if hundreds of thousands of users signed up as content reviewers.

All of the emails I received -- not just "almost" all of them, but really all of them -- contained some insightful suggestions worth mentioning, although there was some duplication between the ideas. If you didn't see the last article, you might consider it worth while to stop reading before proceeding further, and mull over the description of the algorithm above to see how you would improve it. Then read the suggestions that came in to see how well your ideas matched up with the submissions I received.

The upshot is that nobody found what I believed to be completely fatal flaws, although one reader brought something to my attention that might cause trouble for the algorithm after a few more years. Beyond that, reader suggestions could be divided essentially into two categories. The first category of suggestions related to ensuring that the basic premise would actually work -- that the votes cast by a random sample would be representative of general user opinion, and could not be gamed by a coordinated mob or a very resourceful cabal trying to game the system. The second category of suggestions started by assuming that the voting system would work, and suggested other features that could be added to the algorithm -- or, in one case, an entire alternative algorithm to replace it.

To begin with the attacks and counter-attacks against the basic voting algorithm. Walter Freeman and Haydn Huntley independently suggested monitoring for users who vote in a small minority in a significant portion of vote-offs, and reducing their influence in future votes (by either not inviting them to vote on future juries, or sending them the future invites but then ignoring their votes anyway). The assumption is that if a user is frequently among the 10% who vote "Yes [this is abuse]" when the other 90% of respondents are voting "No [this is not abuse]", or vice versa, then that user is voting randomly, or their point of view is so skewed that their votes could safely be ignored even if they are sincere. I like the idea of eliminating deadweight voters, but this might also incentivize voters to vote the way they think the crowd would vote, instead of voting their true opinions -- for example, if they were called to vote on an anti-Obama page that showed Obama wearing a Hitler mustache. Some people's knee-jerk reaction would be to call the page "racist" or "hate speech" or "a threat of violence", even though comparing Obama to Hitler is not, strictly speaking, any of those things. If I were voting my honest opinion, I would count that page as "not abuse". But if I knew that I were voting along with dozens of other people, and my future voting rights might be revoked if I didn't vote with the majority, I might be tempted to vote "abuse".

Similarly, Walter Freeman and reader "mjrosenbaum" both suggested setting deliberate traps for deadweight users, by creating artificial cases where the answer was pre-determined to be obviously yes or obviously no, calling for votes, and revoking privileges for users who gave the wrong answer. This would eliminate the problem of borderline cases like the one above, where smart users think, "I suspect the majority will give the wrong answer, so I'm just going to go with the crowd, to keep my voting rights." On the other hand, it's more labor for Facebook to create the cases, and any public content authored by them -- especially content that is deliberately crafted to be "questionable" -- would probably have to run a gauntlet of being reviewed by lawyers and PR mavens before being released. My suggestion would be to use these artificial scenarios periodically to make sure that the system is working (i.e. that juries are giving the right answers), but it would be too inefficient to use it to try and weed out problem voters.

In fact, these and several other suggestions fell into a category of ideas that could possibly improve the efficiency of the algorithm by reducing voter shenanigans (where "efficient" means that fewer users have to be invited to each vote-off in order to get statistically valid results), but might not be worth the effort. As long as most of the votes cast by users are sane and sincere, all you have to do is invite enough voters to a vote-off, and the majority will still get the correct answer most of the time, even if you have problem voters in the system. That's the simplest possible algorithm. The more complicated an algorithm you come up with, the more likely that Facebook (or any other site you recommended this to) would just throw up their hands and say, "Sounds too hard", and leave the idea dead in the water. That's why I like the algorithm as lean and tight as possible.

So it's not quite like designing an algorithm for your own use, where you could feel free to introduce more complications as long as you're responsible for keeping track of them. In recommending an algorithm for widespread adoption, the basic form of the algorithm should be as simple as possible. In the case of the voting algorithm some interesting wrinkles may come up if you don't eliminate problem voters, but this is not fatal to the idea as long as it's still true that, given a large enough random sample of voters, the majority will tend to vote the correct answer.

For example, James Renken pointed out that as voters dropped out due to boredom, the remaining users casting votes would tend to be either (1) weirdos who just wanted to view questionable material; and (2) prudes bent on removing as much material from Facebook as possible. But that's OK, as long as those two groups vote sanely enough (or as long as there are enough sane users outside those two groups) that material which does violate the TOS, tends to get more "Yes [this is abuse]" votes than material that doesn't. Then all you have to do is make the jury size large enough to make a statistically significant distinction between those two cases.

Similarly, Joshua Megerman suggested surveying users for their religious, political, and other beliefs when they sign up as volunteer reviewers (they could of course decline the survey). This makes it possible, insofar as people answer truthfully, to make sure that a jury is composed of a group with diverse belief sets. (On the other hand, users could game the system by reporting beliefs that are the opposite of what they truly feel. For example, if you're a leftist, register as a right-winger. Then when an abuse case comes before you, if it's a piece of content more offensive to leftists, then the real leftists on the jury will tend to vote against it -- but as a registered right-winger, you'll be able to cast a vote against it as well, and you'll be displacing a real right-wing voter who probably wouldn't have voted that way, so your vote will be worth more!) Again, it's fine if Facebook wants to do this, but even without collecting this data and simply selecting jurors at random, it should still be true that genuinely abusive pages get more "Yes" votes in a jury vote, than non-abusive pages.

Lastly in the "keep the jurors honest" category, Paul Ellsworth suggested allowing jurors to anonymously review each other -- when a given juror is chosen for the "hot seat" (perhaps randomly, perhaps as a result of a history of skewed voting), other jurors are randomly selected from the voting pool, to review that juror's voting record and decide whether that juror has been voting honestly and judiciously, or not. When I first read this idea, I instinctively thought that because a contaminated jury pool would be reviewing itself, it would not be able to reduce the percentage of problem voters, but a little more thought revealed that this isn't true. Suppose initially your jury pool consists of 80% "honest voters" and 20% "dishonest voters", that honest voters who review the voting record of another voter will always vote correctly whether that person is "honest" or "dishonest", and that dishonest voters will always vote incorrectly. It's still the case that when a voter's record is reviewed by a panel of, say, 20 other voters, virtually 100% of the time the majority will get the right answer. If you strip voting rights from a voter whenever a jury of other voters determines them to be a "dishonest voter", then over time, the percentage of honest voters in the system will creep from 80% to 100%. So again, this might work, and again, it might just be adding unnecessary complexity if the basic algorithm could work without it.

Note that none of these precautions would address the case of a "sleeper" voter -- a voter who joins the system with the sole intention of voting incorrectly on particular types of cases (perhaps planning on voting "yes" to shut down pages made by a particular organization, or pages advocating a particular view on a single issue), while still planning to vote correctly on everything else. By voting honestly in all other cases, they prevent themselves from being flagged by the system for casting too many minority votes, or from being blacklisted by other jurors for having a questionable overall voting record. The only real way I can see to address this problem is to hope that such users are outnumbered by the honest users in the system, and that juries are large enough that the chances of "rogue voters" gaining a majority on any one jury are nearly zero.

Which brings us to the one potentially fatal weakness in the system that I'm aware of: reader George Lawton referred me to a program run by the U.S. government to create armies of fake accounts to infiltrate social media, named, apparently without irony, Earnest Voice:

The project aims to enable military personnel to control multiple 'sock puppets' located at a range of geographically diverse IP addresses, with the aim of spreading pro-US propaganda.

An entity with the resources of the U.S. military could potentially create enough remote-controlled voters to overwhelm the system. I'm not sure if there is a way to deal with a system if the majority of voters are compromised. Presumably by making all decisions appealable to a core group of trusted Facebook employees at the top (although this then creates a bottleneck and limits scalability, especially if filing an appeal is free and all the parties who lose abuse cases are constantly filing appeals to the next level up).

Now. On to the second category of suggestions: Assuming the majority of voters are honest, what other features would be desirable to build into the system?

Walter Freeman, on the subject of filing appeals, suggested putting appealed pages in a special queue where they could be publicly viewed and users could comment on the ongoing appeals process, in addition to reading arguments posted by either side; this also negates the censorship itself due to the to the Streisand effect. I agree, but it's not obvious why this is a desirable feature. This does create perverse incentives, since some users could get extra traffic for their content by creating a page that makes whatever argument you're trying to promote, spiking it with some TOS-violating content, waiting for the page to get shut down, appealing the decision, and enjoying all the extra Streisand attention that it gets while on public display during the "appeal".

Meanwhile, James Renken pointed out that the system would work best for content that was originally public anyway, like a controversial Facebook page or event. If someone filed a complaint regarding a private message that they received, and they wanted a "jury vote" about whether the content of the message constituted abuse, then either the sender or the recipient would have to waive their right to privacy regarding the message before it could be shared with jurors. If the message really was abusive, then in some cases the recipient might waive their privacy rights -- reasoning that they didn't mind sharing the nasty message that someone sent them, in order to get the sender's account penalized. The problem arises if the message also contains sensitive personal facts about the recipient, which they wouldn't want to share with anonymous jurors. The system could allow them to black out any personal information before submitting the message for review, but that creates a recursive problem of abuse within the abuse system -- how do you know that someone didn't alter the content (and thus the offensiveness) of the message through their selective blacking-out? So it's not obvious whether this idea could be applied to non-public content at all.

Reader George Lawton suggested allowing content reviewers to vote on the funniest or weirdest content they had to review, to be posted in a public "Hall of Infamy". I love the thought of this, but I think Facebook's lawyers would be uncomfortable glamorizing anything questionable even if it were ultimately voted to be non-abusive (and certainly if it was voted to be abusive). Besides, this also has the perverse-incentives problem -- tie your message to something that you know will not only get an abuse complaint, but will hopefully end up in the Hall of Weird. (Even without the abuse jury system, there are already plenty of incentives for people to make a political point and hope that it will go viral.)

David Piepgrass suggested that new content reviewers should be allowed to specify certain types of content that they don't want to be asked to review -- nudity, graphic violence, etc. This sounds like a good idea. He adds that users probably shouldn't be able to opt-in only to review certain categories of content (or jurors might sign up only to review nudity, and then who would be left to review the death threats?).

Finally, in the other corner: Jerome Shaver suggested bypassing the jury voting system altogether and working on a heuristic algorithm to determine when abuse reports were being submitted by organized mobs of users, based on the patterns shown by mutual friendships between the users filing the abuse reports. The difficulties in designing such an algorithm, are too complicated to summarize quickly, and could fill an entire separate article. (Convince yourself that it's not an easy problem to solve. You can't just ignore abuse complaints from clusters of users that have many mutual friendships, because it can happen that real tight-knit communities of users might file abuse complaints against a piece of content, where the complaints are actually genuine.) But again, there is the problem that if a proposed solution is too complicated or too nebulous, Facebook has the excuse that they are "weighing several options", that they're "already working on something similar internally", etc. The jury vote system has the advantage that it can be described in just a few sentences, and the general public always knows whether it has been implemented or not -- which means that as long as abuses of the complaint system continue, people can ask, "Why doesn't Facebook try this?"

You'll notice this is just a laundry list of the ideas I received, without any definitive conclusions about which ones are good or bad, but that's all I was going for. The original algorithm, I could argue with the force of mathematical proof that, under certain reasonable assumptions, it would work. There's no such proof or disproof for any of the suggested modifications, so I don't feel as strongly about any of them. But at the top of the article I suggested for readers to stop reading and see how many of these ideas they could come up with on their own. How did you do?

The final honor roll of readers who were each the first, or only, person to submit an original idea: Walter Freeman (bonus points for getting in several good ones), James Renken, Joshua Megerman, Paul Ellsworth, George Lawton, Jerome Shaver, and David Piepgrass. Most of them volunteered to donate their winnings to charity, and agreed to let me donate their share to Vittana, which arranges microloans to college students in developing countries. One preferred a charity of their choosing, and only one actually kept the money. To be clear, for future contests, it's awesome if you want to donate the money to charity, but it's not dickish to keep it. That was the original deal after all.

So, all very clever and interesting suggestions, some of which might inspire readers to keep coming up with their own further variations. I said which ideas I probably would have incorporated and which ones I wouldn't, and I'm sure many of you would tell me that I'm wrong on some of those points. Although from here on out you're doing it for free.