Forgot your password?
typodupeerror
The Internet Networking IT

Handling Flash Crowds From Your Garage 74

Posted by kdawson
from the to-scale-or-not-to-scale dept.
slashdotmsiriv writes "This paper from Microsoft Research describes the issues and tradeoffs a typical garage innovator encounters when building low-cost, scalable Internet services. The paper is a more formal analysis of the problems encountered and solutions employed a few months back when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days. In addition, the article offers an overview of the current state of utility computing (S3, EC2, etc.) and of the most common strategies for building scalable Internet services."
This discussion has been archived. No new comments can be posted.

Handling Flash Crowds From Your Garage

Comments Filter:
  • by penguin king (673171) on Tuesday July 08, 2008 @02:56AM (#24096267)
    Here I was picturing a bunch of people showing up in your garage for seemingly no reason. Still interesting to see how they handled the massive increase!
    • Re: (Score:3, Funny)

      by RuBLed (995686)
      And here I was thinking that it was about handling a crowd flashing for you in front of your garage...
      • Re: (Score:2, Funny)

        by dredwerker (757816)
        I thought it was something to with actionscript.
      • Since I can't see any pix in the area near the bottom called "Figure: DNS servers fail over very quickly when an upstream server fails" - does that mean that the flash crowd called "SLASHDOT" has taken down this part of the article called "Handling Flash Crowds..." ?

        I mean sheesh! They even mention slashdot!

        • by damburger (981828)

          Surely nothing can trash their argument so much as an abject (and amusing) failure to put it into practice?

          How can we believe in their solutions for letting small-time web services handle usage spikes when their evidently can't?

          • by Schlage (195535)
            Considering that the paper detailing their argument, and the images/figures used therein, is hosted somewhere other than Microsoft's own website (at usenix.org, to be specific), I don't think its really accurate to say that someone else's failure to effectively handle traffic is a poor reflection on the effectiveness of their solution.
      • by houghi (78078) on Tuesday July 08, 2008 @04:42AM (#24096917)

        And here I was thinking that it was about handling a crowd flashing for you in front of your garage...

        You must be non-American, because in Soviet America you flash the crowd.

      • Re: (Score:2, Funny)

        by MJMullinII (1232636)

        Nah, that's easy.

        TRUST ME, they scatter pretty easy when wheel into your driveway at 40 mph.

        The few that remain can easily be cleaned up with a fire/water hose.

        • by wstfgl (912433)

          ... a fire/water hose.

          Your ideas are intriguing to me and I wish to subscribe to your newsletter.

    • by Divebus (860563)

      When I read "This paper from Microsoft Research..." I thought... oh yeah, they're in Cupertino.

    • Just turn off the local displacement booths when you reach a certain threshold. Make them walk a few blocks extra that's all.

      "...The next thing anyone knows, every man, woman and child in the country has decided that he wants to see the red tide at Hermosa Beach..."

      "Another flash crowd. It figures," said Jerryberry. "You can get a flash crowd anywhere there are displacement booths."

      From Flash Crowd, by Larry Niven.

  • Astro Turf (Score:1, Troll)

    by Frosty Piss (770223)
    Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin' ...
    • Re:Astro Turf (Score:5, Informative)

      by nmb3000 (741169) <nmb3000@that-google-mail-site.com> on Tuesday July 08, 2008 @03:32AM (#24096497) Homepage Journal

      Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin'...

      If you're going to troll, it might be a good idea RTFA beforehand so that you don't make a fool of yourself. Two examples:

      - The web service is implemented in Python and currently deployed on two virtual machines at Amazon EC2.
      - Like Asirra, we implemented Inkblot in Python.

      If they're astroturfing they aren't very good at it.

      The article has very little Microsoft-specific details in it. It's basically a short explanation of high-performance content delivery and a few stories about MS Research [microsoft.com] (link because they have some cool stuff) projects and how they fared with high load traffic surging (aka Slashdotting). They specifically mention getting Slashdotted several times, as well as surviving a DDoS.

      Overall I thought it was an interesting article. I didn't realize Amazon's S3 service was so inexpensive or available to "budget" sites.

      • by Maxmin (921568)

        Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin'...

        If you're going to troll, it might be a good idea RTFA beforehand so that you don't make a fool of yourself. Two examples:

        Erm, for the record they also cover applications created by MS Research, using MS technologies - e.g. one called MapCruncher, which runs/ran on IIS. See 5 Application Design and Flash Crowd Experiences [usenix.org]. Another examples include SQL Server, but the article's pretty much technology agnostic

    • by RulerOf (975607)

      Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot?

      If a sizable portion of your direct competition's most enlightened (using that term loosely) users all congregated in one place for you to spread propaganda on them.... Wouldn't you?

  • Google? (Score:1, Interesting)

    by Anonymous Coward

    This paper from Microsoft Research describes the issues and tradeoffs a typical garage innovator encounters when building low-cost, scalable Internet services.

    Anyone else initially think that Microsoft was talking about Google [wikipedia.org] after reading that first sentence?

  • Misleading pretense (Score:4, Interesting)

    by wombatmobile (623057) on Tuesday July 08, 2008 @03:31AM (#24096491)

    "Our innovator may get only one shot at widespread publicity. If and when that happens, tens of thousands of people will visit her site. But a flash crowd is notoriously fickle; "

    The "researchers" offer a strange view of how the market works. If the idea is good then surely the site will enjoy numerous opportunities for growth and referral every time a happy user recommends it to a friend. A good, innovative idea will not be sunk by one underprovisioned flash crowd.

    • Re: (Score:3, Insightful)

      by spazdor (902907)

      I think you overestimate the attention span of the type of people who compulsively install Facebook apps.

      • I think you overestimate the attention span of the type of people who compulsively install Facebook apps.

        Even if you would lose that first crowd, you might get other users at a later point.

        • by spazdor (902907) on Tuesday July 08, 2008 @05:11AM (#24097099)

          The first crowd is a different class of user from the general public. It's a small subset of the Facebook usership that forwards almost everything they receive to everyone they know. Pandering to that particular crowd is a Facebook developer's foremost goal, because they are the ones who will drive exponential growth, if it's going to happen at all.

          I think market research will show that this core group of irritating people are just as capricious with the "block app" button as with the "forward". So assuming your idea is good enough to spread among the primary group, your first chance is generally the only one you get.

          • Re: (Score:3, Funny)

            by argent (18001)

            It's a small subset of the Facebook usership that forwards almost everything they receive to everyone they know. Pandering to that particular crowd is a Facebook developer's foremost goal, because they are the ones who will drive exponential growth, if it's going to happen at all.

            So basically Facebook selects for applications that are attractive to the kind of people who forward spam.

            Thanks for the warning.

    • can trace their success to that one weekend or month or season where things really took off

      additionally, you misunderstand that the flash crowd is not something that comes and goes, but something that comes and stays

      but sure, you are correct: a good innovative idea will find a way regardless of inability to scale quickly. some other guy will make work what you can't. you could retard your growth for awhile while you tinker with how to scale. but if some other guy takes your good, innovative idea and runs with it further and faster than you do, you are doomed to obscurity while he reaps the benefits of your good idea

      so you shouldn't be giving advice on how the market works, because the fickleness you dismiss really is a big deal and is not to be taken lightly

    • "Our innovator may get only one shot at widespread publicity. If and when that happens, tens of thousands of people will visit her site. But a flash crowd is notoriously fickle; "

      The "researchers" offer a strange view of how the market works. If the idea is good then surely the site will enjoy numerous opportunities for growth and referral every time a happy user recommends it to a friend. A good, innovative idea will not be sunk by one underprovisioned flash crowd.

      I don't see anything in that first quote about the idea being good, only popular for a short period of time.

  • Brand protection: (Score:2, Interesting)

    by Geak (790376)
    FTA:

    including one that was (literally) Slashdotted

    Anybody here think slashdot should be protecting it's brand here? Isn't this similar to using google as a verb? I think this is the only place one should be allowed to use that term. Microsoft most definitely shouldn't be allowed.

    • by drinkypoo (153816)

      Anybody here think slashdot should be protecting it's brand here?

      Yes.

      Isn't this similar to using google as a verb?

      The canonical example is "Photoshop". It predates google considerably. HTH.

      I think this is the only place one should be allowed to use that term.

      I think you're ugly, and your mother dresses you funny.

      Microsoft most definitely shouldn't be allowed.

      Why not? We are free to refer to buying good software and ruining it, then forcing it on people through your monopoly position as "Microsofting" it.

      Intellectual property laws are not handed down from god (unless they're in lost commandments 11-15.)

      • Re: (Score:2, Informative)

        by Anonymous Coward

        The canonical example is "Photoshop". It predates google considerably. HTH.

        You seem to have misspelled Xerox.

        Kevin

  • Then you don't need as much brute force ?

     

  • Great. (Score:5, Funny)

    by ericvids (227598) on Tuesday July 08, 2008 @04:27AM (#24096825)

    A paper on how to avoid slashdotting, posted on slashdot. /me clicks obsessively on links

  • ... when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days.

    Of course, I'm completely guessing here, but they probably required you to invite 20 brazillion of your imaginary Facebook friends before you could install it.

    • ... when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days.

      Of course, I'm completely guessing here, but they probably required you to invite 20 brazillion of your imaginary Facebook friends before you could install it.

      How bad does an application have to be to be brought down by 20 Brazilians?

      • by blueZ3 (744446)

        20 Brazilians is not as bad as six Spaniards, a Frenchman, and two guys from Nigeria.

      • .... and if I had any Brazilian FB friends, I might have bothered to try the app out too :-)
  • by Anonymous Coward on Tuesday July 08, 2008 @04:59AM (#24097019)

    The problem with most sites is that they never expected the sudden burst in popularity, and having never bothered to test if their service was scalable, had to rush and fix it before people start noticing reliability problems. If you at least attempted to write a scalable app to start with, even if you don't have the details nailed down, you've taken a step in the right direction.

    Animoto looks like an interesting case because it's a really resource-hungry app that has to put together a video with effects and music. Most sites have trouble just serving up dynamically-updated text. All those EC2 instances and the high-bandwidth needed sounds like a lot of money. Scaling up a business plan is at least as difficult as scaling software.

  • by gsslay (807818) on Tuesday July 08, 2008 @05:41AM (#24097293)

    Could someone provide a translation of the summary for those of us who speak English rather than promotional BS? .. on second thoughts, never mind.

  • First they use the argument that any geek/nerd with a good idea can put them out of business in their anti-trust case (argument also made in Hackers 3: Antitrust (movie)) and now they try to discourage anyone trying to set up something in their garage (while covering their asses to make you believe they try to help these people by giving them 'good advice'). This is ofcourse is totally unrelated to the fact that MS is gearing towards web 3.0 (turn on sarcasm scanners)...
  • Because in English, all nouns will eventually be verbed, and vice versa. I'm sure there's some ivory tower dweller somewhere who can tell us what the first recorded example is, and that it probably wasn't in English. Xerox, however, has been superseded because only old people use photocopiers any more (except at the library sometimes. Most people don't bring in a scanner.)
  • by rts008 (812749)

    I handle flash crowds from my garage with pepper spray, and if that does not work, having a shotgun for escalation.*sarcasm*

    In this contest to be clever with the language, all I see is a bunch of idiots outsmarting themselves.

  • There's some irony in this appearing on slashdot today: it seems like all I can get out of reddit all morning is a "Service Unavailable" message :-)

The trouble with opportunity is that it always comes disguised as hard work. -- Herbert V. Prochnow

Working...