Forgot your password?
typodupeerror
Cloud Businesses The Internet

AWS Load Balancer Sends 2 Million Netflix API Reqs To Wrong Customer 58

Posted by Soulskill
from the close-does-not-in-fact-count dept.
rsk writes "Amazon Web Services' Elastic Load Balancer is a dynamic load-balancer managed by Amazon. Load balancers regularly swapped around with each other which can lead to surprising results; like getting millions of requests meant for a different AWS customer. Using ELBs can result in AWS unintentionally introducing a man-in-the-middle (attack) into your application environment. Most AWS users do not realize this can happen and have not secured against it."
This discussion has been archived. No new comments can be posted.

AWS Load Balancer Sends 2 Million Netflix API Reqs To Wrong Customer

Comments Filter:
  • It looks more like some client aren't respecting the DNS TTL value, so technically it's not Amazon's fault. You should stick to standards, and if TTL says it's 60 seconds, then it is.
    • Browsers are sometimes forced to disregard TTL values to prevent certain type of attacks which involve quickly changing DNS records.

      • by mysidia (191772) *

        Browsers are sometimes forced to disregard TTL values to prevent certain type of attacks which involve quickly changing DNS records.

        No, they are not "forced" to do so. They have chosen an improper method to "workaround" a security issue that violates other internet standards and causes issues, because they are not implementing DNS resolution in a valid way.

        The TTL in DNS is not an "advisory" value, it is a time after which the old RR in the previous authoritative DNS response must be expunged, a TTL o

        • by DarkOx (621550)

          The browser makers playing fast a lose with standards, outside of html sucks! They all suck, try an find a browser that does PASV ftp *correctly*. They all either as part of a very misguided security attempt or based on the assumption FTP servers are behind NAT and can't be configured to send a correct address in the PASV response don't use the address value returned and stupidly use control sockets remote address as the address.

          That breaks all but the very most common use case and all the browsers do it.

        • If browsers don't impose such a minimum, devices with embedded web servers (think printers and home routers) become vulnerable to Cross-Site Request Forgery. They can potentially defend against this by checking the Host header on requests, but since these devices are only manageable through the web there's no good way to establish what the correct value is.
      • by Anonymous Coward

        Wait a minute. I'm a manager, and I've been reading a lot of case studies and watching a lot of webcasts about The Cloud. Based on all of this glorious marketing literature, I, as a manager, have absolutely no reason to doubt the safety of any data put in The Cloud.

        The case studies all use words like "secure", "MD5", "RSS feeds" and "encryption" to describe the security of The Cloud. I don't know about you, but that sounds damn secure to me! Some Clouds even use SSL and HTTP. That's rock solid in my book.

        An

        • ... and here I am without any mod points.

          Pretend that I marked you Two Thumbs Way Up!, Mr. PHB.

          PS: For those of you without an irony chip installed ... pretend I started my post with </irony>

        • by ATMAvatar (648864)

          A single tear rolled down my cheek as I compared this satire against real, starry-eyed reactions of my company's management with "the cloud".

          You know, this mythical beast that solves all scalability and maintenance issues while simultaneously having absolutely zero downsides...

        • cool story, bro [google.com] - but maybe it was submitted once and some faulty load balancer spread it out.

    • It looks more like some client aren't respecting the DNS TTL value, so technically it's not Amazon's fault.

      "Technically", no. But two people pointing a finger at each other and saying "He did it!" doesn't solve anything, and all the customer gets is the finger.

      • It looks more like some client aren't respecting the DNS TTL value, so technically it's not Amazon's fault.

        "Technically", no. But two people pointing a finger at each other and saying "He did it!" doesn't solve anything, and all the customer gets is the finger.

        Thus Elastic Load Balancer's other name, Erratic Load Balancer.

        • by hedwards (940851)

          If the customer's getting the finger, wouldn't that make it more of an Erotic Load Balancer?

      • by mysidia (191772) *

        Pointing the figure and screaming very loudly would be very good, especially if Amazon does it, as it will help bring attention to broken behavior in DNS and browser software.

        I will agree it hurts Amazon, but it helps the community, for large players like Amazon to help bring attention to broken software, so that it can be fixed.

    • Re:TTL value (Score:5, Interesting)

      by JWSmythe (446288) <jwsmytheNO@SPAMjwsmythe.com> on Saturday October 29, 2011 @02:45PM (#37880668) Homepage Journal

          From what I've seen, it's frequently the client's DNS servers, not the client itself.

          I've used a short TTL (5m) for quite a while. It's intentional, because I've needed to switch things rather quickly in the past, and it's better for it to "just work", rather than waiting hours for everyone to pick up the change.

          I used to work for a place that had a huge traffic load. Our slow days were still millions of unique visitors. When we took a machine out of DNS (DNS round robin between 15+ machines), we'd see the traffic drop significantly in the first 5 minutes. When AOL finally saw our change, it would drop more. There would still be lingering people for about an hour, and then it would finally be idle.

          That was a pretty regular thing for us to do. We scaled our traffic to our various datacenters this way. We'd also load test lines and individual servers with it. If it looked like we were running into a bandwidth limitation, I'd throw a few hundred Mb/s down the line, and see how it performed. If it really was, we'd then switch everything away from it to other datacenters until the provider fixed it.

          In all those circumstances, in 5 minutes most (but not all) of the traffic moved. An hour from the change, the remainder had moved.

          I've seen this with my home provider. I let them handle DNS for my home machine, rather than doing it myself. I've made changes, and they don't respect it within 30 minutes. Within about an hour, the new DNS records show properly.

          Google's public DNS servers seem to do pretty well in that respect. Our changes are reflected properly there in just a few minutes. AOL, TimeWarner/RoadRunner, and a few others are pretty bad. I know why they do it (reducing load on their DNS servers), but it becomes a pain in the ass for places that need to make changes quickly.
         

      • by Kattare (528707)
        Problem with any of these scenarios is that according to the AWS forum post, he's been getting rogue Netflix traffic for 4 days. No dns server or mainstream client is going to keep a 60 second TTL record for 4 days. It's either an issue at AWS completely unrelated to DNS, or an issue in Netflix clients. With it being in TV's, BluRay players, Xboxes, IOS, Wii's, etc... who knows what client the issue could be in... I wonder if the forum poster could capture the browser string and help debug?
        • by bsane (148894)

          There are some clients that cache dns records until they're restarted. I've removed internet facing vips from dns and weeks later there are still 100+ clients making connections, the only thing that would stop them is a client restart.

  • Why doesn't Amazon use a reverse proxy which performs additional checks and routes the requests to the right customer? (With Server Name Indication, that would work for TLS, too.) Without that, it's simply not possible to switch IP addresses quickly between non-cooperating targets.

    • Because Elastic Load Balancer isn't just for HTTP traffic, you can use it with any kind of traffic.
      • Re: (Score:2, Insightful)

        by TooMuchToDo (882796)

        On top of that, their "Elastic Load Balancer" (just another bullshit "cloud" marketing term for their cluster of F5 load balancers at each availability zone) is just, as I mentioned, an array of F5 load balancers. They either a) don't support the functionality OP is speaking about, or, more likely, Amazon chooses not to support handling traffic in that way to simply operations.

  • 1. Write a load balancer
    2. Sell it to customers until it breaks
    3. Patent software anomaly
    ...
    Profit!
    • Re: (Score:2, Informative)

      by TooMuchToDo (882796)

      Actually, they didn't write the load balancer. They just bought F5s and integrated them with their infrastructure to change their configurations programmatically.

      • by Kalriath (849904)

        If they were F5s, they'd actually work. We use F5 here, and from looking at the config, Amazon would have to be literally incompetent to get such basic functionality wrong.

  • Use rewrite rules to do a 301 redirect to goatse.cx when the host is api.netflix.com!
    • by mysidia (191772) *

      Use rewrite rules to do a 301 redirect to goatse.cx when the host is api.netflix.com!

      Why do that when the person to erronously receive the traffic could maybe do something profitable with that? Such as co-opt the Netflix API calls and display "video" or "messages" to convince the user to subscribe to a different service, netting $$ to the unintended target who received Netflix's requests

  • by Junta (36770)

    In this scenario, IPv6 would alleviate the need to so aggressively reuse IP addresses in that scenario.

    Of course, one wonders given the high amount of traffic if amazon is needlessly changing addresses. They probably should make more effort to have a tendency to be more persistent even beyond the 'promise' of the ttl. Sort of how in most DHCP servers, even when your lease expires you'll still often get the last address you had because the DHCP server retained it anyway unless pool exhaustion forces a chan

  • No dns server (or mainstream browser) caches something for 4 days when given a low TTL. I've seen some that cache for a few hours, maybe up to a day, but 4 days? Really? Something else is going on. I kind of wonder about the Netflix clients built into all those TV's, Mobile Phones, and DVD players.
  • So if you're getting millions of requests that aren't actually meant for you, that could drive up your monthly bill as well as your traffic usage. Good thing they caught that...
  • Does this story come with any indication that their isn't a mixup on Netflix's part?

    • by autocracy (192714)

      ... "there" isn't a mixup on their part. Honestly, it'd be great if the Slashdot API reacted in the same year that I clicked on preview.

...when fits of creativity run strong, more than one programmer or writer has been known to abandon the desktop for the more spacious floor. - Fred Brooks, Jr.

Working...