Halo 3 Causing Network Issues

Recently at my university where I'm a student and a sys admin, we have been experiencing some odd outages, in particular since the 25th of September. The outages seemed to occur between 8 PM and 12:00 AM — peak gaming hours for our dorms. It just happens that Halo 3 came out on the 25th of September. Upon further investigation we found that our network routers were shaping TCP packets, but not UDP. Once we applied UDP shaping as well, all network outages ceased. Gamers complained, but university students attempting to access network resources such as our UNIX clusters were satisfied.

Re:Doubts

[MindStalker]

No in large university networks you have to set bandwidth caps for all users, (pst your ISP does this now and sets it at whatever Mbps you buy). His router wasn't even looking at UDP traffic so a few users were getting more than they were assigned.

Re:My school's network sucks :/

[Bert64]

Your 45mb pipe, even shared between 2000 users is a lot more than people had a few years ago... Consider dialups, and the college where i studied had 2mb shared between around 2000 students.
But you need to consider the usage patterns, most ISPs will put way more than 2000 users on a single 45mb pipe, because the average user uses very little bandwidth. Your school clearly has a higher proportion of heavy users, and so it's bandwidth is more saturated.
Do remember that this is a SCHOOL... It's purpose is to educate the kids, not to facilitate them playing games. They really have no obligation to support any use of the internet other than legitimate educational purposes.
They could quite easily filter everything except HTTP, and require that you make and justify a request, via a teacher, for anything else.

Re:Doubts

[X0563511]

The solution? PLAY A LAN GAME.

No need to use the internet if you are all inside the network.

Don't know if Halo 3 has that as an option however... if not, fucking stupid...

Re:Doubts

[arivanov]

Missed one - the routers crapped out on packets per second, not on bandwidth. This is probably the most likely one.

As far as the network protocols being abissmall you are about right. They have devolved over time.

Once past the stage of serial connections, the early gaming protocols efforts tried to use multicast+unicast or broadcast+unicast technologies to run peer-to-peers like networks where people truly played against each other.

These times are gone. It is all client-server now.

This explains the admin problem - I bet that most of the students were fragging each other silly together and played within the same map and the same game. That all ended up as a lot of client-server connections. This does not consume a lot of bandwidth on average, but it is capable of flatlining the network for short periods of time every time something interesting happened in the game because the data is tromboned back and forth across the same bottleneck many times. 1 student moves and the server sends the data to 16 others, and so on. Essentially this is a form of amplification/positive feedback loop. If the same students were playing games with other people located elsewhere the effect would not have occurred.

This is a classic example of devolving and microsoftization of the gaming protocols. If the game was running locally using broadcast+ unicast or multicast_unicast to inform all local participants and only one dedicated hypernode checked what is going on outside the small "local world" there would have been no bandwidth/pps problem.

Re:Doubts

[KiloByte]

Sixteen players in a big outdoors map, where they can see a lot of stuff? That's not an insignificant number of KBs per second. And if your college is like mine was--where some nights, literally 75% of the player population was Haloing--that can easily hit a few MB per second.

Well, trees, doors, walls, signs, etc tend to not move a lot. So usually all you need to send are the players themselves, bullets, and that's about it.

Okay, on second thought, that's not all that huge a number. But doesn't UDP have more overhead than TCP? Eh, I guess I don't know that much about this after all.

Actually, UDP has slightly less overhead: 20 bytes+layer 3 header for TCP (=32 bytes + Ethernet frame over IPv4), and 8 bytes+layer3 = 20 bytes+Ethernet over IPv4 for UDP. Yet, with TCP, the OS forces as least some congestion control, something that game programmers typically completely disregard.

Re:Doubts

[garbletext]

Yet, with TCP, the OS forces as least some congestion control, something that game programmers typically completely disregard. TCP has a lot more than congestion control and packet size that makes it a poor choice for games. It's overhead includes acknowledgements, error correction, and things like guaranteed packet order and retransmission of lost packets. For a game, especially a FPS, you want to fully utilize available bandwidth. If a packet has to get retransmitted, or held because it's not in order, it's going to be stale by the time it arrives, so you might as well ignore it. UDP's connectionless nature means that clients can just spew packets as fast as their link allows.

Re:Doubts

[Anonymous Coward]

For a game, especially a FPS, you want to fully utilize available bandwidth.

I don't know of any FPS programers who would want to fully utilize available bandwidth. If they did, it wouldn't scale and their game would suck.

What they do want to do, is minimize latency to keep character movement smooth and accurate to give the perception of real-time immersion and do that with a minimum of bandwidth usage, so that their game will scale to a practical number of players. Minimizing bandwidth usage helps improve latency and scaling.

Re:TAGGED AS KDAWSONSUCKS

[Nicholas Evans]

I must agree with this Coward. Some random university's network didn't have traffic shaping set up correctly. So? This is somehow newsworthy?

Re:Doubts

[gallwapa]

uh, multicast isn't broadcast

Re:Wait a second.

[arcsimm]

The network is there for research purposes, so thats students can do the research they need to pass their educational courses. Any traffic that facilitates the educational courses of the university should be prioritised, and anything else should get whatever bandwidth remains.

Though the article author doesn't say what university he's from, if the students there pay any kind of significant connection fee for Internet access this argument won't fly. Where I attend, I pay roughly $25 a month -- what a decent midrange residential DSL package costs -- to be able to access the Internet in my dorm room. Above and beyond that, all on-campus intranet services are available regardless of whether or not that fee is paid, including our Blackboard site and libraries. If I'm going to pay that much for what is solely an Internet connection, I expect to be able to damn well whatever I like with it (though the administrators appear to disagree when they blame the perpetually abysmal service on "non-academic use"), and so should the students who are getting their games throttled down to unplayability.

Re:Doubts

[Pc_Madness]

Yeah, you really don't. The amount of bandwidth goes up with the amount of players in a server. The server reports the position of a player, and any actions it might be performing... for example a grenade has been thrown from here with this trajectory. It will also report any kind of moveable objects. Map size doesn't come into it, since its only sending out information on the people and their locations, not every square inch of the map. As for TCP vs UDP. UDP is connectionless, so it has no overhead, while TCP does. Games rely on UDP, as is my understanding, but still use TCP for a few things. As for how much bandwidth a 16 player game of Halo uses.. I would think it would be about 10-15kB/s. CS is usually higher than that, about 15kB while a game of Gears of War is alot lower.. perhaps 5-10kB, but generally most 360 games are pretty good bandwidth wise. (I'm trying to judge speed based on my download usage, so these numbers aren't exactly accurate. :p)

UDP Packets

[LordMyren]

Just for the record, dropped (shaped) udp packets are not recovered. TCP/IP notices dropped packets, has them resent, and automatically lowers the connection's transmission rate, whereas with UDP you're just tossing EMP's into people's datastreams. UDP/IP is much more primitive, and relies on application level consistency checks, which, for the record, almost never ever ever monitor packet drops & throttle themselves down when packets start dropping. Thats why most packet filtering systems simply de-prioritize UDP and will not drop UDP.

Re:Doubts

[FredMenace]

Here's the rub: back in 1994, Bungie put out a Macintosh 1ps game called Marathon. Now this game Marathon had rudimentary "round robin" LAN networking (AppleTalk-based, not IP). While there was a "gathering" machine, there wasn't one authoritative server - each copy of the game ran its own simulation of the world, and passed some information about what keys its player had pressed on to the next machine in the circle, and you just hoped they were all concluding the same things were happening at the same time. The problem was that if they got out of sync due to any excessive delays, packet loss, etc. (imagine two players think they have killed each other, before each receives the message that the other may have killed them first), or even internal bugs where they each calculated and randomly came up with different results, the game would be irretrievable - one player would think they were kicking ass, while the others would wonder why that player was facing a wall at point blank range, repeatedly suiciding with the rocket launcher. Nothing to do but abandon the game and start over. This happened quite a lot. It also precluded Internet play entirely. (Incidentally, since the "film/demo" recording feature used the same technology, it also happened sometimes that films would get "out of sync" with the playback machine relative to the machine that recorded it, and thus not make any sense either.)

Naturally, this was such a successfull network model, Bungie chose to use it again in Marathon 2: Durandal. And again in Marathon: Infinity. As well as Myth: the Fallen Lords, Myth 2: Soulblighter, and Halo: Combat Evolved (the code having been worked over somewhat by this time, without really fixing its fundamental flaws). This despite the fact that id had shown a superior way with Quake (specifically QuakeWorld) since 1996, and that others, such as Epic, had struggled (and eventually succeeded) to create similarly-good networking in games like Unreal Tournament soon afterward. It wasn't until Halo 2 that Bungie finally decided that having crappy 10-year old networking code wasn't really the best idea. More to the point, the game had to run on Xbox Live, and the old networking simply DOES NOT WORK (or at least provide a playable game) for a 1ps game on the internet, though it marginally worked in the RTS Myth series. So they HAD to fix the networking, if they wanted any Xbox Live networking at all.

So yes, as of Halo 2, they have fairly modern networking with decent prediction (borrowing liberally from community-developed ideas that were applied to Counterstrike etc., for instance with each client not only asserting what controls were affected by the player, but what actions it believes should have been taken in the world, which the server will only override if it doesn't fit with its own model), but it sure took them long enough to get there.