A Look At CERN's LHC Grid-Computing Architecture 53
blair1q writes "Using a four-tiered architecture (from CERN's central computer at Tier 0 to individual scientists' desk/lap/palmtops at Tier 3), CERN is distributing LHC data and computations across resources worldwide to achieve aggregate computational power unprecedented in high-energy physics research. As an example, 'researchers can sit at their laptops, write small programs or macros, submit the programs through the AliEn system, find the necessary ALICE data on AliEn servers, then run their jobs' on upper-tier systems. The full grid comprises small computers, supercomputers, computer clusters, and mass-storage data centers. This system allows 1,000 researchers at 130 organizations in 34 countries to crunch the data, which are disgorged at a rate of 1.25 GB per second from the LHC's detectors."
Data to crunch (Score:5, Interesting)
As someone who worked on the processing of HEP experimental data for awhile, let me say that there is a ton of work to do. You have particles entering the detector every ~40ns and hundreds of different instruments making measurements, which leads to a ton of data very quickly. You then have to reconstruct the path of the particle based off of the detector information, but it's not straight-forward. The detector can have gaps in coverage; neutrinos (which are undetectable) can be created removing momentum; particles from the previous event can still be in the detector et cetera.
And all of the data crunching you do must be done in 40ns, so that you're ready for the next set. (Of course, you can do some processing offline, but if you don't maintain a 40ns average, then your data will start piling up.)
USENIX LISA presentation (Score:2, Interesting)
There was a good presentation at LISA '07 on this entitled "The LHC Computing Challenge":
http://www.usenix.org/event/lisa07/tech/tech.html#cass
It was given by Tony Cass, who is/was "responsible for the provision of CPU and data storage services to CERN's physics community". They're planning on collecting 15PB/year.
Re:I want more! (Score:5, Interesting)
Re:1.25GB/sec not that much. (Score:3, Interesting)
You don't have to use TCP/IP over ethernet, you know. AoE & FCoE come to mind.
There are very few ways you might lose packets in a well-built local data-link network. Collisions/congestion are a thing of the past. Modern networking gear is fully capable of forwarding packets at full speed. Packets don't just go flying out of CAT-6A, never to be seen again.
And yes, bonding two 10Gbps ethernet interfaces would give you greater than 1.25GByte/s, though certainly NOT double that...
As a Tier-3 manager... (Score:3, Interesting)
I can say that the article doesn't explain it very well. Since CERN has been calling the sites "Tier", this terminology has become a buzzword, and everything is a Tier (the managers call their services "Tiered" just to make them sound important).
Tier0 and Tier1 are well described by the article. Tier2 are mostly computing clusters, with of course big storage, but they're mainly for analysis. Tier3 are like Tier2 but not really. They are "uncertified" Tier2 in the sense that they do not strictly adhere to the Tier2 standards in terms of middleware and configuration and policies.
Tier4... never heard of that, I think the buzzword Tier backfired and they're calling their desktops Tiers. When I started managing the Tier3 we did not even call it like that... it was just a cluster.