The Astronomical Event Search Engine

eldavojohn writes "Google has signed on with the Large Synoptic Survey Telescope project that will construct a powerful telescope in Chile by 2013. Google's part will be to 'develop a search engine that can process, organize, and analyze the voluminous amounts of data coming from the instrument's data streams in real time. The engine will create "movie-like windows" for scientists to view significant space events.' Google's been successful on turning its search technology on several different media and realms. Will they be successful with helping scientists tag and catalog events in our universe?" The telescope will generate 30 TB of data a night, for 10 years, from a 3-gigapixel CCD array.

Re:30 TB will be nothing in a few years.

[Spikeles]

You are a bit behind the times there.. 1TB consumer drives are here http://www.engadget.com/2007/01/05/hitachi-breaks- 1tb-hard-drive-barrier-with-7k1000/ [engadget.com]

Re:Near Earth Objects

[Wavicle]

Correct me if I'm wrong, but from what I can tell of this project, it's going to do exactly that (and more), but on a larger scope, and with better accuracy?

Well, I was a very small cog for a very large telescope. But my understanding is pretty much exactly what you just said.

Re:30 TB a night...

[Capt'n Hector]

You can't compress this stuff unless you do it losslessly. Compression artifacts mess up photometry - if you're trying to compute apparent brightness, you need to factor in things like how bright the ambient sky is, and how much point sources get spread out (FWHM, seeing). That is, a point source that passes through the atmosphere looks like a normal probabliity distribution because of atmospheric distortions. So to get an apparent brightness, you have to correct for this effect. If compression artifacts are introduced, FWHM is thrown off, and you have no idea how "crisp" your image really is. That's why these data sets are so large. Quite literally, they're doing a pixel dump from their massive ccd all night. But hey, somehow I doubt they'll be using this telescope for anything but object detection. There's no reason to store it all except to compare a current picture to one in a base set, kinda like KAIT [berkeley.edu] on stearoids.

Lots of data, but not as much as the LHC

[Phat_Tony]

That's a lot of data, but it's less than 1/10 as much data [physorg.com] as the Large Hadron Collider [web.cern.ch] will put out, and the LHC is supposed to be coming online within a year, not in six years. By the time the Large Synoptic Survey Telescope comes online, the LHC may have produced more data than the Large Synoptic Survey Telescope will over the life of the project.

I'd be interested to know more about the data handling methods they have in place for the LHC. I don't think they'll be using Excel.

*Note the correct, non-Frudian-Slip spelling of "hadron [google.com]"

Re:30 TB a night...

[hogghogg]

You can't compress this stuff unless you do it losslessly. Compression artifacts mess up photometry

This is not strictly true. What's true is that the current standard lossy compression techniques mess up photometry. However, if you know what you are going to photometer and how you are going to photometer it, it is certainly possible to compress in a lossy way without ruining the photometry. In a trivial sense, photometry is lossy compression of data (you have turned huge images into a few numbers with error bars)!