Examining Mac OS X 10.4's Spotlight 440
Ton writes "Apple has published a discussion of Spotlight, the radical systemwide search technology that will be part of Mac OS X 10.4 'Tiger'. The really interesting part is that metadata will be playing a big role in Spotlight while just a few years ago people were afraid metadata in Mac OS X was going the way of the dodo."
Radical (Score:2, Insightful)
I haven't seen any mainstream implementations (WinFS?) of it, but I didn't know it was a brand new concept.
Re:Radical (Score:5, Informative)
A Typical use will be making query's such as: Show me everything agent dero sent me between tuesday and thursday last week. Mails, IM transfered images, you name it... Best of all, since this is metadata based, it's supposed to be lightning fast
You could envision a plugin that would Spotlightify slashdot threads you read, in theory, and apply the power of a database to it.
but really, you should RTFA
Re:Radical (Score:3, Interesting)
The DB for it was custom designed for fast unicode text searches. As far as i know Apple isn't going to document the DB format but will be providing a C based API to search it.
Does the world need another DB file format? We'll see....
Comment removed (Score:5, Interesting)
Re:Radical (Score:5, Informative)
The radical difference is that Spotlight generates the metadata itself rather than you having to tag stuff yourself. It has content handlers to intelligently tag all kinds of different "stuff" so it "knows" what a Word document is and what a web page is and what a .png file is etc etc.
Re:Radical (Score:5, Informative)
You have a program called iTunes that creates a database of your music so you can search for a song by any one of a number of tags, including genre, play time, title, author, etc plus any of the keywords the user adds and how they rated it.
You have another program called iPhoto that does the same for image files because iPhoto understands the internal tags in a jpg (or other image) file.
You have another program called Finder that indexes based on file data. It knows what size the mp3 is, but not how long the song is -which iTunes does know.
You have all this separate programs for dealing with different kinds of files because they all contain different kinds of metadata and internal tags.
Spotlight puts all these kinds of searches in one place, and allows you to combine them. So with the appropriate plug-in filter, it can search any file type and take advantage of any internal tags in the file to speed up the search. Its much faster and more accurate than searching based on the entire contents of the file.
So Spotlight combines metadata it generates itself (file content), with basic file metadata (file size, creation date...) and file type specific metadata (image dimensions or song duration).
Then, IIRC, you can save your search and the results will be updated in real time as files are added or deleted.
Re:Radical (Score:5, Informative)
To my knowledge, other metadata-based search systems have not had a similar degree of extensibility. Please correct me if I'm wrong.
Re:Radical (Score:5, Informative)
I still have to be convinced that full-content indexing is a good idea. I very rarely need to search for something in the contents of a group of files, and when I do it's usually such a small group that the time saved would not outweigh the disk space used by such large indexes. On the other hand, this problem should get better over time, since the largest files are usually video, and have little indexable content, meaning that the index is likely to get relatively smaller over time (until someone writes a plug-in that can interpret objects in images, and applies this to every frame in a movie. Fortunately, I think this is still a long way off).
Re:Radical (Score:3, Interesting)
Re:Radical (Score:3, Interesting)
It's not just about search. The really cool thing about it is saving the search with Smart Folders. The biggest advantage (for me) is being able to arrange my files in several different hierarchies at the same time, without hav
Re: (Score:3, Interesting)
Re:Radical (Score:5, Informative)
* Desktop-metaphor based GUI for a personal computer
* WYSIWYG publishing with a laser printer
* PDAs via Newton
* AppleLink (err, AOL now)
* QuickTime (movies, QTVR, 3D, etc)
We could go on and on. Give Apple props where due, huh?
And please consider modding the troll down...
Re:Radical (Score:5, Insightful)
You've got to give them credit for product design as well. Nobody makes more desirable-looking software and hardware. Is it any wonder that Apple's fiercest supporters are graphic designers?
Re:Radical (Score:3, Insightful)
* Desktop-metaphor based GUI for a personal computer
Xerox invented that one.
Xerox did not invent a desktop-metaphore, they invented windows-icons-menu-pointer, look back at grandparent, apple took existing ideas and did them right.
* WYSIWYG publishing with a laser printer
Xerox invented that one too.
Xerox invented laser printing, Apple invented Publishing.
* PDAs via Newton
Invented by Psion in 1984 with the Psion 1.
Psion 1 was a digital diary, the Newton was a digital assistant
Re:Radical (Score:5, Insightful)
Name one other multimedia framework that has been around as long as Quicktime. And don't mention Video for Windows [wikipedia.org]...I'll take your response off the air.
Re:Radical (Score:3, Informative)
Re:Radical (Score:3, Insightful)
Re:Radical (Score:3, Insightful)
you can make a darn good argument that *that* is precisely the definition of 'radical.' if it wasn't implemented in a useful way and didn't see the light of the day, we wouldn't be talking about it as being 'radical' because we wouldn't be talking about it at all to begin with.
Lobbied for WiFi radio spectrum (Score:5, Interesting)
Re:Radical (Score:5, Informative)
http://developer.apple.com/macosx/tiger/spotlight
Re:Radical (Score:5, Interesting)
Re:Radical (Score:5, Insightful)
Re:Radical (Score:4, Insightful)
The metadata part was noteworthy because MacOS has always had metadata, but Apple looked like it was abandoning, or at least deprecating the concept in OS X. The fact that Spotlight will use it shows that metadata on MacOS still has a future.
Comment removed (Score:5, Informative)
Re:Radical (Score:3, Informative)
Meanwhile, which filesystem is better - one that can handle named forks or one that can't? I agree that they cause portability problems (and bundles are far more elegant), but the filesyst
Reiser (Score:5, Interesting)
He saw all this stuff comming from way back. If you read the LKML, you will remember that he warned us.
Its a pity no one listens to him.
Re:Reiser Links (Score:3, Informative)
http://kerneltrap.org/node/view/3727
http://lw
Is THIS the discussion? (Score:5, Informative)
Re:Is THIS the discussion? (Score:3, Informative)
http://developer.apple.com/macosx/tiger/spotlight
nice and long article that gets into the meat of things.
the actual discussion/article (Score:5, Informative)
What's really funny is that there's no link to the actual published discussion... but anyway...
http://developer.apple.com/macosx/tiger/spotlight
Sounds like Beagle for linux (Score:5, Interesting)
Re:Sounds like Beagle for linux (Score:3, Interesting)
Spotlight is cool, but (Score:3, Interesting)
The first thing I'll do is try making an Automator to create thumbnails. Currently I'm using a bash script I wrote on my Linux box to do this. This will be the first time I've paid for an OS upgrade since Win98, so I hope it's worth it.
FYI... (Score:5, Informative)
Re:FYI... (Score:4, Informative)
for non-beos users, here's what you need to know about befs (note that it was pretty much complete by 1995):
1) FAST. super fast. seriously.
2) 64 bit, with support for giant volumes and files (10 years ago!)
3) journaled filesystem. no fsck, no corruption on crash (trust me, my daily use system had bad ram for a while and crashed hourly).
4) metadata built in and instantly accessed. change the name of a file or any other metadata, and all your "live queries" would reflect the changes.
how long must my linux desktop wait for what beos had 10 years ago?
How long? (Score:3, Funny)
Re:How to do the hard part easily on Linux or BSD. (Score:5, Informative)
Wow. Check it out. Everything you said here is completely 100% wrong.
Spotlight is filesystem-independent. It runs as a set of daemons and stores its metadata database in a hidden directory called ".Metadata" at the root level of the volume.
All your "could be" talk is basically a summary of how Spotlight works.
Im very interested... (Score:5, Interesting)
Coming from a WindowsXP background, some things Ive noticed so far:
- Clicking the 'X' doesnt actually close the application. This annoyed me to start with, but ive slowly gotton used to it.
- Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
- Love the dock. Its just
..... right.
- Most of the file system is hidden from you, which I like. Put my data where I want it and ignore the rest.
- The ability to access the underlying BSD OS easily. Love it.
- Everything looks and feels 'polished'. THats what I always hated about KDE/Gnome when I tried them, the features were there, but noone had taken the time to step back and polish the entire thing off so it all looks and feels together.
- Every time I boot the Mac, my TFT display is 'wavey' until i have the monitor do an autoadjust. Dont really know whoes fault this is, tho its fine under windows and linux.
So, final conclusion? I love it, so much that I have already placed an order for a G5 Imac. And in the meantime, Ive purchased a G4 upgrade for this little baby, just to help it alongRe:Im very interested... (Score:5, Informative)
If you want to quickly quit a load of apps or switch application, hit cmd-Tab, and then cycle through the apps with the tab key.
However you have one gig of RAM on the system. You have no need to quit the programs when switching between them. They'll be paged out to disk as necessary if you manage to fill the available RAM. Multi-tasking works very well as processes aren't in general allowed to hog the processor.
I think this is a common thing amongst people who're used to windows - the windows in OS X represent documents, not applications, so that's why they can be closed without quitting the application. You will find Apple managed to balls this up by being inconsistent though - some applications DO quit on closing the window, but in theory they're applications which only have one window, and are utilities, like the Address Book.
Be sure to try expose as well, though I doubt it'd work well on that older system.
http://www.apple.com/macosx/features/exp
Re:Im very interested... (Score:5, Informative)
OK, use Splat-Tab (Apple/Command/Cloverleaf, call it what you will) to switch between apps. When you get to the one you want, hold down Splat and press Q. It quits the application. Press H instead and it Hides it. There's more of these... [macosxhints.com]
Hope this helps.. It seems this is OS X 10.3 only, so you might want to check out LiteSwitch X [proteron.com] which does the same thing.
Mark
Re: (Score:3, Informative)
Re:Im very interested... (Score:5, Informative)
As someone replied earlier, this is a new paradigm in app management: the top menu controls the application, and the window menu controls the window. More importantly, OSX apps are designed to be left open -- keep them open, close or hide their windows, and they'll use virtually no resources, but will start significantly faster the next time you use them.
Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
Learn your keyboard shortcuts. Take the ten minutes to learn them, and you'll regain hours of your time. Cmd-Q is the shortcut for quit, for example. If you're used to Windows machines, you can switch the cmd key with the Windows key.
Love the dock. Its just ..... right.
Check out Quicksilver, from http://quicksilver.blacktree.com . Once you get used to it [and once it gets used to you], it's phenominally faster than the Dock.
The ability to access the underlying BSD OS easily. Love it.
iTerm, from http://iterm.sourceforge.net , is a great OSX terminal app.
Here [unxmaal.com]'s a list of favorite OSX apps I posted a while back. Most are free/OSS, and they're all some of the best apps for any platform.
Menu behavior not new. . . (Score:3, Informative)
Actually, this behavior is not a new paradigm as it has been a feature of the Mac OS back before it was Mac OS -- all the way back to The Beginning.
There are a few reasons for this behavior, but the most important one is that in good UI design, each widget should serve a clear purpose. On a Mac, the "close window" widget closes windows and that's it (unles
Re:Keyboard shortcuts better on MS Windows, any ti (Score:3, Informative)
This allows you to tab between gui elements. Ctrl-F2 activates the menus for keyboard access. And you can edit shortcuts for every application you have.
Re: (Score:3, Funny)
Re:Im very interested... (Score:3, Funny)
And speaking as a former shareholder who bought at 20 and sold and 27 only to see it go to 50... screw you!
=)
Re:Im very interested... (Score:3, Insightful)
LOL. Quitting the app just because I closed a window is one of the things that annoys me the most about Windows. If I'm done working with one document in Word, I have to be sure to open up the next one before I close the first or I have to wait for Word to start up again.
Don't confuse this with anything you've seen ... (Score:5, Interesting)
This really is a big deal, much bigger than Microsoft's feeble attempts at full text search, or Google's desktop search. In many way's this much, much more useful than full-text search, especially for developers.
At home I have about 6,000 MP3s, a 1000 photos, 500 scientific articles in PDF format and hundreds of words files that I need to juggle. Each one has its own metadata database, and none of them are updated in real time.
Databases:
MP3 - WinAmp & AudioTron
Photos - Photoshop
PDFs - Acrobat Indexer
Word files - MS Indexer
That doesn't include any of the other data that is stored completely databases and would have been easier to store in the file system - like email, guitar tab files and god knows what else.
A properly implemented global meta-data store (that works at the filesystem level, not as an iterative service) profoundly changes how one uses the system, making sorting and finding data actually almost pleasurable.
Quicksilver (Score:5, Informative)
http://quicksilver.blacktree.com/
It's an app that indexes parts of your file system and supports plugins to to index application data. The best part is that it is keyboard based. For example. type command-space "slash" enter and it fires off Safari opening
I'm not sure how Apple will improve on this.
Quicksilver Versus Launchbar (Score:3, Informative)
However Launchbar has since updated to 4.0 beta release, and in doing so has pre-empted spotlight, as it does (right now, in 10.3) index system-wide metadata. So now you can cue up songs by entering
Re:Quicksilver Versus Launchbar (Score:4, Informative)
Re:Quicksilver (Score:3, Informative)
QuickSilver and Spotlight seem very similar at first glance, but are in fact very different creatures. They have the same appeal, but very different, but overlapping functions. QuickSilver is still basically a launcher, and Spotlight is still basically a Find function.
I've found that the things QuickSilver excells at are the things that Spotlight can't inherently do, like abbreviated searches (try "sl do" to launch Slashdot), complex actions, certain applicat
German tanks? (Score:5, Funny)
disk space (Score:5, Interesting)
Devon
Re:disk space (Score:3, Informative)
Re:disk space (Score:3, Insightful)
I'd say that 1GB is a lot larger than it will ever be, so it's not a concern for me at all
I'll happily spend a couple of dollars on drive space for instant searches on my local machine.
Kai
linux - OSX coexistence with spotlight (Score:3, Insightful)
stat file.jpg
in linux. Would be nice in linux to beef up on metadata too.
I hope that spotlight will work also, if you have a linux partition exported to the Mac via NFS. Will file information of NFS mounted systems also stored in the database?
Having linux and OS X working together is already now not without issues. If you have a file Test.jpg and test.jpg in your Linux partition and you copy both to the same place in OSX, the finder (on the mac) complains, because the two files are considered the same.
Re:linux - OSX coexistence with spotlight (Score:3, Interesting)
A working example of metadata use in images (Score:3, Interesting)
YOu can try the system out here [stanford.edu] with a collection of almost 4k images.
Shameless Plug (Score:3, Interesting)
Of course, this metadata will be so much cooler when something like spotlight is there to take advantage of it...
lock your screens (Score:5, Funny)
I've tried Spotlight and suggest that when it comes out, every time you step away from your computer make sure to lock your screen. All someone has to do is type 'porn' into the little search toolbar and within seconds it's all nicely listed.
Perhaps Apple needs to add a feature to turn off indexing for certain directories.
Re:lock your screens (Score:3, Informative)
Re:New app poposial - Blacklight (Score:3, Funny)
Plug-Ins (Score:4, Interesting)
How well this system works will in part depend upon how many data format plug-ins are provided. For example, take something like the SID audio format. It's relatively unknown, but has an officially registered MIME type with IANA [iana.org] giving it a status above many other file format types, and it is used to provide background sounds on some web sites. Will it make the cut?
This is just one file format chosen at random. There are thousands out there, some of which are used pretty heavily for documentation in certain circles. How about all of the OpenOffice file formats, or the AbiWord format?
I can see this feature being hugely useful if Apple does a good job of providing plug-ins, and making it easy for third-parties to add more.
Mac OS Metadata Is Just One Kind Of Metadata (Score:4, Insightful)
The kind of metadata that was almost deprecated by Apple isn't quite same thing as the "modern" concept of metadata. The classical HFS metadata covered concepts like file type, file creator, and "Finder bits" that aren't handled at the file system level in other OSes. This, combined, with the Mac OS's historical use of resource forks for storing developer defined data records, made perserving such data difficult or impossible in heterogenous environments like the Internet. It's really a shame; I've always thought this concept was the most elegant attempt to solve the problem of "rich data" associated with data files without requiring the data in the file itself to have some form of universal container format.
The metadata concept used by Spotlight is going to be based in part on a plug-in system that allows the Mac OS to reconstruct metadata information from the data within files themselves, rather than just using the metadata facilities provided by HFS and Mac OS resource forks. That means that each different kind of file, from Word documents to PDFs to Postscript jobs, needs its own special kind of processing to read its own format of storing such data. It's less elegant and more processor intensive that just using the historical HFS system, but it's more likely to to be useful for extracting metadata from files provided by Windows and other Unix variant users.
Still needs work (Score:4, Interesting)
I'm tired of apple ripping off ideas from developers without (A) Giving them credit or (B) developing something equivalent so the new as at least as feature-full as the old. Based on apple's history, the first version of Spotlight will likely be a horribly dumbed down version of Launchbar in terms of tech, since apple is obsessed with "ease of use": i.e. a three year old has to be able to work it.
Rant aside, there are a few key pieces I think apple is missing:
(1) User-created metadata. I should be able to tag anything I want with any metadata I want so the organization system follows ME and MY preferences, instead of the system determining it for me. Apple should be thinking about taking the insanely wonderful metadata system they created in iTunes and applying that to the finder. It is essential you be able to tag metadata in, because you don't always access the same objects for the same purposes.
(2) Flexible file system. This is a concept I've developed which basically says that the file system should be dynamic and adaptable to match the thought flow of the user (only possible with a good metadata file system). If you've ever seen this app on the PC, think: "The Brain". What that means is that if apple does #(2) right, it should be easy as hell to tag things, and then basically I can create relationships which let me "flow" through my files by navigating CONCEPTS instead of folder heirarchy. A good app that does this is Devonthink. Devonthink will grab the contents out of your files, and when you do a search, you can not only see your search term but "related" search terms. Click on a new search term and you get a new listing. So as you come up with ideas about what you want to do, you can easily and naturally branch off into other parts of your file system. This methodology models the way the human brain actually works- thinking in concepts and spacial organization, rather then structure. (The "flexible" comes because the system takes your tags and adapts the search around them, allowing you to change how the "flow" works, depending upon what topics are most important to you.)
(3) The next level after metadata search is a new way of visually interpreting the metadata and relationships between. Which means a NEW FINDER. I can't believe Steve actually threw this comment out after demoing Spotlight: "With this, you probably won't even need to use the finder any more." Well then why even have the Finder at all, Steve?! There IS a reason for the finder, which is why it's stayed around all these years, and that is that people think SPACIALLY. People are creatures of habit, and one way we remember where things are is if we know where to look for it and it's always in the same place. Which means there needs to be a visual grounding to the above dynamic files system, to give people a sure footing to all of this. I'm talking about things like a window that always stays in the same spot and always performs the same task, like showing you what new files have been added to the system, or actively updating your list of word documents wherever they are. Right now in the finder, a window is a window is a window. That shouldn't be. If a search is applied to a window, then that window isn't just showing you files, it's performing an active function. The finder needs to evolve to take on the new roles and responsiblities it should have in the context of a metadata files system. Spotlight should replace the finder: the two should work together seamlessly.
The good news is that Spotlight is built into the system, so even if apple screws up the implimentation (likely), the next generation of 3rd party apps will hopefully be able to fill in the gaps.
Re:Still needs work (Score:3, Funny)
Finder comments do this.
(2)
Whatever.
(3)
Whatever.
Spotlight vs. Quicksilver/Launchbar (Score:3, Insightful)
The technologies are barely related; Apple is not ripping off QS/LB in the least here. Spotlight is a technology for searching through files based on their conent and metadata. QS/LB are utilities for finding files based on easily typed mnemonics. You are looking at one aspect of Spotlights appearance (the dropdown search pane in the corner) and assuming it's a ripoff based on some similarity to the appearance of the other utilities.
In fact, the Spotlight indexing technology will be a boon to the utilities
Great. Just fricking great. (Score:3, Funny)
I guess I now have to go back to a "download as needed then delete" paradigm.
Sheesh, I wish they'd think these things through.
Re:Great. Just fricking great. (Score:3, Informative)
Extrinsic vs intrinsic metadata (Score:3, Insightful)
However, from what I've seen, that's not the sort of thing Spotlight is about. The plugins we're talking about make use of intrinsic metadata - information extracted from the datastream itself. Many common file types include some descriptive information: EXIF data in pictures, MP3 tags in audio files, meta tags in HTML files, and so on. Spotlight is a way of extracting and using that data.
The practical differences include, OTTOMH:
- Spotlight's information won't be lost when files get stored on other file systems, sent over email, processed on other platforms, &c.
- Spotlight uses information that's already in the files - you won't have to set it up manually.
- You can use existing tools to see and edit the metadata - MP3 taggers, photo editors, whatever. And you can do so on any machine and OS.
This is probably one of those rare cases when that foul word 'leverage' might be appropriate -- Spotlight should allow you to make much better use of an existing resource. As such, it sounds like a jolly neat idea!Search Ontology (Score:3, Insightful)
Simson Garfinkle's "Sbook.app" from NeXT in the 90's.
The usefulness of Sbook.app ability to add tokens in a flat file for instantaneous searches enabled people to apply Sbook.app outside its realm of address book that it originally was designed.
Abstracting its functionality and interoperating at the kernel level is pure Apple polish on the brand. Until people start using "Spotlight", the verdict will be out on adoption across the platform.
I will venture it will be one of the defining characteristics of the Mac platform into the future.
Backups? (Score:3, Interesting)
Also, how about remote file systems (nfs for example). Resource files are mapped as regular files with a
Search on the desktop (Score:3, Interesting)
Is search really so relevant for a single computer and the average desktop user? Most people already organise their files in a somewhat structured way, and generally know where to find stuff. (Especially if they use OS X)
Sure powerful file search might be useful occasionally, but i don't see it as a huge issue that companies like M$ think it is.
Practical examples of what spotlight does... (Score:3, Insightful)
If you have a mac with a ton of files, various "Previous System Folders" etc...follow along :)
I have smart folders for pdfs, avis, mpgs, and wmvs
I have these sorts of files *all over the place*...movie clips, test files, you name it.
I go to the finder, "open" the Windows Media Files folder, and they are all "there"
Or all the "archive" files (zip, rar, sit/sitx etc) i've collected and not erased in the last year...
or all of the emails i've received from japanese users...
it goes on and on.
To me, its like the whole star trek "Computer..find all of the blah blah blah for sector Whatever"
It concentrates on the "what you want" as opposed to the current paradigm of where did i pit it/what app did i use, etc
Re:Pre-emptive post (Score:3, Funny)
Re:um.. (Score:4, Informative)
Could you send me the source for the version you have installed that does that?
Re:um.. (Score:2, Informative)
That might be the case. [freshmeat.net]
Re:Sounds like Windows, actually (Score:3, Interesting)
Re:Sounds like Windows, actually (Score:2)
~phil
Re:Sounds like Windows, actually (Score:5, Informative)
However it only comes with a few configured filetypes settings, and no way to set a default "When no searchFilter available, treat as plain text" setting.
I stressed and strained about this when XP came out initially. The only way I found to do it so I got expected results was to build myself a scanner.
It searched through a drive, and identifies EVERY file extension.
It then looks through the registry to see which Extensions have linked Handlers.
It generates a reg file containing stub links for every unmatched filetype.
Its a bit shotgun, but allowed me to continue using the Text search for XP.
Microsoft have released their own shotgun registry pack, for more info see here:
http://support.microsoft.com/default.aspx?scid=kb
(I have since moved myself into using my own full search tool, but at least the XP search doesn't miss files which are clearly within visible range).
[Now for the science part..]
Take a file, something like "PunchTheMonkey.asp".
Make sure you have it open in notepad, and make sure there is a certain text string - for instance "spyware".
Open the windows XP search in that folder, tell it to search *.ASP, and give it the phrase "spyware".
Windows XP will NOT find this file.
-----
The Windows
[HKEY_CLASSES_ROOT\.txt\PersistentHandler]
@="
Adding an entry like the one above for each required filetype will restore the full text search functionality.
So, I add the following entry into the correct
[HKEY_CLASSES_ROOT\.ASP\PersistentHandler]
@="
After I have logged off/rebooted, I try the same again, and XP will now identify the file.
Re:Sounds like Windows, actually (Score:5, Interesting)
I believe though that the indexing is done during saves, so you'll not notice a general system slow down. What you will notice is a slow down on file saves.
Re:Sounds like Windows, actually (Score:5, Informative)
Re:Sounds like Windows, actually (Score:3, Informative)
Apps need to be made "Spotlight-aware" in order to invoke the Spotlight indexing on save.
Re:Sounds like Windows, actually (Score:5, Informative)
For one, it doesn't take half an hour, it shows you the results as you type, instantaneously.
Secondly, via plugins it can understand *any* file, such as an image metadata importer that uses OCR so you can search for words, or a Flesh-tone detector so you can search for all your porn that way.
Re:Sounds like Windows, actually (Score:3, Informative)
makewhatis.cron can be a pain on Linux as well, if it is on a workstation which is mostly switched off.
Unfortunately for windows boxes, they do tend to be left shut down a lot of the time, so more of their runtime is spent rebuilding the search database when the machine is being used for something, rather than in the middle of the night, which is the preferred way
Re:Sounds like Windows, actually (Score:5, Informative)
Already the differences in Fat32/NTFS versus HFS+ (the mac filesystem) yield significantly faster searches before spotlight is introduced. Sit down on an OSX apple and notice that an entire search of the HD is actually a fast operation, not the waiting many-minute exercise that it is on windows.
Now since spotlight is built into the core of the system, and isn't just a tack-on service like the windows indexer is, there are significant speed advantages, updating the SQL database when files are modified, added, etc is incredibly light on the CPU, and is equivalent to doing something like changing the file name.
What spotlight isn't, and this might be where you are getting confused, spotlight isn't a spider that crawls from folder to folder cataloguing information about each file, which is what the windows indexer was doing, hence why it was resource intensive, as it was busy checking files and folders that you have possibly not made any changes to.
As a counter to the 'Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.' Microsoft, google and apple would disagree. Having an up-to-date catalogue without the CPU strain is a must have, go figure MS have been trying to implement it since NT4.0.
a problem that doesn't really exist (Score:5, Informative)
Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.
On the contrary, this is a *better* solution to a very basic problem that has plagued computers since they were invented.
The problem
How do I organise and access the data I use every day (emails, letters, images, music etc)?
The old solution
You can put your files in folders (one per file). You can name the files with a short description, ending with a cryptic 3 letter code to denote the file type. Files *must* be in one category/folder only at a time. Limited meta-data (date modified, file-type etc) may be stored.
The new solution
You add meta-data to files (often automatically) saying who created them, what project it's under, whether it's 'to do' or 'unfinished' or whatever. You'd do this in a save dialog for the application, as you saved the file. All other applications which use searchlight will update their view of this stuff for free, in real time.
When you want to work on a project, you click on the live project folder, and immediately you see all the files, emails, images etc for that project, no more, no less, regardless of where they are on the disk and what other projects they're shared with.
Want to see all the stuff to do with John, 5 months ago? On this project? Containing the word gizmo? That sort of query will be easy to make.
If you have an image editing application, it can show you all the images taken in Paris in 2002, without having to build a database application into it. This makes adding this kind of feature to applications trivial.
Ideally adding meta-data tags like 'project-1', and 'To do' should be as easy as choosing them in the save dialog or applying them like a label in the Finder. It's not quite at that stage yet, but that should come later. Some of these ideas are quite old (Be), but they are long overdue in a desktop operating system.
Re:Sounds like Windows, actually (Score:5, Informative)
Apple has had this type of search engine before, they called it V Twin and it was a basic part of Copland. This is what Sherlock used in Classic and why it was so fast. The idea is even older, it's from a conceptual computer interface Apple dubbed the Knowledge Navigator. All this appears to be is V Twin running on SQLite instead of a proprietary method.
The interesting part to me is the focus on metadata. I loved this feature in BFS that metadata was king. This is going to lead the way to better file management. Hopefully the Finder will integrate it.
Re:Sounds like Windows, actually (Score:5, Insightful)
Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.
Doesn't exist *for you* perhaps. Perhaps you don't have a lots of user data, or you have taken time to sort it into useful folders. I'd say it's about as useful as the incremental seach in iTunes is. Sure I could remember what artist did a track, and access a track by scrolling down to that artist, then finding the track. Or I could scroll down the list of thousands of track names, remembering my alphabet ordering, and locate the track that way. Assuming I've remembered the exact wording of track name. But I've always found it easier to type whatever word comes to mind first from artist or track into the search box.
And so it is with documents. Even if I do remember the file name and folder that a particular piece of information is stored in, I still need to navigate there. Most times it will be quicker just to type in whatever it is you remember about the data you want into a search box - even if you know where the data is stored.
Re:What's so special about searching (Score:4, Insightful)
Try to pretend that you're managing 2 or 3 or more major projects that can change or be passed along to someone else every few months with mails, im's, files, reports you don't look at, media submitted by other people in different countries, to-do lists and other project management data...
Now imagine someone asks you, the project manager (or just the last person still around) on a project from 3 years ago, what the initial proposal from that guy in japan who did the Flash files was versus what we paid him and what the VP's said about that....
People *will* have copies of these files still floating around *somewhere* in e-mail or im history, at least. You may not, I may not, but that's where this will come in handy.
A few years ago, hd space was not large enough to think that you'd keep all that data around, but gmail's new 1Gb e-mail storage just showcases the lack of a need to dump all that crap off your media if you can just organize it well, and who needs that when you can keyword search, anyway?
Re:What's so special about searching (Score:3, Insightful)
Re:What's so special about searching (Score:5, Funny)
Re:What's so special about searching (Score:5, Funny)
Solution?
Save the porn / super personal stuff on an encrypted disk image saved somewhere inconspicuous, and set cronned (or logout) scripts to scrub your various histories and recent items. Make sure that the machine logs you out after no more than 10 minutes of activity.
Hypothetically, that is.
Hi sweety!
Re:What's so special about searching (Score:3, Informative)
Re:Metadata? In a time like this? (Score:4, Funny)
A disorganized revolution is just a waste of time.
Keeping one's data organized is a priority, bucko.
Re:Spotlight/ Google Desktop Search/ Win FS ? (Score:5, Interesting)
Spotlight can support arbitrary file types, entirely dependant on what an application developer decides to supply, and you decide to install. Google is limited to the file types Google implements.
WinFS is an overly complicated pile of steaming pooh, that Microsoft are having trouble delivering.
Re:Spotlight/ Google Desktop Search/ Win FS ? (Score:3, Interesting)
While schematized semi-structured DAGs of data may be overkill for many applications, you might be surprised how often something like this is needed, and how few developers actually have the skill to build it when it is necessary.
It is not uncommon for Windows developers to use a Jet database as their "file format", and just rename the extension to something else. Right off the top of my head I can think of three [1,2,3] apps that do this. CityDesk and ContentSaver would both be much better
Re:Geeky question on instant search results (Score:4, Interesting)
The exact method for matching the search string to the lexicon isn't clear. It could be a suffix tree, but it may be as simple as grep-like scanning of the words, since there aren't that many relative to the text size.
Looking at mail.app it seems to do this process on each keystroke. It's not terribly fast, but it gets the job done.
Re:Smart folders (Score:3, Interesting)