Slashdot Log In
Examining Mac OS X 10.4's Spotlight
Posted by
michael
on Mon Nov 08, 2004 05:04 AM
from the seek-and-ye-shall-find dept.
from the seek-and-ye-shall-find dept.
Ton writes "Apple has published a discussion of Spotlight, the radical systemwide search technology that will be part of Mac OS X 10.4 'Tiger'. The really interesting part is that metadata will be playing a big role in Spotlight while just a few years ago people were afraid metadata in Mac OS X was going the way of the dodo."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Reiser (Score:5, Interesting)
He saw all this stuff comming from way back. If you read the LKML, you will remember that he warned us.
Its a pity no one listens to him.
Is THIS the discussion? (Score:5, Informative)
the actual discussion/article (Score:5, Informative)
What's really funny is that there's no link to the actual published discussion... but anyway...
http://developer.apple.com/macosx/tiger/spotlight
Sounds like Beagle for linux (Score:5, Interesting)
FYI... (Score:5, Informative)
Re:FYI... (Score:4, Informative)
for non-beos users, here's what you need to know about befs (note that it was pretty much complete by 1995):
1) FAST. super fast. seriously.
2) 64 bit, with support for giant volumes and files (10 years ago!)
3) journaled filesystem. no fsck, no corruption on crash (trust me, my daily use system had bad ram for a while and crashed hourly).
4) metadata built in and instantly accessed. change the name of a file or any other metadata, and all your "live queries" would reflect the changes.
how long must my linux desktop wait for what beos had 10 years ago?
Parent
Re:How to do the hard part easily on Linux or BSD. (Score:5, Informative)
Wow. Check it out. Everything you said here is completely 100% wrong.
Spotlight is filesystem-independent. It runs as a set of daemons and stores its metadata database in a hidden directory called ".Metadata" at the root level of the volume.
All your "could be" talk is basically a summary of how Spotlight works.
Parent
Im very interested... (Score:5, Interesting)
Coming from a WindowsXP background, some things Ive noticed so far:
- Clicking the 'X' doesnt actually close the application. This annoyed me to start with, but ive slowly gotton used to it.
- Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
- Love the dock. Its just
..... right.
- Most of the file system is hidden from you, which I like. Put my data where I want it and ignore the rest.
- The ability to access the underlying BSD OS easily. Love it.
- Everything looks and feels 'polished'. THats what I always hated about KDE/Gnome when I tried them, the features were there, but noone had taken the time to step back and polish the entire thing off so it all looks and feels together.
- Every time I boot the Mac, my TFT display is 'wavey' until i have the monitor do an autoadjust. Dont really know whoes fault this is, tho its fine under windows and linux.
So, final conclusion? I love it, so much that I have already placed an order for a G5 Imac. And in the meantime, Ive purchased a G4 upgrade for this little baby, just to help it alongRe:Im very interested... (Score:5, Informative)
If you want to quickly quit a load of apps or switch application, hit cmd-Tab, and then cycle through the apps with the tab key.
However you have one gig of RAM on the system. You have no need to quit the programs when switching between them. They'll be paged out to disk as necessary if you manage to fill the available RAM. Multi-tasking works very well as processes aren't in general allowed to hog the processor.
I think this is a common thing amongst people who're used to windows - the windows in OS X represent documents, not applications, so that's why they can be closed without quitting the application. You will find Apple managed to balls this up by being inconsistent though - some applications DO quit on closing the window, but in theory they're applications which only have one window, and are utilities, like the Address Book.
Be sure to try expose as well, though I doubt it'd work well on that older system.
http://www.apple.com/macosx/features/exp
Parent
Re:Im very interested... (Score:5, Informative)
OK, use Splat-Tab (Apple/Command/Cloverleaf, call it what you will) to switch between apps. When you get to the one you want, hold down Splat and press Q. It quits the application. Press H instead and it Hides it. There's more of these... [macosxhints.com]
Hope this helps.. It seems this is OS X 10.3 only, so you might want to check out LiteSwitch X [proteron.com] which does the same thing.
Mark
Parent
Re:Im very interested... (Score:5, Informative)
As someone replied earlier, this is a new paradigm in app management: the top menu controls the application, and the window menu controls the window. More importantly, OSX apps are designed to be left open -- keep them open, close or hide their windows, and they'll use virtually no resources, but will start significantly faster the next time you use them.
Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
Learn your keyboard shortcuts. Take the ten minutes to learn them, and you'll regain hours of your time. Cmd-Q is the shortcut for quit, for example. If you're used to Windows machines, you can switch the cmd key with the Windows key.
Love the dock. Its just ..... right.
Check out Quicksilver, from http://quicksilver.blacktree.com . Once you get used to it [and once it gets used to you], it's phenominally faster than the Dock.
The ability to access the underlying BSD OS easily. Love it.
iTerm, from http://iterm.sourceforge.net , is a great OSX terminal app.
Here [unxmaal.com]'s a list of favorite OSX apps I posted a while back. Most are free/OSS, and they're all some of the best apps for any platform.
Parent
Don't confuse this with anything you've seen ... (Score:5, Interesting)
This really is a big deal, much bigger than Microsoft's feeble attempts at full text search, or Google's desktop search. In many way's this much, much more useful than full-text search, especially for developers.
At home I have about 6,000 MP3s, a 1000 photos, 500 scientific articles in PDF format and hundreds of words files that I need to juggle. Each one has its own metadata database, and none of them are updated in real time.
Databases:
MP3 - WinAmp & AudioTron
Photos - Photoshop
PDFs - Acrobat Indexer
Word files - MS Indexer
That doesn't include any of the other data that is stored completely databases and would have been easier to store in the file system - like email, guitar tab files and god knows what else.
A properly implemented global meta-data store (that works at the filesystem level, not as an iterative service) profoundly changes how one uses the system, making sorting and finding data actually almost pleasurable.
Quicksilver (Score:5, Informative)
http://quicksilver.blacktree.com/
It's an app that indexes parts of your file system and supports plugins to to index application data. The best part is that it is keyboard based. For example. type command-space "slash" enter and it fires off Safari opening
I'm not sure how Apple will improve on this.
Re:Quicksilver Versus Launchbar (Score:4, Informative)
Parent
German tanks? (Score:5, Funny)
disk space (Score:5, Interesting)
Devon
lock your screens (Score:5, Funny)
I've tried Spotlight and suggest that when it comes out, every time you step away from your computer make sure to lock your screen. All someone has to do is type 'porn' into the little search toolbar and within seconds it's all nicely listed.
Perhaps Apple needs to add a feature to turn off indexing for certain directories.
Plug-Ins (Score:4, Interesting)
How well this system works will in part depend upon how many data format plug-ins are provided. For example, take something like the SID audio format. It's relatively unknown, but has an officially registered MIME type with IANA [iana.org] giving it a status above many other file format types, and it is used to provide background sounds on some web sites. Will it make the cut?
This is just one file format chosen at random. There are thousands out there, some of which are used pretty heavily for documentation in certain circles. How about all of the OpenOffice file formats, or the AbiWord format?
I can see this feature being hugely useful if Apple does a good job of providing plug-ins, and making it easy for third-parties to add more.
Mac OS Metadata Is Just One Kind Of Metadata (Score:4, Insightful)
The kind of metadata that was almost deprecated by Apple isn't quite same thing as the "modern" concept of metadata. The classical HFS metadata covered concepts like file type, file creator, and "Finder bits" that aren't handled at the file system level in other OSes. This, combined, with the Mac OS's historical use of resource forks for storing developer defined data records, made perserving such data difficult or impossible in heterogenous environments like the Internet. It's really a shame; I've always thought this concept was the most elegant attempt to solve the problem of "rich data" associated with data files without requiring the data in the file itself to have some form of universal container format.
The metadata concept used by Spotlight is going to be based in part on a plug-in system that allows the Mac OS to reconstruct metadata information from the data within files themselves, rather than just using the metadata facilities provided by HFS and Mac OS resource forks. That means that each different kind of file, from Word documents to PDFs to Postscript jobs, needs its own special kind of processing to read its own format of storing such data. It's less elegant and more processor intensive that just using the historical HFS system, but it's more likely to to be useful for extracting metadata from files provided by Windows and other Unix variant users.
Still needs work (Score:4, Interesting)
I'm tired of apple ripping off ideas from developers without (A) Giving them credit or (B) developing something equivalent so the new as at least as feature-full as the old. Based on apple's history, the first version of Spotlight will likely be a horribly dumbed down version of Launchbar in terms of tech, since apple is obsessed with "ease of use": i.e. a three year old has to be able to work it.
Rant aside, there are a few key pieces I think apple is missing:
(1) User-created metadata. I should be able to tag anything I want with any metadata I want so the organization system follows ME and MY preferences, instead of the system determining it for me. Apple should be thinking about taking the insanely wonderful metadata system they created in iTunes and applying that to the finder. It is essential you be able to tag metadata in, because you don't always access the same objects for the same purposes.
(2) Flexible file system. This is a concept I've developed which basically says that the file system should be dynamic and adaptable to match the thought flow of the user (only possible with a good metadata file system). If you've ever seen this app on the PC, think: "The Brain". What that means is that if apple does #(2) right, it should be easy as hell to tag things, and then basically I can create relationships which let me "flow" through my files by navigating CONCEPTS instead of folder heirarchy. A good app that does this is Devonthink. Devonthink will grab the contents out of your files, and when you do a search, you can not only see your search term but "related" search terms. Click on a new search term and you get a new listing. So as you come up with ideas about what you want to do, you can easily and naturally branch off into other parts of your file system. This methodology models the way the human brain actually works- thinking in concepts and spacial organization, rather then structure. (The "flexible" comes because the system takes your tags and adapts the search around them, allowing you to change how the "flow" works, depending upon what topics are most important to you.)
(3) The next level after metadata search is a new way of visually interpreting the metadata and relationships between. Which means a NEW FINDER. I can't believe Steve actually threw this comment out after demoing Spotlight: "With this, you probably won't even need to use the finder any more." Well then why even have the Finder at all, Steve?! There IS a reason for the finder, which is why it's stayed around all these years, and that is that people think SPACIALLY. People are creatures of habit, and one way we remember where things are is if we know where to look for it and it's always in the same place. Which means there needs to be a visual grounding to the above dynamic files system, to give people a sure footing to all of this. I'm talking about things like a window that always stays in the same spot and always performs the same task, like showing you what new files have been added to the system, or actively updating your list of word documents wherever they are. Right now in the finder, a window is a window is a window. That shouldn't be. If a search is applied to a window, then that window isn't just showing you files, it's performing an active function. The finder needs to evolve to take on the new roles and responsiblities it should have in the context of a metadata files system. Spotlight should replace the finder: the two should work together seamlessly.
The good news is that Spotlight is built into the system, so even if apple screws up the implimentation (likely), the next generation of 3rd party apps will hopefully be able to fill in the gaps.
Re:Radical (Score:5, Informative)
A Typical use will be making query's such as: Show me everything agent dero sent me between tuesday and thursday last week. Mails, IM transfered images, you name it... Best of all, since this is metadata based, it's supposed to be lightning fast
You could envision a plugin that would Spotlightify slashdot threads you read, in theory, and apply the power of a database to it.
but really, you should RTFA
Parent
Re:Radical (Score:5, Interesting)
Apple has had a few developer kitchens on writing Spotlight importers. The idea is that any given app developer might have his own ideas as to what constitutes the interesting searching criteria for his file types. Apple has importers for common image formats, plain text, rich text, mail messages, etc.
If you were a photographer, for example, and you have a fancy camera that puts a lot of info into the EXIF tags of the image files it generates, you could search for "all images I made using this particular lens with a f-stop setting between 2.5 and 3", or if you're looking through files from a music notation program, you could search for "all files in 5/8 time in the key of G minor".
-jcr
Parent
Re:Radical (Score:5, Informative)
The radical difference is that Spotlight generates the metadata itself rather than you having to tag stuff yourself. It has content handlers to intelligently tag all kinds of different "stuff" so it "knows" what a Word document is and what a web page is and what a .png file is etc etc.
Parent
Re:Radical (Score:5, Informative)
You have a program called iTunes that creates a database of your music so you can search for a song by any one of a number of tags, including genre, play time, title, author, etc plus any of the keywords the user adds and how they rated it.
You have another program called iPhoto that does the same for image files because iPhoto understands the internal tags in a jpg (or other image) file.
You have another program called Finder that indexes based on file data. It knows what size the mp3 is, but not how long the song is -which iTunes does know.
You have all this separate programs for dealing with different kinds of files because they all contain different kinds of metadata and internal tags.
Spotlight puts all these kinds of searches in one place, and allows you to combine them. So with the appropriate plug-in filter, it can search any file type and take advantage of any internal tags in the file to speed up the search. Its much faster and more accurate than searching based on the entire contents of the file.
So Spotlight combines metadata it generates itself (file content), with basic file metadata (file size, creation date...) and file type specific metadata (image dimensions or song duration).
Then, IIRC, you can save your search and the results will be updated in real time as files are added or deleted.
Parent
Re:Radical (Score:5, Informative)
To my knowledge, other metadata-based search systems have not had a similar degree of extensibility. Please correct me if I'm wrong.
Parent
Re:Radical (Score:5, Informative)
I still have to be convinced that full-content indexing is a good idea. I very rarely need to search for something in the contents of a group of files, and when I do it's usually such a small group that the time saved would not outweigh the disk space used by such large indexes. On the other hand, this problem should get better over time, since the largest files are usually video, and have little indexable content, meaning that the index is likely to get relatively smaller over time (until someone writes a plug-in that can interpret objects in images, and applies this to every frame in a movie. Fortunately, I think this is still a long way off).
Parent
Re:Radical (Score:5, Informative)
* Desktop-metaphor based GUI for a personal computer
* WYSIWYG publishing with a laser printer
* PDAs via Newton
* AppleLink (err, AOL now)
* QuickTime (movies, QTVR, 3D, etc)
We could go on and on. Give Apple props where due, huh?
And please consider modding the troll down...
Parent
Re:Radical (Score:5, Insightful)
You've got to give them credit for product design as well. Nobody makes more desirable-looking software and hardware. Is it any wonder that Apple's fiercest supporters are graphic designers?
Parent
Re:Radical (Score:5, Insightful)
Name one other multimedia framework that has been around as long as Quicktime. And don't mention Video for Windows [wikipedia.org]...I'll take your response off the air.
Parent
Lobbied for WiFi radio spectrum (Score:5, Interesting)
Parent
Re:Radical (Score:5, Informative)
http://developer.apple.com/macosx/tiger/spotlight
Parent
Re:Radical (Score:5, Interesting)
Parent
Re:Radical (Score:5, Insightful)
Parent
Re:Radical (Score:4, Insightful)
The metadata part was noteworthy because MacOS has always had metadata, but Apple looked like it was abandoning, or at least deprecating the concept in OS X. The fact that Spotlight will use it shows that metadata on MacOS still has a future.
Parent
Re:Radical (Score:5, Informative)
Well, Apple *did* deprecate the old file type and creator tags, and resource forks in files that the Mac file system had always had since 1984. There were lot of problems with the metadata in the original MFS, not the least of which was that each file on the Mac was actually two files.
-jcr
Parent
Re:um.. (Score:4, Informative)
Could you send me the source for the version you have installed that does that?
Parent
Re:Sounds like Windows, actually (Score:5, Interesting)
I believe though that the indexing is done during saves, so you'll not notice a general system slow down. What you will notice is a slow down on file saves.
Parent
Re:Sounds like Windows, actually (Score:5, Informative)
Parent
Re:Sounds like Windows, actually (Score:5, Informative)
For one, it doesn't take half an hour, it shows you the results as you type, instantaneously.
Secondly, via plugins it can understand *any* file, such as an image metadata importer that uses OCR so you can search for words, or a Flesh-tone detector so you can search for all your porn that way.
Parent
Re:Sounds like Windows, actually (Score:5, Informative)
Already the differences in Fat32/NTFS versus HFS+ (the mac filesystem) yield significantly faster searches before spotlight is introduced. Sit down on an OSX apple and notice that an entire search of the HD is actually a fast operation, not the waiting many-minute exercise that it is on windows.
Now since spotlight is built into the core of the system, and isn't just a tack-on service like the windows indexer is, there are significant speed advantages, updating the SQL database when files are modified, added, etc is incredibly light on the CPU, and is equivalent to doing something like changing the file name.
What spotlight isn't, and this might be where you are getting confused, spotlight isn't a spider that crawls from folder to folder cataloguing information about each file, which is what the windows indexer was doing, hence why it was resource intensive, as it was busy checking files and folders that you have possibly not made any changes to.
As a counter to the 'Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.' Microsoft, google and apple would disagree. Having an up-to-date catalogue without the CPU strain is a must have, go figure MS have been trying to implement it since NT4.0.
Parent
a problem that doesn't really exist (Score:5, Informative)
Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.
On the contrary, this is a *better* solution to a very basic problem that has plagued computers since they were invented.
The problem
How do I organise and access the data I use every day (emails, letters, images, music etc)?
The old solution
You can put your files in folders (one per file). You can name the files with a short description, ending with a cryptic 3 letter code to denote the file type. Files *must* be in one category/folder only at a time. Limited meta-data (date modified, file-type etc) may be stored.
The new solution
You add meta-data to files (often automatically) saying who created them, what project it's under, whether it's 'to do' or 'unfinished' or whatever. You'd do this in a save dialog for the application, as you saved the file. All other applications which use searchlight will update their view of this stuff for free, in real time.
When you want to work on a project, you click on the live project folder, and immediately you see all the files, emails, images etc for that project, no more, no less, regardless of where they are on the disk and what other projects they're shared with.
Want to see all the stuff to do with John, 5 months ago? On this project? Containing the word gizmo? That sort of query will be easy to make.
If you have an image editing application, it can show you all the images taken in Paris in 2002, without having to build a database application into it. This makes adding this kind of feature to applications trivial.
Ideally adding meta-data tags like 'project-1', and 'To do' should be as easy as choosing them in the save dialog or applying them like a label in the Finder. It's not quite at that stage yet, but that should come later. Some of these ideas are quite old (Be), but they are long overdue in a desktop operating system.
Parent
Re:Sounds like Windows, actually (Score:5, Informative)
Apple has had this type of search engine before, they called it V Twin and it was a basic part of Copland. This is what Sherlock used in Classic and why it was so fast. The idea is even older, it's from a conceptual computer interface Apple dubbed the Knowledge Navigator. All this appears to be is V Twin running on SQLite instead of a proprietary method.
The interesting part to me is the focus on metadata. I loved this feature in BFS that metadata was king. This is going to lead the way to better file management. Hopefully the Finder will integrate it.
Parent
Re:Sounds like Windows, actually (Score:5, Insightful)
Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.
Doesn't exist *for you* perhaps. Perhaps you don't have a lots of user data, or you have taken time to sort it into useful folders. I'd say it's about as useful as the incremental seach in iTunes is. Sure I could remember what artist did a track, and access a track by scrolling down to that artist, then finding the track. Or I could scroll down the list of thousands of track names, remembering my alphabet ordering, and locate the track that way. Assuming I've remembered the exact wording of track name. But I've always found it easier to type whatever word comes to mind first from artist or track into the search box.
And so it is with documents. Even if I do remember the file name and folder that a particular piece of information is stored in, I still need to navigate there. Most times it will be quicker just to type in whatever it is you remember about the data you want into a search box - even if you know where the data is stored.
Parent
Re:Sounds like Windows, actually (Score:5, Informative)
However it only comes with a few configured filetypes settings, and no way to set a default "When no searchFilter available, treat as plain text" setting.
I stressed and strained about this when XP came out initially. The only way I found to do it so I got expected results was to build myself a scanner.
It searched through a drive, and identifies EVERY file extension.
It then looks through the registry to see which Extensions have linked Handlers.
It generates a reg file containing stub links for every unmatched filetype.
Its a bit shotgun, but allowed me to continue using the Text search for XP.
Microsoft have released their own shotgun registry pack, for more info see here:
http://support.microsoft.com/default.aspx?scid=kb
(I have since moved myself into using my own full search tool, but at least the XP search doesn't miss files which are clearly within visible range).
[Now for the science part..]
Take a file, something like "PunchTheMonkey.asp".
Make sure you have it open in notepad, and make sure there is a certain text string - for instance "spyware".
Open the windows XP search in that folder, tell it to search *.ASP, and give it the phrase "spyware".
Windows XP will NOT find this file.
-----
The Windows
[HKEY_CLASSES_ROOT\.txt\PersistentHandler]
@="
Adding an entry like the one above for each required filetype will restore the full text search functionality.
So, I add the following entry into the correct
[HKEY_CLASSES_ROOT\.ASP\PersistentHandler]
@="
After I have logged off/rebooted, I try the same again, and XP will now identify the file.
Parent
Re:What's so special about searching (Score:4, Insightful)
Try to pretend that you're managing 2 or 3 or more major projects that can change or be passed along to someone else every few months with mails, im's, files, reports you don't look at, media submitted by other people in different countries, to-do lists and other project management data...
Now imagine someone asks you, the project manager (or just the last person still around) on a project from 3 years ago, what the initial proposal from that guy in japan who did the Flash files was versus what we paid him and what the VP's said about that....
People *will* have copies of these files still floating around *somewhere* in e-mail or im history, at least. You may not, I may not, but that's where this will come in handy.
A few years ago, hd space was not large enough to think that you'd keep all that data around, but gmail's new 1Gb e-mail storage just showcases the lack of a need to dump all that crap off your media if you can just organize it well, and who needs that when you can keyword search, anyway?
Parent
Re:What's so special about searching (Score:5, Funny)
Parent
Re:What's so special about searching (Score:5, Funny)
Solution?
Save the porn / super personal stuff on an encrypted disk image saved somewhere inconspicuous, and set cronned (or logout) scripts to scrub your various histories and recent items. Make sure that the machine logs you out after no more than 10 minutes of activity.
Hypothetically, that is.
Hi sweety!
Parent
Re:Metadata? In a time like this? (Score:4, Funny)
A disorganized revolution is just a waste of time.
Keeping one's data organized is a priority, bucko.
Parent
Re:Spotlight/ Google Desktop Search/ Win FS ? (Score:5, Interesting)
Spotlight can support arbitrary file types, entirely dependant on what an application developer decides to supply, and you decide to install. Google is limited to the file types Google implements.
WinFS is an overly complicated pile of steaming pooh, that Microsoft are having trouble delivering.
Parent
Re:Geeky question on instant search results (Score:4, Interesting)
The exact method for matching the search string to the lexicon isn't clear. It could be a suffix tree, but it may be as simple as grep-like scanning of the words, since there aren't that many relative to the text size.
Looking at mail.app it seems to do this process on each keystroke. It's not terribly fast, but it gets the job done.
Parent