Mac OS X’s “Spotlight” feature, present in Tiger, proposes to search your entire hard drive not just for filenames but for words in the content of those files, and to do it with super speed. The demos are impressive, and it works in many cases, but as a user you should understand that it is not exactly a system-level feature. Rather, it requires some kind of cooperation on the part of your content-editing applications. If the application doesn’t write “spotlight-able” files, they won’t show up.

I found this out this weekend when I was searching for a file on my disk that had an unusual word in it (“maypole”, if you must know). It turned up a couple of email hits, but no documents. I froze in shock. None? I stared at the results in dismay for a few seconds, figuring that I must have deleted the file I had in mind, but I was certain I would not have, which meant that maybe it had been corrupted or destroyed by an errant backup utility, or cosmic ray, or something.

The reason I entertained the thought of such extreme failures is because I imagined that Spotlight was showing me The Truth of my disk, that no characters in any of my documents escaped its search. I had read that Spotlight involved some system shims that updated its database every time anything was written to disk.

But this is not so. Frustrated, I used the Finder to navigate to my Documents directory and look for it, and it was right where I thought it was—but in Mellel format. I opened the doc in Mellel, exported it as RTF—and at once, the new RTF document showed up in the Spotlight search.

Now I could be more understanding of this behavior if Mellel saved its documents in some kind of encrypted (or virtually encrypted) format that included neither ASCII nor Unicode streams of characters. But opening my original file in Emacs showed that indeed the characters were there to find.

A second clue came in the form of the documentation for The Missing Sync. In it, they boast that their Notebook application (which syncs to Palm Memos) has notes that can be searched with Spotlight. If Spotlight worked like I thought it did, this would be like boasting that it had mouse support. The fact that it’s worth mentioning as value which they have added indicates that applications need to somehow provide their content to Spotlight cooperatively, something which is done by Mail, Appleworks, TextEdit, Preview, iCal, and Mark/Space’s Notebook app, and presumably some others.

I tested this theory again by trying to use Spotlight to reveal journal files created by MacJournal. Again, no dice until I told MacJournal to export as RTF. I also tried an experiment wherein I wrote my own program to write out a simple text file containing the word “isopattern” (used only four other places on my disk), and seeing if a Spotlight search would illuminate the new entry. Curiously, if the name of the file was “junk”, it didn’t; but if it was “junk.txt”, it did. Since my tiny program obviously was not explicitly cooperating with Spotlight, I must have been half right. There is something operating at the system level, but it’s only searching for files of some types (read: extensions).

The bottom line, then, is that your data needs to be among the dramatis personae of Apple’s play in order to be illuminated by the spotlight. If you need to crawl through the chaos of your hard drive looking for survivors, you’re going to have to grep through the darkness on your own.