Most search engines use programs called bots, spiders, or crawlers to download pages from the sites they index, index the content into a database, and follow the links in those pages (if there any) to more pages.

For the last week, all of the sites on our Conversant servers have been hit many thousands of times by the 'bots, as they are apparently in the process of refreshing their content. We support the bots, we believe they're a necessary evil because without them the search engines would be fairly useless, and search engines are one of the primary ways that a web site is found.

Unfortunately, in the last week we've had about 50,000 page requests from the bots, and last night was our busiest ever. That's just a little ridiculous. The servers can handle it, but they're wasting a lot of resources by serving unchanged pages to the bots (which means the bots are wasting a lot of resources also).

Userland Software is having the same problem with their sites. Yesterday I wrote to Dave Winer, president of Userland, and mentioned that we're in the same boat. Specifically, I asked if he had ever seen any interest in any sort of file format which could be used to tell the spiders what has been updated or added to a site, and when, so that they'd know what to download, and what to skip.

(Background: At least a year ago, Dave had tried to push the search engines to accept such a file format. This would be something that goes far behond the Robot Exclusion Standard.)

He didn't write back, but he did post a note on his site this morning which once again invites the search engines to work with UserLand to develop such a file format. Well, here's the official word: Macrobyte would like to be in that loop also, if at all possible.

Most likely, the search engine companies will completely ignore the whole problem, but if they don't then we want to be included in the process.

My friend Dave Bilello is out of the ICU, and all of the tubes have been removed. He still has a ways to go before he'll be throwing elbows on the basketball court again, but the worst is behind him now.

Here's to a quick recovery, Dave!