“Google Not Welcome at Editthispage.com”

Yesterday Duncan posted a message on his Jotter which mentioned that Google was no longer indexing sites hosted by editthispage.com. I thought that was interesting, so I followed his links to Andrea's and David's sites, to learn more.

Other sites mentioned the problem too. There was a lot of speculation about what might have happened. Some thought it was because Userland's servers were down the last time Google tried to index them, some thought that Google didn't like Userland and had removed all Userland-hosted sites from their index.

Knowing that Userland has load problems when the indexers start crawling, I thought it more likely that they'd simply shut them out.

I was right. Masquerading as Google's indexer, I requested a page from an editthispage.com site, and here's the response they sent back:

<H1>Inktomi, your crawler is repeatedly hitting our servers getting the same WAP files over and over. Please stop pounding us, it's hurting the service we provide to our customers. Thanks. webmaster@userland.com. </H1>

So, I reported this on Andrea's site, just to answer her question. Then I put the URL on David's site, so that he'd have the answer too.

Today, Dave Winer flamed me a little for my comments. The "contradiction" he refers to is that Brent Simmons (who works for Userland) said that Google is allowed to index the sites, but I said otherwise. At this moment, they're still not allowed.

Dave's now saying on Scripting News that he has no idea why Google isn't indexing their sites, in spite of the fact that I sent him the proof (directly to his email) that they weren't allowing Google to index them! Just in case he changes his site, here's what it says at this moment:

Apparently Google decided for some reason to stop indexing many of the free sites we host. We are just as clueless as to the reason for this as anyone else.

He's right. I'm not clueless about it, and he's just as clueless as I am.

Draw your own conclusions as to WHY he's saying this to his customers.