Google cleans house and why we should all be grateful!

Late many will remember we had an overt competition going on betwixt Yahoo and Google to see which of them could get more of the web indexed. The point buttocks this was presumptively that because they could call upon a wider pick of web sites they could tax return a more appropriate response to any hunt query.

Google, at least, seems to have tired of this. Google CEO Eric Helmut Schmidt was reported as expression that Google's machines were full (New York Times, April 21st), a claim widely considered risible at the time.

It may not have been so far off the mark, although. Let's consider the web from the engines' point of view. Let's face it; there is a great deal of jolly useless web sites out there. It's not like you have to pass suitableness tests to have a web site, and web space is given away free with just about every net account that there is. Neither Google nor the other engines will want to be indexing each and every site that's out there just for the sake of it. Think of the resources that would use! Consider Eric Schmidt's remark in that linguistic context and all of a sudden it takes on a different perspective.

Nor, as occupant on this spinning ball, the third stone from the sun, would we want the engines to be indexing indiscriminately. Run server farms, the arrays of computing machine that every engine stores its indexes on, uses up an awful lot of electricity. There seems little point polluting the environment (and paying big bucks to do it, if you're an engine) just to index all the sites there are on the web that no-one wants to know about anyhow, sites like pointless (read "thin") affiliate sites, content-less Adsense sites, and soon I'm guess all the duplicated book content sites for the same reasons.

You might suggest that Google won't de-index many of these. After all, Google itself makes money from the Adsense income such sites can generate. It must be remembered though that Google makes more money overall from the continued existence of the web as a useful information resource and that environment is being threatened now by all the useless sites that are out there clogging up the indexes. We've seen Adsense sites taken to what might be their logical conclusion where the only way out of a site, once you innocently stumble into it, seems to be by clicking on an Adsense link. People could navigate out using the back button in their browser but I doubt that the majority of surfers would think of that first if at all.

The best way to fight these is probably to de-index them and in turn the sites that link to them. Lacking merit in themselves, any site overtly made purely to generate Adsense income that shows up high in the ranks is very probably linked to by parties owning them or having an association with their owners. It can be speculated that the Page Rank they accrue is falsely generated and so acts like a beacon to Google saying, in effect, "Something spammy going on here!" and by tracing the back-links Google should be able to work out who the link-spammers are. This, I think, is one of the real reasons Page Rank survives in any shape or form.

So, for a better web and a better world, Google are cleaning house. In many ways, it's our house they're cleaning too. We should be grateful.