Spidered Searches

Just curious what people may know about spidered searches by Google? It has been brought to my attention that they and other search engines do it 24/7 and put the results on the web for all to see. I am wondering what guys think of these actions?

By barney290 | 12/11/2011 07:37 AM | 4 comments

Comments are disabled for this blog post.

If you have your own web pages (where you write your own HTML) you can stop a spider with this command in your heading section
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW">

PDQuesnell 12/14/2011 08:21 AM

This is nothing new and it's pretty much what set Google apart from other search engines back in the day. Now every search has their algorithm to do this. There's really nothing wrong with this, and it's good to utilize good SEO (Search Engine Optimization) practices to start owning certain keywords, then you'll start getting a better page rank.

If you have a site, you set what directory's the search engines can crawl, or set them not to crawl your site at all.

So really, having Google crawl your site is a good thing, as far as SEO and traffic is concerned.

bearcub4u 12/11/2011 03:49 PM

This is how search engines have worked since the mid 90s. If you don't want your site indexed you can prevent it using a robots.txt file (see here http://en.wikipedia.org/wiki/Robots_excl usion_standard ).

Wiseass 12/11/2011 03:46 PM

To my knowledge they are not the only ones who perform "spider" searches. Not all of the websites out there place their sites up on various search engines, however doing so will usually get them placed higher in the order of what is returned to the person doing the search. I also think there is a way for the website to not be placed on the search engine if the spider finds it.
I personally do not have an issue with the search engines doing their own search for websites. There is so much crap and misinformation out there it nice to be able to check multiple sites when doing any type of research.

VARickbear 12/11/2011 08:19 AM