photo credit: cote

Spiders and Crawlers and Bots, Oh my!

Two Faced Googlebot

I often hear webmasters speaking about being crawled, spidered by the Infamous Googlebot.  Many seem overly anxious for said “Bot” to make it’s appearance. Dude, I haven’t been crawled or indexed yet, my site hasn’t been spidered in weeks, what did I ever do to

Hey relax the reality is the Googlebot in one form or another will get to your site. Have you submitted a xml site map, is there fresh content on your site. Did your blog issue a ping? You see Googlebot is actually 2 seperate “Bots” used by Google to index the web: Deepbot and Freshbot and they have completely different missions to accomplish for their overlord. Deepbot’s marching orders are to “crawl” every web document on the planet and report back to base with what it has found. That’s a lot of documents and takes a long time about 30 days on average.

Freshbot on the other hand has orders to crawl the web looking for new documents and files. It visits sites that change frequently (like my blog) and aggording to how historically frequent it changes and a webmaster has a few tools available to speed that process up. (Uodate your sitemap, send an RSS Feed, Ping your Blog, laydown a new outbound link to name a few.

Now here is the part I find that most people i talk to do know or understand. The Bots in and of themselves are latently stupid, they just collect data, they than pass that data on to a processor called a parser where Google applies the algorithm and makes decisions about ho, why, where, keywords and such to display or categorize this data deleivered to it by the Bot Boy’s Deep and Fresh. To see when a GoogleBbot last visited your site visit Gbotvisit.

Use Sitemeter SEO to discover what the Bots told the parser about your website. Take a chill pill and do the right things that make life easier for the Bot boys and your website. Deploy content designed for human consumption. Make sure your meta robot values are in place telling the Bot Boys what to brigng back to the parser and include a few suggestions to the parser itself. meta name “robot” “index, follow, archive” these are the defaults Googlebots use but you can use the additional variable where appropriate “nofollow, noarchive” I wouldn’t use these in less your certain of their meaning. I highly recommend these robot values be placed on every page in the header between the  tags.  meta name “robot” “index, follow, archive, noodp, noydir”  The search engine will than be more inclined to tell the parser to use your meta description instead of the DMOZ or the one from Yahoo or the 1st few snippets from the 1st paragraph of your body copy. Than you can write your meta description with a subtle call to action right? Make it intriguing for the SERP visitor to” click on to the other side”. (Your Site)

Remeber I said earlier Deepbot and Freshbot aren’t the brightest bots on the web. Here’s a few tricks that will make them really happy when they return to their buddy “Parser”. When we want to display video on our sites unless you are kicking a screaming “Index me Deepbot, I think your cute Freshbot” it’s going to be hard to get their attention. Because so few webmasters actually know how to do this you can gain a huge advantage using what I’m about to cover to get a huge leg up on the SERP Rankings.

Video (and other rich media) can be displayed on your site in one of three ways. Self Hosted, page embedded, uploaded to video sharing sites like YouTube and Google Video than code embedded on your site.

1. Add Metadata values similar to the id3 tags for mp3 music files. Grab this awesome free software that can display tons of metadata about your video and use that information in metadata tags in the header to describe your video files being displayed, hosted, embedded whatever.

MediaInfo is a software that supplies technical and tag information about a video or audio file. It’s free to use and it’s one of my secret weapons. It’s incredibly robust.

What information can I get from MediaInfo?

· General: title, author, director, album, track number, date, duration…
· Video: codec, aspect, fps, bitrate…
· Audio: codec, sample rate, channels, language, bitrate…
· Text: language of subtitle
· Chapters: count of chapters, list of chapters

and use this data (Bot perfume) to strike a loving relationship with Deep and Fresh and strike Gold in the SERP’s. Tomorrow I’ll give you my secrets to optimizing YouTube Video’s and really drive Deep and Fresh Crazy.

to be continued…

Tags: Bots, crawlers, crawling, Google, googlebot, Metadata, RSS, search, Search engine results page, SEO, spiders, video optimization, Web search engine, Yahoo, YouTube