065: Optimizing “Crawl Budget” To Improve Your Web Traffic w/Vanessa Fox

While there’s no official school, certification or standards you need to know to be an SEO – I don’t understand why what’s in this episode is NOT required material for all people calling themselves ‘SEOs’.

You see, despite ranking factors always changing, there are still core fundamentals to how search engines discover, crawl and index content. These pillar concepts are not as much the ‘black box’ that ranking factors are. They are fairly known and accepted fundamentals.

Every time I study the idea of crawling I feel like a MUCH better SEO than I was before. And I think for anyone calling themselves an SEO, or really wanting to feel like they understand core concepts – you need to study this stuff too.

Of course, this intro is to compel you into listening to every minute of this episode. Which is not hard for me to sell you because I really believe in its importance.

One hundred percent.

Enjoy!

Don’t forget to subscribe on iTunes!

Links

• Vanessa’s Article On Crawl Budget
• Starting Keylime Toolbox
• Google Webmaster Blog: Crawl Budget
• John Mueller’s Comment In Help Forums
• Changing Googlebot’s crawl rate
• Submitting pages to the index
• The Keylime Toolbox to analyze log files
• Seeking (Slate article on search addiction)
• Vanessa Fox Ignite Keynote

People

• Vanessa Fox on Twitter

Books

• Marketing In The Age Of Google (Vanessa’s Book)

Agenda

Vanessa’s Background

• How Journalism led to SEO
• The REAL reason to not bury the lede
• Singing at Zillow
• What is Key Lime Toolbox?

Crawl Budget + Crawling Factors

• The two tools Vanessa helped launch in Google Webmaster Tools
• Google thinks of URLs, NOT pages
• How Googlebot discovers URLs
• Why discovering an URL is not the same as crawling
• Crawling Signals
• Why you should care!

Crawl Rate Limit

• Why site speed matters
• Server speed vs Client side speed
• How slow is too slow according to Google?
• URL vs Server Errors
• When to use Google’s crawl rate tool
• How Google crawls big sites like NY Times differently

Crawl On Demand

• What is ‘crawl demand’?
• How Google chooses the priority of what pages to crawl

Making Crawl Rate and Demand Actionable

• Why you NEED to analyze log files
• Making URLs discoverable
• The huge problem of infinite URLs (and what types of pages to block from Google)
• What to do when pages in your XML sitemap are not indexed
• How to STOP Google from crawling unimportant content

Bonus Questions

• Moz’s early days
• How Search is actually addictive
• What voice search really is
• Google Home (voice search output)
• The challenge of creating Task Based Search
• Hidden segment: The Google Update That Was Named After Vanessa