<\/p>\n
Google\u2019s newly updated \u201cCrawling infrastructure\u201d documentation includes \u201cOptimize your crawl budget<\/a>,\u201d a section with pointers on how websites can expose more pages to the search giant\u2019s bots. The topic is important, as pages must be crawled to appear in Google\u2019s index and search results.<\/p>\n The documentation addresses the need for a \u201cbudget\u201d given the sheer number of global URLs. A poorly organized website will likely take longer to crawl and thus have fewer indexed pages<\/a>.<\/p>\n Here are takeaways from the new guidelines.<\/p>\n Every new site starts with a \u201cconservative\u201d crawl budget. <\/strong>When launching a site, start with a modest number of pages to ensure that \u201cGoogle can cover all the important content without overloading your servers.\u201d<\/p>\n Over time, the crawl budget increases based on the site\u2019s size and updates. <\/strong>Regular updates (especially if automatically included in its XML sitemaps) encourage the Googlebot to return and likely increase the crawl budget over time.<\/p>\n Update your product inventory and your blog regularly and gradually. For example, instead of launching 100 products in one day, consider three per day for a month, roughly.<\/p>\n Improving your site\u2019s load time may increase its crawl budget. <\/strong>We\u2019ve long known that faster websites are better crawled. Monitor your site\u2019s performance<\/a>, and strive to improve its speed<\/a>.<\/p>\n Popular pages receive more frequent crawls. <\/strong>Googlebot crawls pages with many inbound links (external<\/a> and internal<\/a>) more frequently.<\/p>\n Other ways to improve crawl frequency and speed:<\/p>\nGoogle Guidelines<\/h3>\n
\n
\n
\n