Bing Improves Crawl System Efficiency with Revamp

February 1, 2023

According to a recent study by Microsoft's Bing, most websites have XML sitemaps that include a critical component known as the "lastmod" tag. This tag indicates the last time a webpage linked in the sitemap was modified, and is used by search engines to determine how often to crawl a site and which pages to index. Unfortunately, the study revealed that many of these tags were set incorrectly with identical dates across all sitemaps. Upon further research, it was discovered that web administrators had mistakenly set all dates to the date of sitemap generation rather than content modification.

In response to this issue, Bing is now revamping their crawl scheduling stack [https://www.bing.com/webmaster/help/how-bing-crawls-websites-852e4ff2] to better utilize information from the "lastmod" tag in XML sitemaps. This will improve crawl efficiency by reducing unnecessary crawling of unchanged content whilst prioritizing recently updated content for indexing purposes.

Microsoft has announced improvements to sitemap.org in June, including an update to the “lastmod” tag. This tag is used to indicate the last time a page was modified and should be set in W3C Datetime format (YYYY-MM-DD or YYYY-MM-DDThh:mm:ssTZD). Additionally, Microsoft has added a note that this date should reflect the page’s last modification and not when the sitemap is generated. An example code snippet of how this should be implemented can be seen here: http://www.sitemaps.org/schemas/sitemap/0.9 .

Google has advised that only significant changes to a page should trigger an update of the “lastmod” tag, so as not to impact search engine ranking algorithms by overusing it on pages with minor updates or no changes at all.

Both Microsoft and Google have recently been emphasizing the importance of the “lastmod” tag in sitemaps, a powerful tool for ensuring that webpages are crawled and indexed more effectively. Google Search Advocate John Mueller recently discussed the lastmod tag in the January edition of Google’s office-hours Q&A sessions, encouraging its use for substantial modifications. Microsoft has also released a study emphasizing its importance and how to best utilize it. It is worth noting, however, that changing the date in the lastmod tag after minor edits may be viewed as an attempt to manipulate search snippets.

To ensure their pages are correctly indexed and accessible by search engines, publishers should regularly update their sitemaps and lastmod tags.

Microsoft recently discussed the importance of setting the "lastmod" tag in your Sitemap.xml file to ensure that your website is properly indexed and updated on search engine results pages (SERPs). The "lastmod" tag indicates when content on a website was last modified, which helps search engines provide users with more accurate, up-to-date information. By setting the "lastmod" tag, webmasters can improve their website's visibility to potential customers and ensure that their content is always fresh and relevant. Additionally, Microsoft pointed out that it is crucial for webmasters to be proactive about updating their Sitemap.xml files as well as any other changes made to a website in order for them to be reflected in SERPs.

By keeping their Sitemaps up-to-date and setting the "lastmod" tag correctly, website owners can ensure that potential customers are able to easily find the most current version of their content. This article from Microsoft provides an overview of why setting this specific tag is important as well as instructions on how to set it correctly.

Source: Microsoft{rel="nofollow"

Google Enhances Search Console Video Indexing Report
Google has released an update to the Search Console Video Indexing Report, designed to help publishers and content creators gain more comprehensive insights into how their videos are performing in sea...
Read More
OpenAI Introduces ChatGPT Plus Subscription Plan Starting at $20 Per Month
OpenAI has announced the launch of ChatGPT Plus, a premium version of their popular ChatGPT chatbot. This service aims to provide subscribers with a faster response time, priority access to new featur...
Read More