Google has recently introduced two new crawlers to its Googlebot family of web crawlers: GoogleOther-Image and GoogleOther-Video. These new crawlers are optimized for fetching image and video bytes and were launched to better support the crawling of binary data that may be used for research and development.
GoogleOther-Image is designed specifically for fetching publicly accessible image URLs. It will go under the user agent tokens of GoogleOther-Image and GoogleOther, with the full user agent string being GoogleOther-Image/1.0. On the other hand, GoogleOther-Video is optimized for fetching publicly accessible video URLs. Its user agent tokens are GoogleOther-Video and GoogleOther, with the full user agent string being GoogleOther-Video/1.0.
Why did Google introduce these new crawlers? According to Google, they were launched to better support the crawling of binary data that may be used for research and development purposes. By optimizing the crawlers for image and video fetching, Google aims to improve its capabilities in analyzing and understanding multimedia content on the web.
It’s important to note that these new crawlers should not cause alarm when you see them in your crawling activities and bot activity on your website and log files. They are legitimate Googlebots and part of Google’s efforts to enhance its crawling capabilities.
In addition to the new crawlers, Google has various other types of crawlers in its Googlebot family. The main crawler is simply called Googlebot and is responsible for crawling the web for Google’s search products. This crawler always respects robots.txt rules, which are instructions given by website owners to specify which parts of their site should or should not be crawled by search engines.
There are also special-case crawlers, such as AdsBot, which perform specific functions and may or may not respect robots.txt rules. These crawlers are designed to support specific features or services offered by Google. Lastly, user-triggered fetchers are tools and product functions where the end-user triggers a fetch. For example, if a user requests a page fetch using the Google Search Console tools, Google will send its crawler to fetch that page.
To conclude, Google’s introduction of GoogleOther-Image and GoogleOther-Video crawlers marks a significant step in enhancing its capabilities in crawling and analyzing multimedia content on the web. These new crawlers are optimized for fetching image and video bytes and are part of Google’s efforts to support research and development activities. As website owners, it’s important to be aware of these new crawlers and not be alarmed when you see them in your crawling activities. They are legitimate Googlebots and reflect Google’s ongoing commitment to improving its search and indexing capabilities.
