In a surprising move, Reddit has updated its robots.txt file, blocking Bing and several other search engines from crawling the site. This news comes after Reddit recently secured a licensing deal with Google, allowing them to play hardball with other search engines and AI tools.
On July 1, 2024, Reddit implemented the updated robots.txt file, preventing most search engines from crawling its content. However, it’s important to note that Google is still able to crawl the site, contrary to what some people may have assumed. The decision to block other search engines was made due to Reddit’s inability to reach agreements with them regarding the use of Reddit content, including its use for AI.
Following the implementation of the updated robots.txt file, Bing stopped crawling Reddit. When filtering Reddit results in Bing Search for the last week, users will find nothing. Microsoft confirmed this change, stating that they respect the robots.txt standard and therefore ceased crawling Reddit after the update.
Reddit spokesperson Tim Rathschmidt clarified that this action was not related to their recent partnership with Google. Instead, it was a result of failed discussions with multiple search engines. Some search engines were unable or unwilling to make enforceable promises regarding the use of Reddit content.
This move by Reddit raises questions about the future of large websites and their ability to control access to their content. With Reddit’s successful partnership with Google and its blocking of other search engines, it could set a precedent for other websites to follow suit. This could potentially leave smaller publishers and content producers in a difficult position.
In the near future, users should not expect to see much new Reddit content being surfaced on Bing. The impact of this decision on Bing’s search results and user experience remains to be seen.
Overall, this development highlights the power dynamics between websites, search engines, and AI tools. It showcases Reddit’s ability to assert control over its content and raises concerns about the potential consequences for smaller publishers. As the digital landscape continues to evolve, these actions may have far-reaching implications for the accessibility and visibility of online content.
