Reddit is considering blocking search engine crawlers like Google and Bing to leverage its data for lucrative licensing deals with AI companies, according to a Washington Post report. This could remove Reddit content from search results.
Initially, Reddit denied the Post’s claims it may require login to view content. But after clarifications, Reddit admitted it may restrict crawlers if unable to strike data deals, while maintaining free public access.
Without crawler access, Reddit posts wouldn’t appear in search engines. This happened temporarily during API protest blackouts, showing one potential impact.
Reddit wants AI firms to pay for using its data to train models. Its prior API pricing changes aimed to spur licensing deals with AI companies and app developers.
The Post report highlights the trend of news organizations blocking scrapers from AI training. Twitter also blamed data scraping for its API changes under Elon Musk.
Losing Reddit’s trove of communities could degrade search results. But for Reddit, lucrative AI licensing contracts may provide more economic value than indirect search visibility.
However, blocking search crawler access risks public backlash given Reddit’s open ethos. The company may be underestimating the reliance of its users and culture on search discovery.
For now, Reddit’s crawler plans remain unclear. But leveraging its platform data for revenue is an escalating trend. How Reddit balances financial incentives with user expectations will shape its future.
