As AI scraping surges, AI search traffic fails to follow: Report

AI search engines contribute little to publishers' traffic, which in turn has intensified web crawling behavior. This is an important finding in the recent report of TollBit, a content monetization platform.

Click-through rate comparison: The report shows that the average click-through rate of Google search is 8.63%. However, the click-through rate of AI search engines is only 0.74%, while the click-through rate of AI chatbots is even lower, only 0.33%. This means that AI search brings a 91% reduction in recommended traffic than traditional searches, while chatbots bring a 96% reduction in traffic.

Important: This is bad news for publishers because it shows that AI search does not replace traditional search traffic. This trend is expected to continue as AI-generated answers replace direct access to the website.

Data Interpretation: From the third quarter to the fourth quarter of 2024, AI robot crawling behavior more than doubled (117%). also:

• In the fourth quarter, each website was crawled by AI robots on average, and another 1.89 million were from hidden AI crawlers.
• The number of AI robots that ignored robots.txt in the fourth quarter increased by 40% compared with the third quarter.
• ChatGPT-User robot activity surged by 6767.60%, becoming the most aggressive gripper.
• The AI ​​robot with the highest proportion of crawling activities:
  • ChatGPT-User (15.6%)
  • Bytespider (ByteDance/TikTok) (12.44%)
  • Meta-ExternalAgent (11.34%)
• PerplexityBot continues to send recommended traffic to sites that explicitly block its access, raising concerns about its unpublic crawling behavior.

Related Background: Chegg is trying to sue Google for AI overview features. Chegg claims Google's search capabilities have seriously damaged its traffic and revenue.

• Google announced last week that it expanded AI overview capabilities. Now, even users who are not logged in can see an AI overview.

Data Source Description: The report does not provide a methodological part, so it is not clear how many websites were analyzed, and only mentioned that the data is based on "all websites that have joined the ToolBit platform in the fourth quarter of 2024". Toolbit says it "helps over 500 publisher websites."

Full text of the report: TollBit Robot Status Report – Fourth Quarter 2024 (Registration required)

The above is the detailed content of As AI scraping surges, AI search traffic fails to follow: Report. For more information, please follow other related articles on the PHP Chinese website!