Cloudflare’s Explanation of the Tuesday Outage: How ChatGPT Was Temporarily Taken Down

Cloudflare offers bot controls to address issues such as web crawlers scraping data for generative AI training. Recently, they introduced a new system utilizing Generative AI to create the “AI Labyrinth,” a unique approach that uses AI-generated content to confuse and slow down AI crawlers and non-compliant bots.

However, the recent problems experienced by Cloudflare were not related to generative AI technology, DNS, or suspected cyberattacks. Instead, they stemmed from changes in the permissions system of a database.

According to the CEO, the machine learning model supporting Bot Management had an updated configuration file to identify automated requests. A modification in the ClickHouse query behavior inadvertently led to the generation of duplicate ‘feature’ rows, causing issues.

The changes in the query triggered the ClickHouse database to produce duplicate information, exceeding memory limits and impacting the core proxy system that processes customer traffic. This resulted in false positives for companies blocking specific bots and disruptions for those not utilizing the bot scores in their rules.