Details

Written by: IT Pro

Category: Blog

Published: 22 November 2025

Hits: 5055

Introduction

Modern computing runs on silicon—and GPUs have become the new gold. Whether for gaming, AI research, VFX, 3D rendering, crypto-mining, or data-center operations, demand for powerful graphics processors has exploded in the past several years. The result has been a prolonged, global GPU shortage that has affected everyone from individual consumers to hyperscale cloud providers.

What began as a supply disruption has evolved into a complex, multi-layered global crisis involving advanced semiconductor manufacturing bottlenecks, geopolitical constraints, massive AI investment, gaming demand, soaring cloud consumption, and technology transitions.

This article breaks down why global GPU scarcity persists, why new chips remain expensive, and—most importantly—when (and if) this shortage will finally end.

---

1. Why GPUs Are Different From Other Chips

GPUs are not CPUs.

They require:

• more transistors per mm²

• more advanced lithography (down to 3nm / 5nm)

• high-bandwidth memory integration (HBM)

• advanced packaging (CoWoS, EMIB, 3D-stacking)

• extremely low defect tolerance

• specialized fabrication lines

• limited global suppliers

This means:

• GPU production cannot simply be “scaled up”

• new factories cannot be switched on overnight

• only a handful of companies can make them at all

95%+ of bleeding-edge GPU production is dependent on TSMC, the Taiwanese semiconductor giant.

That is a single point of global failure.

---

2. What Triggered the Shortage? (Multiple Waves)

The GPU shortage is not one event—it's an overlapping series of waves:

Wave 1 — Pandemic Supply Disruption (2020-2021)

Factories closed.
Shipping froze.
Demand spiked.

Result: zero inventory at launch for most consumer GPUs.

Wave 2 — Crypto Mining Frenzy

Ethereum mining sent GPU demand through the roof.

Gamers competed with industrial-scale mining farms.

Prices shot up 200%–400%.

Wave 3 — Cloud Computing Explosion

Hyperscalers expanded GPU capacity for AI dramatically:

• AWS

• Google Cloud

• Microsoft Azure

• Oracle Cloud

• Tencent Cloud

• Alibaba Cloud

Every hyperscaler ordered millions of units.

Wave 4 — AI Gold Rush (2023-2025)

The rise of:

• ChatGPT

• GPT-4 family

• Llama models

• Stable Diffusion

• MidJourney

• AI training everywhere

turned GPUs into strategic infrastructure.

Corporations, governments, and defense contractors entered the bidding war.

Wave 5 — Semiconductor Packaging Bottleneck

CoWoS packaging bottleneck delayed shipments by months.

It does not matter if a GPU die is ready—if it cannot be bonded with HBM, it is unusable.

---

3. Why AI Is the Main Driver Now

This is crucial:

AI is the #1 consumer of high-end GPUs today.

Generative AI requires:

• billions-scale training parameters

• continuous inference workloads

• enormous parallel computation capability

• high-bandwidth memory throughput

Training a frontier-tier model can require tens of thousands of H100/H200 class GPUs—and that’s for a single model.

Then, inference (ongoing use) consumes even more hardware over time.

Demand has gone from thousands → hundreds of thousands → millions of units globally.

No manufacturing industry can absorb that shock instantly.

---

4. NVIDIA Dominance = Market Bottleneck

NVIDIA controls:

• 80–90% of the global AI GPU market

• nearly all hyperscale training hardware

• CUDA ecosystem lock-in

GPU quantity is limited.
GPU alternatives are limited.
GPU switching costs are enormous.

Companies have no choice but to wait and pay.

---

5. Why Consumer & Gaming GPUs Remain Expensive

You would think consumer GPUs would be cheap by now.

However:

1. Manufacturing prioritizes data-center GPUs

(H100, GH200, B200 etc.)

because…
profit margin per chip:
$2000+ → $30,000+

vs
consumer card:
$200 → $1600

Manufacturers prefer the profitable chips.

2. Gaming demand remains high

New AAA titles require more power.

3. Used market is dry

Mining collapse flooded supply once—but that supply is now gone.

4. AI hobbyists are now competing with gamers

More competition → higher pricing.

---

6. Supply Bottlenecks Explained

The biggest constraints today:

• Lithography

Only TSMC, Samsung, and Intel can build advanced nodes.

• Packaging capacity

CoWoS is limited and complex.

• HBM production

Only a few vendors supply:

• SK Hynix

• Samsung

• Micron

and yield rates are low.

• Inventory depletion

no warehouse stock exists anymore.

• Shipping logistics

hardware travels through dozens of steps:
fab → packaging → memory → board assembly → testing → validation → distribution

---

7. Geopolitical Risk Amplifies Everything

GPU production depends massively on Taiwan.

Risk factors include:

• China–Taiwan tensions

• U.S. export controls

• sanctions

• trade restrictions

• chip embargo policies

The U.S. controls access to AI chips for China.
China is now stockpiling aggressively.
This drives additional scarcity.

---

8. When Will the GPU Shortage Actually End?

Short answer:

Not soon.

Realistic timeline considerations:

2025

• supply constraints loosen slightly

• new fabs begin limited ramp

• more HBM availability

• but AI demand increasing faster than supply

2026

• additional packaging lines completed

• some regions see price stabilization

• corporate backlog decreases

2027+

• next-gen fabs come online

• global supply significantly expands

• shortage meaningfully declines

Most analysts project meaningful normalization between 2026–2028.

Not in 2025.

Certainly not in 2024-2025.

---

9. Will GPU Prices Drop?

They will, but slowly—because:

• corporations will still pay premiums

• high margins are now normal

• AI demand won't collapse

• gaming cycles continue

• annual tech refreshes are accelerating

Price collapse only occurs when:

supply > demand

We are far from that.

---

10. Could Another Shortage Happen Again?

Yes—and easily.

Top risk triggers:

• conflict in Taiwan

• AI arms race escalation

• export bans

• HBM shortage

• logistic collapse

• new mining boom

• supply chain cyber-attack

Semiconductor fragility remains extremely high.

---

Conclusion

The global GPU shortage is not a temporary inconvenience—it is the result of a structural imbalance that has reshaped the computing industry.

For the first time in history:

GPUs are more strategically important than CPUs.

Demand from AI, cloud computing, gaming, and industrial simulation has outgrown the world’s manufacturing ability to supply advanced graphics processors. This shortage will likely continue into the second half of the decade, easing only as new fabs, packaging plants, and memory facilities mature and stabilize globally.

Will the shortage end?

Yes.

But not this year.

Not next year.

We are on a multi-year timeline—and the world's AI appetite is still accelerating.

Until production finally outpaces demand, GPUs will remain one of the most precious—and expensive—assets in the technology world.

Details

Written by: IT Pro

Category: Blog

Published: 22 November 2025

Hits: 5715

Introduction

In 2025, the massive surge in investment into AI-specific data centre infrastructure is unmistakable. From billions in capital commitments by tech giants to sovereign funds aggressively backing new facilities, the world’s digital economy is pivoting into what might be called the “AI compute arms-race.” Below, we explore the major forces driving companies to pour billions into AI-data-centres, the architectural and operational changes underpinning the shift, how business models are adapting, and what the risks and future implications are for organisations like yours (with deep interest in infrastructure, benchmarking, compute off-loading, etc.).

---

The scale of the investment

To grasp the momentum, here are some representative data points:

• Microsoft plans approximately US$80 billion in fiscal 2025 to build AI-enabled data centres, particularly in the United States. Reuters

• The global data-centre investment boom tied to AI is estimated in the trillions: one article noted “a $3 trillion AI data-centre spending boom” underway. The Guardian

• According to a 2025 review of data-centre investors, firms such as Blackstone, Bain Capital, and others were actively deploying capital into large-scale hyperscale and GPU-rich facilities. STL Partners

These numbers reflect that this isn’t incremental growth — this is a strategic, large-scale shift in infrastructure.

---

Why now? — Key drivers

1. Explosion of AI model complexity & demand

The rise of large language models (LLMs), generative-AI systems, simulation workloads and other compute-heavy tasks has fundamentally changed the demand profile of data centres:

• Training and inference at scale require massive GPU clusters, high-density racks, advanced networking and cooling.

• As one article describes: “Every extra token generated by AI algorithms depends on this layer.” Gainify

• Companies are shifting from traditional CPU-centric workloads to GPU/ASIC-accelerated ones, which drives new architectural requirements (power density, cooling, connectivity).

In short: the compute demand is growing both horizontally (more models/users) and vertically (larger models, more parameters, more data).

2. Competitive advantage & first-mover investments

For many large tech firms and cloud providers the race is about more than just cost-efficient computing: it's about building the infrastructure moat:

• Firms like Microsoft, Amazon AWS, Google Cloud and Meta are not content to simply “rent” infrastructure—they are building their own next-gen facilities to gain operational, latency, cost and control advantages. 174 Power Global+1

• For enterprises (including your own context of benchmarking, GPU off-load, virtualization etc), having access to specialized infrastructure gives a differentiator: faster model iteration, lower latency inference, higher throughput training.

Hence, companies are willing to commit “billions” now to lock in that future value.

3. Infrastructure as strategic asset

Data-centres are no longer just static “hosting” assets—they are strategic infrastructure for AI:

• They represent long-lived assets (10+ years) and are increasingly treated like critical industrial infrastructure (power, cooling, fibre, renewable energy).

• Investors and infrastructure funds are moving in: the list of “top data-centre investors” now includes infrastructure/real-asset firms seeing data centres as core growth platforms. STL Partners

• The nature of AI compute means that what matters is not just “more servers” but “right servers in the right place” (with efficient power, low latency, high bandwidth).

Thus, for companies, building the right AI-data-centre often means building the future of their business.

4. Energy, location and scaling economics

Large-scale AI data centres are power-intensive, heat-intensive, space-intensive, and benefit from economies of scale:

• One technical paper shows how co-locating AI data centres with renewable generation and smart energy-management systems can significantly reduce cost and environmental impact. arXiv

• Another shows how distributed, grid-aware data centres could help stabilise grids while absorbing massive compute loads. arXiv

• Strategic location, access to cheap/renewable power, favourable grid policy, land & permits all matter. Companies trying to build AI-centrically are factoring in not just compute cost but “compute + energy + cooling + real estate + connectivity” cost.

5. Sovereignty, regulation & geostrategic concerns

Compute matters not only commercially but politically:

• A recent study of 775 non-US data-centres found that control of data-centre infrastructure (which nation, which operator) is increasingly a lever of digital sovereignty. arXiv

• Some nations are explicitly trying to attract AI data-centre investments to capture downstream AI value domestically.

• Firms, beyond latency/cost, are thinking of risk: regulatory risk, export-controls, supply-chain constraints—all of which push towards owning or tightly controlling infrastructure.

---

What does “AI-ready data centre” mean – key architectural shifts

Building data centres for AI workloads is materially different than traditional enterprise or cloud-hosting data centres. Some of the key differences:

• Power density: AI racks may require tens of kilowatts (kW) per rack rather than a few. Cooling and power distribution must support this.

• Cooling systems: Liquid cooling, direct-to-chip cooling, immersion cooling are now becoming more common for dense GPU clusters.

• Connectivity & latency: Large GPU clusters often require very fast interconnects (NVLink, CXL, PCIe, high-speed Ethernet) and low-latency links to storage, network, edge services.

• Modular design & rapid deployment: Some newer operators are designing modular “GPU-pods” or containerised data-centres so that they can deploy large capacity rapidly.

• Energy and sustainability infrastructure: Because power is expensive and increasingly scrutinised, many facilities are co-locating renewables, using smart load-shifting, building in sites with cheap power, or negotiating large-scale power deals.

• Specialised hardware lifecycle: Unlike typical servers, AI clusters hinge on GPU/accelerator refresh cycles (e.g., every ~18-24 months), meaning infrastructure must support upgrades, cooling, high-density power loads.

• Location strategy: Proximity to AI model research hubs, data sources, user endpoints, and connectivity to cloud/hybrid setup matter.

For anyone in your field (AI benchmarking, heavy GPU usage, virtualization, etc.), the takeaway is: infrastructure is now a primary differentiator, not just a cost.

---

Business model implications — Why companies are investing

From a business-perspective, the logic of investing heavily in AI-data-centre infrastructure falls into several buckets:

• Enabling new revenue streams

Companies see the transition to AI as creating new business lines: model training, inference-as-a-service, enterprise AI consulting, edge AI deployments. To support them, you need the infrastructure. Without it, you risk being dependent on third-parties.

• Cost control and margin improvement

By owning or controlling infrastructure optimized for AI workloads, companies aim to reduce operational costs per inference or training hour. For hyperscalers, economy of scale can push down cost enough to enable new services with attractive margins.

• Strategic advantage and lock-in

Infrastructure investments create moats: once an organization owns or controls significant AI compute capacity, it becomes harder for competitors to match. Also, integration with proprietary hardware, software stacks, custom cooling, etc., increases switching costs.

• Supporting internal innovation

In your world of GPU-offload, AI benchmarking, virtualization, tools development: having access to large compute facilities enables faster iteration, larger experiments, and internal competitive advantage. It’s a productivity investment, not just infrastructure.

• Infrastructure as service for others

Some companies are building AI-data centres to serve their own needs and offer capacity to others (e.g., AI start-ups, SaaS companies). This dual-model allows monetisation of excess capacity.

• Risk hedging and control

As AI becomes central to business models, reliance on external suppliers or cloud only may become a bottleneck or risk (latency, data-sovereignty, cost inflation). Investing in infrastructure is a hedge.

---

Regional & industry dynamics

• The investment boom is global: Asia-Pacific, Europe, Middle East all seeking AI-compute campuses. For example, France announced major investment to get “back in the race” with dedicated AI-supercomputing/data-centre campuses. Le Monde.fr

• Emerging markets may become attractive because of land, power or regulatory advantages (particularly for energy-intensive AI infrastructure).

• Industries outside pure tech are also involved: financial services, automotive, healthcare, manufacturing are increasingly investing in internal AI infrastructure and thereby fueling demand for “AI data-centres”.

---

Key challenges & risks

While the rationale is strong, these investments are not without significant risk and complexity:

• High capital intensity: These are multi‐billion-dollar commitments with long horizons before payback.

• Rapid technological change: The hardware, cooling, networking landscape for AI evolves fast; investment in today’s architecture may become sub-optimal in a few years (e.g., new generation of GPUs, new memory/architecture, optical interconnects).

• Energy & sustainability pressures: As AI compute grows, so does energy consumption and carbon footprint. Regulators, communities and companies are under pressure to ensure sustainability. Papers show how renewable‐co‐located data centres can help—but they also add complexity. arXiv

• Grid and power constraints: Many regions struggle to provide the necessary power or reliable connectivity, or may face permitting/power-contract delays.

• Geopolitical/regulatory risk: Infrastructure may become subject to export controls, data sovereignty laws, government intervention. Papers studying non-U.S. data centres show that operators’ nationality and control matters. arXiv

• Demand uncertainty: While demand for AI is growing, the exact shape, timing and business model of future workloads is still uncertain. There is a risk of overcapacity or wasted spend if demand evolves differently.

• Cooling/thermal risk: As rack densities escalate, cooling management becomes non-trivial (risk of failure, heat mitigation, cost escalations).

• Return on investment (ROI) pressure: Investors (infrastructure funds, REITs, etc) are assessing what the revenue model of AI-data-centres will be, beyond “just hosting.”

---

What this means (and what you should consider)

Given your interest in GPU benchmarking, AI workflows, virtualization and infrastructure, here are some actionable implications and considerations:

 Plan for higher compute-capability access

• If you’re developing AI benchmarking suites or off-load strategies (GPU/CPU/DirectML/ONNX etc.), anticipate that large organisations will increasingly have in-house or outsourced access to “AI-ready” clusters.

• If you rely only on commodity cloud/virtualization, you may find cost/performance sub-optimal compared with organisations that have custom AI data-centres.

 Infrastructure strategy should evolve

• Consider where to run your workloads: internal cluster vs. third-party vs. hyperscale AI-data-centre.

• Evaluate whether your benchmarking or provisioning tools are adapted to the new “dense GPU cluster” paradigm (e.g., high-bandwidth interconnect, direct-to-chip cooling, rack > 50 kW).

• Think about scalability, energy cost, cooling and power infrastructure as part of your stack (not just compute).

Sustainability and energy should be part of planning

• As compute loads rise, so will energy/cooling costs. Building or using AI infrastructure in efficient locations with renewable energy access may substantially affect TCO and scheduling.

• If you benchmark systems, include energy-per-token or energy-per-inference metrics.

Vendor and hardware ecosystems matter

• The component supply-chain (GPUs, ASICs, interconnects, memory) is increasingly tied to large-scale data-centre deployments. That means the infrastructure you benchmark or develop for will evolve rapidly and may depend on partnerships or scale.

• Access to next-gen AI hardware (e.g., GPUs designed for data-centre scale, custom ASICs, CXL interconnect, liquid cooling) might be a differentiator.

Risk-mitigation strategy

• Because investment cycles are large and long, consider diversification (hybrid cloud + on-prem + edge) rather than assuming all compute will migrate to “AI-data-centres”.

• Monitor regulatory/sovereignty risks around where data centres are located or how they’re operated.

• Be aware of possible overcapacity scenarios which might drive down margins for data-centre operators (which could impact availability, pricing).

Benchmarking & tooling opportunity

• Your interest in AI-Benchmark suites, GPU off-load and virtualization could align with the emerging trend of “AI-data-centre” architecture. There will be opportunity in benchmarking new architectures, comparing on-prem vs. cloud vs. AI-dedicated data-centres, modelling energy/cost/throughput trade-offs.

• Consider building modules/tools that help enterprises evaluate when building their own AI-data-centre makes sense vs. leasing capacity from hyperscale operators.

---

Looking ahead: What to watch for

Here are some forward-looking themes that companies and benchmarkers (like you) should monitor:

• Architectural leaps: The next generation of AI hardware (e.g., more efficient GPUs, custom accelerators, chiplets, memory disaggregation) will influence what “AI-data-centre” means in 2026-27.

• Edge AI data centres: While much investment is for hyperscale campuses, edge-AI (closer to users) may drive mini-data-centres for low-latency inference.

• Energy and cooling innovation: Immersion cooling, liquid cooling, renewable co-location, smart load scheduling will become increasingly important as power becomes the limiting factor.

• Sovereign compute and regional hubs: More governments may incentivise local AI-data-centre development for sovereignty/privacy reasons. This could open new markets and regulatory pushes.

• Business model evolution: “Compute-as-a-service” models for AI may grow: enterprises buying custom clusters for AI training/inference, rather than renting generic cloud capacity.

• Sustainability & carbon footprint: As AI compute grows, public and regulatory scrutiny around energy, emissions and sustainability will increase — data-centre operators will need to measure and optimise energy/performance metrics.

• Risk of overbuilding: As with any infrastructure boom, the risk of “too many racks chasing not yet-mature workloads” is real. The timing of demand vs. capacity will matter.

---

Conclusion

The flood of investment into AI-data-centres in 2025 is not simply a continuation of cloud growth—it’s a structural shift in how computing infrastructure is built, deployed, and monetised. For companies, the decision to pour billions into AI-data-centre capacity is driven by:

• The sheer scale and velocity of AI workloads.

• The strategic imperative to own the infrastructure (or have preferential access) that powers AI.

• The economics of scale, energy and performance which favour large-scale specialised facilities.

• The evolving notion of data-centres as strategic, competitive assets rather than just “server farms.”

Details

Written by: IT Pro

Category: Blog

Published: 19 November 2025

Hits: 6633

On November 18, 2025, a huge slice of the internet fell over.
If you opened ChatGPT, X (Twitter), League of Legends, Shopify, Coinbase, or countless smaller sites, you were greeted with a Cloudflare-branded 5xx error page—or the sites just wouldn’t load at all. What looked at first like yet another big “the internet is broken” moment turned out to be something more subtle and, in some ways, more worrying: a self-inflicted bug deep inside Cloudflare’s own infrastructure.

Below is a detailed walkthrough of what happened in yesterday’s Cloudflare outage (18 November 2025), why it happened, who it affected, and what lessons infrastructure teams should take away from it.

 

---

What actually happened yesterday?

On Tuesday, November 18, 2025, around late morning UTC, Cloudflare began returning large volumes of HTTP 5xx server errors for traffic that passed through its network. For end users, that meant “Internal Server Error” or “Gateway Error” pages when trying to access many popular websites and apps.

According to Cloudflare’s own post-incident blog, the outage:

• Started impacting customer HTTP traffic at 11:28 UTC

• Saw widespread 5xx errors across core CDN and security services

• Had major mitigation steps around 13:05–14:30 UTC

• Returned 5xx error volume to baseline by 17:06 UTC The Cloudflare Blog

Cloudflare itself described it as its worst outage since 2019, because it didn’t just affect one feature or dashboard – it disrupted the core proxy layer that routes the majority of customer traffic through its network. The Cloudflare Blog

Third-party monitoring backed this up. Cisco ThousandEyes saw a global outage affecting Cloudflare, with timeouts and 5xx errors on services like X, OpenAI (ChatGPT), and Anthropic, while network paths themselves looked healthy. That pointed strongly to a backend service failure, not an ISP-level or routing issue. ThousandEyes

 

---

Who was affected?

Because Cloudflare sits in front of a massive portion of the internet (around 20% of the web’s sites rely on Cloudflare for performance and security), the blast radius was enormous. AP News+1

Among the services reported as impacted:

• ChatGPT / OpenAI

• X (formerly Twitter)

• Canva, Shopify, Dropbox, Coinbase

• League of Legends and other gaming platforms

• Various public transit and government sites, including New Jersey Transit and France’s SNCF railway digital systems AP News+1

Outage trackers like Downdetector recorded thousands of concurrent issue reports at the peak. Reuters reported about 5,000 affected users for X alone at one point, before counts declined as fixes rolled out. Reuters

From a user’s perspective, this manifested as:

• Sites not loading at all

• Login flows hanging or failing (especially where Cloudflare Access or Turnstile were involved)

• APIs responding intermittently or with 5xx errors

• Dashboards and admin panels timing out

In other words: huge parts of the internet “felt down”, even though the root cause was concentrated in a single provider’s internal systems.

 

---

How Cloudflare normally works (in simple terms)

To understand why this outage was so severe, it helps to know the rough path of a request through Cloudflare’s network.

Cloudflare acts as a reverse proxy CDN and security layer:

1. Your browser or app connects to Cloudflare instead of directly to the origin site.

2. Cloudflare terminates TLS and HTTP at its edge.

3. Requests flow into Cloudflare’s core proxy system, called FL (“Frontline”) and its newer generation FL2.

4. That core proxy:

   • Applies WAF (web application firewall) rules

   • Runs Bot Management models

   • Handles DDoS protection, caching, egress to origin

   • Routes traffic to other internal products like Workers, R2, Access, etc. The Cloudflare Blog

In normal operation this architecture is highly resilient: if one data center has a problem, traffic is routed through others; configuration changes are rolled out carefully; individual features should fail in contained ways.

Yesterday’s outage was precisely bad because the failure was inside the common proxy path itself, and it was tightly coupled with a configuration file that gets pushed worldwide frequently and automatically.

 

 

---

The root cause: a bot-management feature file gone rogue

Cloudflare’s official explanation points to one key culprit:
a feature configuration file used by their Bot Management system. The Cloudflare Blog

Here’s the chain of events in plain language:

1. Bot Management uses a “feature file”

   • Cloudflare’s bot-detection model relies on a set of “features” – signals about each request used to decide if it’s human or a bot.

   • These features are bundled into a configuration file that is regenerated every few minutes and rolled out globally, so Cloudflare can adapt quickly to new attack patterns. The Cloudflare Blog

2. A change in ClickHouse query behavior

   • The feature file is generated by queries against a ClickHouse database.

   • Cloudflare made a change around 11:05 UTC to improve security and permissions for distributed queries – allowing users to see metadata not just from a default schema but also from underlying r0 tables. The Cloudflare Blog

   • The query that builds the feature list didn’t filter by database name; suddenly it started getting duplicate columns from both default and r0, effectively doubling the number of feature rows.

3. The feature file exploded in size

   • The Bot Management module has a hard limit on how many features it will accept (set to 200, well above the ~60 normally in use).

   • When the newly generated file exceeded that limit, the module hit the cap and panicked, due to an unhandled error in Rust code that used Result::unwrap() on an error value. The Cloudflare Blog

4. Core proxy services started returning 5xx errors

   • Because Bot Management is integrated into the core proxy path, the panic surfaced as HTTP 5xx responses for any traffic that depended on that module.

   • On the new FL2 engine, customers saw explicit 5xx errors.

   • On the older FL engine, bot scores silently went to zero, which could cause false positives in bot-blocking rules. The Cloudflare Blog

5. The really nasty part: the file kept flipping between “good” and “bad”

   • The ClickHouse cluster was being gradually updated, and the feature file was regenerated every five minutes.

   • Sometimes the query ran on updated nodes (producing a bad file), sometimes on non-updated nodes (producing a good file).

   • That meant for a while, Cloudflare’s network oscillated between normal operation and failure as different versions of the file were propagated. The Cloudflare Blog

This oscillation made the situation extremely confusing internally. At first, Cloudflare’s teams suspected a massive DDoS attack because the error pattern didn’t look like a simple software crash. Even the Cloudflare status page, which is hosted outside their own infrastructure, briefly showed errors – a coincidence that further fueled the suspicion of an external attack. The Cloudflare Blog+1

Only once they realized the common factor was the bot feature file did the picture become clear.

 

 

---

Timeline of the incident

Based on Cloudflare’s postmortem and third-party reports, we can piece together a rough timeline for November 18, 2025: The Cloudflare Blog+2ThousandEyes+2

• 11:05 UTC – A database access control change is deployed in ClickHouse.

• 11:20–11:30 UTC – Bad versions of the Bot Management feature file begin being generated and propagated.

• 11:28 UTC – First customer impact: elevated HTTP 5xx errors seen on customer traffic.

• 11:30–11:32 UTC – External monitoring tools and automated tests start detecting intermittent failures.

• 11:35 UTC – Cloudflare opens an internal incident call; investigation begins.

• ~11:48 UTC – Cloudflare publishes a status update confirming an incident. Resend

• 11:30–13:05 UTC – Teams focus on what appears to be degraded Workers KV behavior and investigate multiple possible causes (including attack scenarios).

• 13:05 UTC – Key mitigation: Workers KV and Cloudflare Access are shifted to bypass the core proxy; impact is reduced. The Cloudflare Blog

• 14:30 UTC – Root cause identified; generation and propagation of bad feature files is stopped. A known-good configuration file is manually inserted and the core proxy is restarted. Most core traffic returns to normal. The Cloudflare Blog

• 14:40–15:30 UTC – Dashboard and login issues linger as Turnstile and backlog of authentication attempts create secondary load spikes. The Cloudflare Blog

• 17:06 UTC – Error rates return to baseline; Cloudflare declares systems fully normal. The Cloudflare Blog

From a user’s point of view, the outage felt worst in the late morning to early afternoon UTC, though exact impact windows varied by region and by which Cloudflare products each service depended on.

---

Why this outage matters so much

Centralization risk

Cloudflare is part of a small set of central internet infrastructure providers, alongside the major cloud platforms (AWS, Azure, GCP) and other large CDNs. When one of these players fails, the impact is wide and often non-obvious.

This outage:

• Didn’t come from a BGP routing mishap or an ISP cable cut.

• Didn’t come from a malicious attack (despite initial suspicions).

• Came from a single configuration and limits bug in an internal component.

That’s important because it shows how complex, tightly-coupled systems can fail catastrophically even without external interference. When many organizations build on the same provider, that provider becomes a de-facto systemically important piece of the internet.

“Soft” dependencies hurt too

Some of the affected services weren’t just using Cloudflare as a dumb CDN. They were:

• Using Cloudflare Access for authentication and zero-trust access.

• Using Workers KV as part of internal control planes.

• Relying on Turnstile for bot-resistant logins. The Cloudflare Blog+1

When those products failed, it wasn’t just website content that went down – logins, admin functions, and internal APIs broke as well. That makes recovery more complex: your status page, incident tooling, or admin UI might also rely on the very provider that just failed.

 

 

---

What Cloudflare says it will change

Cloudflare’s blog outlines several remediation steps the company is already taking to reduce the risk of anything similar recurring: The Cloudflare Blog

1. Harden ingestion of auto-generated configuration files
   Treat internally generated configs with the same skepticism and validation as user-supplied input, including strict schema and size checking before rollout.

2. More global kill switches
   Make it easier to quickly disable problematic internal modules (like Bot Management) across the network, so they fail open instead of panicking the entire proxy path.

3. Protect system resources from error storms
   Ensure that core dumps, debug metadata, and observability tooling cannot overwhelm CPU and memory when errors start to spike.

4. Review failure modes across core proxy modules
   Systematically audit how each internal module behaves under unexpected input or configuration, and ensure graceful degradation instead of global failure.

5. Refine rollouts and isolation
   While not spelled out in huge detail, the incident suggests Cloudflare will likely further segment how new configs and DB behaviors propagate, to reduce the chance that a single bad change affects the entire fleet.

They also framed the incident as an absolute failure of their resiliency expectations, calling it “unacceptable” and explicitly acknowledging the pain it caused both customers and ordinary internet users. The Cloudflare Blog

---

Lessons for infrastructure & SRE teams

Even if you’re not running something as huge as Cloudflare, there are some very practical design and operational lessons in this outage:

Treat internal config like untrusted input

It’s easy to assume that “our own” generated configuration is always correct. Yesterday shows why that’s dangerous:

• Always validate size, shape, and limits of configuration files before applying them.

• Consider canary application of config to a small subset of traffic or nodes first, with automated rollback on anomalies.

• Keep strict upper bounds and circuit breakers around feature counts, memory preallocation, and CPU usage.

Design for graceful partial failure

One bug in the Bot Management module should not be able to panic the entire proxy path:

• Default to fail-open vs fail-closed in some layers of security when the alternative is complete outage.

• Build clear, tested kill switches for non-core features.

• Ensure critical sub-systems (auth, status page, incident tooling) can operate in degraded mode or via alternate routes.

Observe the right signals

The oscillation between “good config” and “bad config” every five minutes made the signal look like attack traffic or noisy external behavior:

• Make sure you have per-version or per-config correlation in your observability pipeline.

• Build dashboards that make configuration changes visually obvious on top of error graphs.

• Include strong synthetic tests from an external vantage point, so you can quickly distinguish internal failure from network/path issues.

Don’t put all your eggs in one infra basket

For organizations using Cloudflare:

• Consider multi-CDN setups for truly mission-critical properties.

• Avoid making your status page entirely dependent on the same provider as your primary stack (Cloudflare does this, but there was coincidental trouble with their status page host yesterday which confused things further). The Cloudflare Blog+1

• Think twice before tightly coupling your authentication, API control planes, and frontend delivery to the same vendor without fallback paths.

---

The bigger picture

In the last few months alone, we’ve seen major outages at Microsoft Azure, Amazon Web Services, and now Cloudflare, all of which have temporarily knocked large chunks of consumer and enterprise services offline. AP News+2The Washington Post+2

The pattern is clear:

• The internet is increasingly dependent on a handful of giant infrastructure providers.

• Outages are often self-inflicted, coming from complex internal changes rather than external attacks.

• Even providers with world-class SRE practices can still be tripped up by unexpected interactions between configuration, database behavior, and hard-coded limits.

Yesterday’s Cloudflare incident is a stark reminder that “the cloud” isn’t magic. At the bottom, it’s still software written by humans, subject to the same classes of bugs as any other application—just with orders of magnitude more people depending on it.

For users, the incident will mostly be remembered as “that morning when X and ChatGPT wouldn’t load.”
For engineers, it will likely be studied as a textbook example of how subtle configuration bugs in a core distributed system can ripple out into a global internet event.