All topics
movementevergreen
Research-ready episode

The Cloud Needed a Clock, a Cable, and a Power Plant

云为什么需要一座钟、一条电缆和一座电厂

What this research must answer

A 2024 West African cable outage opens a much larger story: the internet was invented as a way to route around damage, but it became a physical industrial system of packet switching, fiber, landing stations, data centres, clocks, databases and privately owned long-haul capacity. The episode follows why traffic must be handed off, how light carries it, why a healthy data centre can still fail, why Google needed a global time service for Spanner, why Meta and other hyperscalers now build routes themselves, and why AI is turning network geography into strategy again.

2024年3月14日，西非海岸附近的四套海底电缆系统同时受到影响。事件暴露出一个常被忽视的区别：电缆数量不等于独立的故障域。流量可以重路由，但剩余路径的容量有限。随后，微软南非Azure区域出现拥塞、丢包和服务故障，尽管数据中心本身仍在运行。本集沿着共享地理风险、容量损失、云端决策、基础设施采购和正向反例展开，回答什么样的冗余才真正有效。

Opening tension

On 14 March 2024, four apparent roads into West Africa failed near the same coast. A user saw a timeout. What actually broke was a century of invisible engineering: the routes between networks, light inside glass, landing stations, cloud dependencies, and the assumption that several lines on a map mean several ways home.

Opening variants
  1. ·

    Azure pressure

    On March 14, 2024, Microsoft's data centres in South Africa were still running. The servers were alive. Several underwater paths into the region were not, and the remaining capacity began to fill.

    Start with a measurable cloud consequence before revealing the shared physical cause.

  2. ·

    False map

    A map of West Africa showed several submarine cables. They looked like independent alternatives. Then four systems were affected near the same coast.

    Start with the visual contradiction that drives the episode's central question.

  3. ·

    Bought survival

    When the west-coast cables failed, Google's Equiano remained live. Google later reported that some affected customers bought fiber and capacity through the surviving route.

    Start with infrastructure as a commercial product, then trace back to the failure domain.

How the episode moves
  1. 01

    Why was the internet designed to keep talking after a route failed?

    Packet switching breaks a message into units that can be forwarded through different networks and routes. That principle made the internet more survivable than a single dedicated circuit — but it never removed the need for physical paths between networks.

    Internet Society infrastructure primer; Congressional Research Service cable overview.

    What does one packet actually cross before it reaches a distant service?

  2. 02

    How did a network of routers become a planetary physical machine?

    Traffic is handed from access network to ISP, backbone, landing station, submarine fiber, terrestrial fiber and data centre. Optical fiber turns electrical data into pulses of light; amplifiers and switches extend and direct it, but every hand-off creates capacity, ownership and failure boundaries.

    Internet Society West Africa report; Congressional Research Service cable overview.

    Why do several named cables sometimes behave like one fragile road?

  3. 03

    What did the West Africa outage reveal about that machine?

    ACE, SAT-3, WACS and MainOne were affected across 13 countries near the Côte d’Ivoire coast. Separate cable names did not guarantee independent corridors or landfalls. Rerouting preserved some reachability, but it could not manufacture headroom; queues, latency and packet loss rose when demand met the remaining capacity.

    Cloudflare; Internet Society West Africa report; Internet Society resilience policy brief.

    Why can servers remain alive while an online service fails?

  4. 04

    What is a cloud service really depending on?

    Microsoft reports that concurrent east- and west-coast cable losses reduced WAN capacity. Availability fell as low as 77%; engineers reduced inter-region traffic, reconfigured services, throttled traffic, and shifted capacity from Lagos Edge.

    Microsoft Azure Post Incident Review, including the event timeline, affected services, congestion, and mitigation actions.

    Why does global computation also need a shared notion of time?

  5. 05

    Why did Google build a clock into its database story?

    Spanner’s TrueTime gives distributed servers bounded time uncertainty so a globally replicated database can assign transaction timestamps consistent with what clients observed. It is a reminder that ‘the internet’ is not only transmission: applications need ordering, replication and coordination across far-apart machines.

    Google Cloud Spanner TrueTime documentation.

    When that much application state and traffic cross the same geography, why rent the network instead of building it?

  6. 06

    Why are hyperscalers becoming infrastructure owners?

    At enough scale, capacity, latency, route control and resilience become product constraints. Google’s Equiano added a high-capacity Atlantic route; Meta has built terrestrial fiber between data centres and announced multi-year subsea investment. They are buying more than bandwidth: they are designing failure domains around their own compute and traffic.

    Google Cloud Equiano announcement; Congressional Research Service; UK Parliament subsea resilience report.

    Why is AI making this private build-out faster and more geopolitical?

  7. 07

    What does the AI future change — and what does it not?

    Large AI clusters intensify demands for power, data-centre networking and long-haul capacity. More fiber can increase throughput, but the West Africa case keeps the hard lesson intact: new capacity is not resilience if it shares the same physical risk or leaves the same cross-region dependency graph.

    Meta Project Waterworth announcement; Internet Society resilience brief.

    What would genuine resilience look like from a user’s side?

  8. 08

    What does the timeout finally mean?

    East Africa’s later outage showed the countercase: other cables, terrestrial routes, IXPs and caches could preserve service. Return to the first user. A timeout is not immaterial software failure. It is a map of choices about routes, light, landings, data placement, clocks and capital. The more AI makes the world depend on a few immense systems, the more those physical choices become a question of power as well as speed.

    Internet Society West Africa report; Internet Society East Africa report.

    The future internet will be judged not by how invisible it feels, but by whose connection and whose data can survive its visible failures.

Story bank
  1. Opening pressure

    Four roads, one morning

    The 14 March 2024 West Africa disruption affected four named submarine systems and 13 countries.

  2. Physical setup

    The shared coast

    The four systems converged near the reported failure area along the Cote d'Ivoire coast.

  3. Mechanism

    Rerouting cannot create bandwidth

    Traffic shifts to surviving paths, where congestion, packet loss, and latency determine usable service.

  4. Institution under pressure

    Azure loses headroom

    Microsoft reconfigured traffic, reduced inter-region load, throttled, and moved capacity from Lagos Edge.

  5. Conceptual reversal

    A cable count is not a failure-domain count

    Cote d'Ivoire's five listed systems did not provide five independent protections.

  6. Application/value

    Buying an independent route

    Google's Equiano investment and capacity purchases show why firms pay for route control and spare capacity.

  7. Countercase

    Where redundancy worked

    East Africa's May 2024 outage retained service through other cables, terrestrial links, IXPs, and caches.

  8. Return/payoff

    The waiting screen

    Users experience exhausted headroom as a payment waiting, a work system timing out, or a lesson that will not load.

How the system works
Input
Optical traffic, route announcements, user demand, cloud API calls, and the available submarine and terrestrial capacity.
Transformation
A physical path is removed; routing reconverges; operators redistribute, prioritize, throttle, or reconfigure traffic. Queues form and latency or packet loss rises as surviving capacity approaches saturation.
Output
Partial connectivity, degraded cloud performance, service-specific failures, or preserved operation through a genuinely independent route.
Limit
Routing cannot manufacture bandwidth. Cable names do not prove independent geography, and a local cloud region cannot eliminate cross-region dependencies.
Mechanism cards

  1. Physical path loss

    Input
    Optical traffic on a submarine system and its landing/terrestrial handoffs.
    Transformation
    A cable system is removed or degraded by a physical event; routes reconverge onto remaining systems.
    Output
    Reduced total regional capacity and altered paths.
    Limit
    Multiple cable systems can share a geographic failure domain.
    Evidence
    Internet Society West Africa report; CRS cable overview.

  2. Capacity transformation

    Input
    Demand plus the residual capacity of surviving links.
    Transformation
    Operators redistribute, prioritize, throttle, or reconfigure traffic.
    Output
    Partial reachability with changed latency, packet loss, and throughput.
    Limit
    Rerouting cannot create bandwidth; an online route can still be unusable to an application.
    Evidence
    Cloudflare; Microsoft Azure PIR.

  3. Cloud dependency

    Input
    Local user requests, cloud APIs, identity, monitoring, and cross-region calls.
    Transformation
    Requests cross WAN paths whose capacity and latency are altered by the outage.
    Output
    Some local services continue while externally dependent services degrade or fail.
    Limit
    A healthy data centre does not guarantee a healthy dependency graph.
    Evidence
    Microsoft Azure PIR.

  4. Procurement value

    Input
    Demand forecasts, route risk, landing options, and required capacity.
    Transformation
    Hyperscalers and telecom partners fund and design cable, landing, terrestrial, and cloud-region infrastructure.
    Output
    More capacity, route control, and potentially independent failure domains.
    Limit
    Capacity growth without physical separation can increase throughput without proportional resilience.
    Evidence
    Google Equiano announcement; Google Cloud WAN; UK Parliament report.
Where this changes a real decision

SignalCable alarms, route withdrawals, Cloudflare outage telemetry, rising latency, packet loss, and reduced WAN capacity.

Decision ownerMicrosoft's African network operations and cloud service teams.

ThresholdResidual capacity and packet-loss/latency levels begin to threaten service availability and regional SLOs.

ActionReduce inter-region traffic, reconfigure services, use traffic engineering and throttling, and shift capacity from Lagos Edge to South Africa.

ConsequenceAvailability recovered above 97% and 99% during mitigation but degraded again as traffic volumes rose; multiple Azure and Microsoft 365 services experienced varying impact.

Application chains

  1. Azure traffic engineering

    SignalConcurrent east- and west-coast cable losses reduced WAN capacity; latency, packet drops, and availability crossed service-risk thresholds.

    Decision ownerMicrosoft African network operations and cloud service teams.

    ActionReduce inter-region traffic, reconfigure affected services, throttle, and shift capacity from Lagos Edge.

    ConsequenceAvailability recovered during mitigation but degraded again as traffic volume increased.

  2. Capacity as a purchased product

    SignalA surviving Equiano route remained available while other paths were affected.

    Decision ownerGoogle network operators and customers needing continuity.

    ActionProtect traffic and purchase or reallocate fiber/capacity through the surviving route.

    ConsequenceGoogle reports that Equiano users stayed live and affected customers used the route to get back online; treat this as company-reported evidence.

The limit this episode must keep

The same incident did not affect every place equally. South Africa had nine listed submarine systems and multiple terrestrial routes; Niger retained terrestrial paths through neighboring countries; and MTWA/MoovAfrica remained operational in Cote d'Ivoire. The stronger countercase is Internet Society's May 2024 East Africa report: damage to SEACOM and EASSy did not cause a total national outage because other submarine and terrestrial routes, IXPs, and local caches remained available. Resilience therefore depends on independent physical paths, terrestrial diversity, spare capacity, and local service placement—not cable count alone.

Key sources