It Starts with a Compute Shortage
In 2026, scrambling for compute has become the norm in the AI industry. China’s compute crunch traces back to one dependency: AI needs GPUs, and for high-end GPUs, NVIDIA is virtually the only option. Many people know NVIDIA sells graphics cards, but fewer are familiar with CUDA. CUDA is NVIDIA’s most critical competitive moat — one that took twenty full years to build. Understanding how CUDA got here is the key to understanding today’s compute market.
What CUDA Actually Is
CUDA stands for Compute Unified Device Architecture. It’s essentially a programming interface and development toolkit that lets developers use languages like C and C++ to directly harness GPUs for general-purpose computation.
Here’s an analogy: a GPU is like a muscle, with massive parallel compute power. CUDA is the nervous system that directs that muscle — it’s what schedules work across thousands of cores. Without CUDA, a GPU can only render game graphics. With CUDA, it can run AI training, scientific simulations, and other complex workloads. CUDA’s core contribution was turning the GPU from a graphics rendering tool into a general-purpose computing device.
The GPU World Before 2006
Before CUDA, GPUs had exactly one job: rendering graphics. If you wanted to use a GPU for scientific computing like matrix multiplication, you had to disguise your computation as a graphics rendering task. This approach was called GPGPU (General Purpose GPU). It’s like wanting to hire a chef to cook a meal but only being able to communicate by ordering dishes — you’d describe the color and shape of what you want on the plate. Incredibly inefficient and painful.
Jensen Huang’s Big Bet
In 2006, NVIDIA CEO Jensen Huang made a decision that looked insane at the time: invest roughly $10 billion into building the CUDA platform. For a company with a market cap of just a few tens of billions at the time, this was essentially betting the entire company. Wall Street’s skepticism was overwhelming. Investors couldn’t understand why a gaming GPU company was burning cash on a general-purpose computing platform with no clear future. Jensen’s thesis was simple: GPUs shouldn’t be limited to gaming.
The G80 architecture (GeForce 8800 GTX) was the first CUDA-capable GPU. It introduced the unified shader architecture, allowing all GPU cores to execute arbitrary compute tasks rather than being confined to drawing triangles. This architectural design established the foundational paradigm for GPU computing over the next twenty years.
The Long Cold Start
CUDA didn’t take off after launch. Instead, it endured six or seven years of near-indifference. From 2006 to 2008, almost nobody was using it — just a handful of researchers experimenting. From 2008 to 2010, academia started paying attention, but industry remained unmoved. From 2010 to 2012, NVIDIA’s stock price languished, and investor skepticism never let up.
During this period, NVIDIA did three things that mattered. First, it made the CUDA toolkit free — any developer could download and use it with no licensing fees. Second, it funded university education programs, pushing CUDA into computer science curricula to cultivate the next generation of developers. Third, it kept iterating on hardware architecture: from Fermi to Kepler to Maxwell, each generation brought meaningful improvements to CUDA’s performance and capabilities. Those six years of investment were like planting a tree — no fruit visible above ground, but roots quietly anchoring deep in the soil.
AlexNet Changed Everything
In 2012, the ImageNet image recognition competition hit a turning point. Graduate student Alex Krizhevsky used two GTX 580 GPUs to train a deep neural network called AlexNet. The error rate dropped to 15.3%, while traditional methods were still above 26%. Training time went from months to weeks. Hardware cost went from millions of dollars to a few thousand.
| Comparison | AlexNet | Traditional Methods |
|---|---|---|
| Error rate | 15.3% | 26%+ |
| Training time | Weeks | Months to years |
| Hardware cost | A few thousand dollars | Millions of dollars |
The critical point is that AlexNet’s implementation was deeply dependent on CUDA. Without CUDA’s parallel computing capability, this experiment couldn’t have been completed in any reasonable timeframe. This single result convinced AI researchers worldwide that GPU + CUDA + deep neural networks was the breakthrough path for AI. From that day on, CUDA went from being a researcher’s experimental tool to the standard setup for AI research.
Twenty Years of Iterative Accumulation
CUDA evolved from 1.0 to 14.x over twenty years of continuous iteration, with each major version delivering real capability gains.
| CUDA Version | Year | Key Feature |
|---|---|---|
| 1.0 | 2006 | Foundational programming model |
| 2.0 | 2008 | Double-precision floating-point support |
| 4.0 | 2011 | Dynamic parallelism |
| 6.0 | 2014 | Unified memory |
| 8.0 | 2016 | Volta architecture optimization |
| 10.0 | 2018 | Turing architecture, Tensor Cores |
| 11.0 | 2020 | Ampere architecture, Multi-Instance GPU |
| 12.0 | 2022 | Hopper architecture, H100 |
| 14.x | 2026 | Rubin architecture support |
By 2026, the numbers from twenty years of accumulation are substantial: over 5 million developers, more than 3,000 optimized operators, tens of millions of CUDA-dependent projects on GitHub, and over 95% of AI academic papers defaulting to CUDA. These numbers weren’t built overnight — they’re the natural result of two decades of hardware and software iteration, generation after generation.
A Multi-Layered, Nested Moat
CUDA’s moat isn’t just about technical leadership — it’s a multi-layered, nested system that makes it nearly impossible for competitors to enter. The outermost layer is ecosystem network effects: 5 million developers worldwide are fluent in CUDA, academic papers default to CUDA, every open-source project is built on CUDA, and newcomers to AI learn CUDA first. One layer in is software stack depth: over 3,000 highly optimized operator libraries, plus core compute libraries like cuDNN, cuBLAS, and NCCL, and toolchains like TensorRT and RAPIDS — an extraordinarily complete development ecosystem. Deeper still is the talent and knowledge accumulation: twenty years of GPU programming experts, compiler teams, and performance optimization experience. Even deeper is the hardware manufacturing gap: 3nm process leading competitors by two to three generations, HBM3e memory technology, and NVLink high-speed interconnect — all held by NVIDIA. The innermost layer is architecture design capability: while some papers are publicly available for reference, catching up still requires massive investment in both talent and capital.
These five layers are nested — each one builds on the layer before it. Even if a competitor breaks through one layer, four more are waiting.
Software Stack Lock-In
NVIDIA has built a complete software stack from the hardware layer up to the frameworks developers use. At the bottom sits NVIDIA GPU hardware. Above that, CUDA Runtime and Driver API provide the programming interface. Then come the core compute libraries: cuDNN, cuBLAS, NCCL, cuSPARSE. Above those are performance optimization tools: TensorRT, RAPIDS, Numba. At the top are the deep learning frameworks developers work with directly: PyTorch, TensorFlow, JAX.
The key issue: every single layer is deeply optimized only for NVIDIA hardware. PyTorch’s CUDA optimization is the most complete; AMD’s ROCm support is always incomplete. cuDNN’s performance far exceeds any alternative implementation — AMD still doesn’t have a comparable compute library. NCCL’s multi-GPU communication is mature and stable; AMD’s RCCL still has a long list of unfixed bugs. Even if developers want to choose different hardware, the entire software ecosystem won’t support them.
The Cost of Developer Migration
Suppose an AI company wants to migrate its infrastructure from NVIDIA to AMD. The challenges are daunting. Half a million lines of CUDA code need to be rewritten from scratch. The team needs retraining. CI/CD pipelines need to be rebuilt. Third-party open-source library compatibility needs to be verified one by one. Performance and stability will inevitably degrade during migration.
| Challenge Dimension | Specifics |
|---|---|
| Code assets | Hundreds of thousands of lines of CUDA code need full rewrite |
| Talent pool | Team is expert in CUDA, needs retraining |
| Infrastructure | CI/CD is entirely NVIDIA GPU-based, needs rebuild |
| Third-party dependencies | Open-source libraries are CUDA-bound, migration compatibility is poor |
| Migration risk | More bugs and performance degradation during migration |
The migration cost is so high that the vast majority of companies simply can’t afford the switch. Developer lock-in isn’t about CUDA being technically perfect — it’s about the cost of switching to an entirely different ecosystem being unbearably high.
Academia’s Self-Reinforcing Loop
Academia’s CUDA binding deserves its own section. A researcher writes their experiment code in CUDA. The paper gets published and the code goes open-source. Subsequent researchers who want to reproduce the results must use CUDA, then write more CUDA code themselves. This loop continuously reinforces itself, and the result is that 95% of AI papers default to CUDA. If you want to publish, reproduce others’ results, or use open-source models, you need NVIDIA GPUs. Academia is firmly locked into the CUDA ecosystem.
Why Competitors Can’t Catch Up
AMD launched ROCm (Radeon Open Compute) in 2016 as a CUDA counterpart. The gap is obvious: CUDA has been accumulating since 2006, with 5 million developers and 3,000+ operator libraries. ROCm started a full decade later, has an estimated fewer than 50,000 developers, around 200 operator libraries, and its stability and compatibility are nowhere near production-ready. Much CUDA code can’t be directly ported to ROCm. Documentation is incomplete. Community activity is low.
Intel is in an even worse position. oneAPI didn’t arrive until 2020 — 14 years behind CUDA. Intel’s own GPU hardware performance can’t keep up, no developer wants to invest in learning yet another new API, and its market share is negligible.
| Comparison | CUDA | ROCm | oneAPI |
|---|---|---|---|
| Launch year | 2006 | 2016 | 2020 |
| Developer count | 5 million+ | Under 50,000 | Negligible |
| Operator library | 3000+ | ~200 | Negligible |
| Stability | Very high | Frequent bugs | Unverified |
| Framework support | Native-level | Incomplete adaptation | Almost none |
There’s a cruel paradox in the catching-up game. Even if AMD matched CUDA’s 2022 level today, CUDA would already be at version 14.x in 2026, and NVIDIA would have shipped the Rubin architecture. NVIDIA iterates annually; AMD chases annually. The gap always exists. The catcher’s dilemma: you’re always catching up to a past target.
The Hardware Gap
CUDA’s moat isn’t only software — hardware is an equally high wall. NVIDIA has built a deep partnership with TSMC. Every GPU generation gets the most advanced process available. H100 uses 4nm, B200 uses 4NP, and the Rubin R100 goes straight to 3nm, expected to enter mass production in H2 2026. TSMC’s most advanced process is allocated to NVIDIA first — a partnership built on massive order volumes and long-term trust.
High-end AI training also requires HBM (High Bandwidth Memory), the most advanced memory packaging technology, produced by SK Hynix and Samsung. H100 ships with 80GB HBM3 at 3.35 TB/s bandwidth. B200 has 192GB HBM3e at 8 TB/s. B300 and Rubin push further. NVIDIA gets priority HBM supply, so competitors start from a generation behind on memory specs.
| GPU | Memory Spec | Bandwidth |
|---|---|---|
| H100 | 80GB HBM3 | 3.35 TB/s |
| B200 | 192GB HBM3e | 8 TB/s |
| B300 | 192GB+ HBM3e | 8 TB/s+ |
Multi-GPU training needs high-speed interconnect. NVLink 4.0 delivers 900 GB/s bandwidth; NVLink 5.0 will reach 1.8 TB/s. AMD’s Infinity Fabric sits at around 400 GB/s, with a clear gap in both performance and stability. Hardware-level leadership means that even if a competitor’s software ecosystem catches up, the physical performance gap remains unbridgeable.
A Moat Dug Over Twenty Years
Back to the original question: how was CUDA’s moat dug, step by step? The core logic isn’t complicated. Jensen Huang bet on a direction almost nobody believed in back in 2006 — that GPUs could become general-purpose computing tools. Through six years of cold start, NVIDIA quietly cultivated the first generation of developers with free tools and university programs. In 2012, AlexNet proved the value of that bet, and academia began its large-scale shift to CUDA. Over the next ten years, NVIDIA built out the complete software stack ecosystem. By the time large models exploded in 2022, CUDA was everywhere — the de facto standard for AI infrastructure.
What competitors face isn’t a technical advantage from some brilliant design. It’s the ecosystem depth that comes from twenty years of sustained investment. Five million developers, 3,000 operator libraries, 95% of papers bound to CUDA — the time cost behind those numbers is what the moat really means. Replicating all of that would take an equally long time. And NVIDIA isn’t standing still.