What is CUDA?

CUDA is the software layer that lets programs use an Nvidia GPU for work other than drawing graphics. Nvidia released it in 2007. It includes a programming language extension, compilers, and libraries of ready-made math routines for things like matrix multiplication, the core operation in AI.

What it does

A GPU is powerful, but code has to be written in a way the chip understands. CUDA is the translator and the toolbox. A programmer writes a small function called a kernel, and CUDA runs thousands of copies of it across the GPU's cores at once.

Most people who build AI never write CUDA by hand. Frameworks such as PyTorch call CUDA libraries underneath. You type a line of Python, and CUDA does the heavy lifting on the chip.

Why it matters

CUDA is a big part of why Nvidia leads AI hardware. Over many years, researchers, universities and companies wrote their tools, tutorials and optimized code for CUDA. New models usually work on it first. That pile of existing software is hard to replace, which is sometimes called a moat.

For a competitor, the chip is only half the problem. A rival accelerator also needs a software stack that runs the same programs without a painful rewrite. AMD, Google and Amazon each offer their own toolkits for this reason, and many efforts try to let existing PyTorch code run on non-Nvidia chips with few changes.

Where the lock-in shows up

Standard model code often moves between chips with modest effort. The trouble starts with custom kernels. Teams that squeeze extra speed out of a model by writing CUDA kernels by hand, for example to speed up inference, must rewrite that code for any other chip. That switching cost is what keeps many companies on Nvidia even when another chip looks cheaper on paper.

CUDA and neighboring terms

CUDA sits between your model and the hardware. Above it are frameworks and the model itself, with its weights. Below it is the chip and its memory, such as VRAM and high-bandwidth memory. Techniques like speculative decoding and quantization are often shipped as CUDA kernels.

What to take from it

When you read that a chip works with PyTorch, ask how much of your actual code depends on CUDA. If the answer is "only the framework", moving is likely easy. If your team has custom kernels, plan for real porting work and test speed and cost on your own workload before you decide.