Definition
Gradient descent is Gradient descent is the training method that adjusts a model's parameters in many small steps, each one nudging them in the direction that most reduces the model's error on its training examples.
Gradient descent is the basic engine of how neural networks learn. A model starts with parameters set to nearly random values. It makes predictions, and a score called the loss measures how wrong they are. Gradient descent then changes every parameter by a small amount in the direction that lowers the loss. Repeat that millions of times and the model gets good at its task.
Picture a hiker in thick fog trying to reach the bottom of a valley. She cannot see the valley, but she can feel which way the ground slopes under her feet. So she takes a step downhill, checks the slope again and repeats. The slope is the gradient, and walking down it is the descent.
In practice, training uses variants such as stochastic gradient descent, which computes the gradient on small random batches instead of all the data, and optimizers such as Adam, which adapt the step size for each parameter. The core loop is the same.
Almost everything expensive about AI traces back to this loop. Pretraining runs it across enormous amounts of text. Fine-tuning runs it again on a smaller set of examples, and methods like LoRA run it on only a small set of added weights to cut the cost. The step size and the quality of the training data decide whether the model learns well, learns slowly or fails to learn.
It also explains a common limit. Gradient descent finds a good setting of the parameters, not necessarily the best one, and it only learns patterns present in the data it is shown.
Inference does not use it. When you chat with a model, the parameters stay fixed, and inference only runs the trained network forward. Researchers are also exploring ways to change a model's weights without this loop, but gradient descent remains the standard method.
See reinforcement learning, which also uses gradient steps but gets its score from rewards, and transformer for the network design most often trained this way.
The learning rate is the size of each step gradient descent takes when adjusting parameters. Too large and training becomes unstable, too small and it is very slow.
No. Gradient descent is used during training. When you chat, the model's parameters are fixed and it only runs the trained network to produce a reply.
MIT CSAIL researchers say a frontier LLM, used as is, read and rewrote neural network weights without gradient descent, lifting a small Qwen model's MATH-500 score from 7.9% to 48.1%. It is an early blog post, not a peer-reviewed paper, and still the strangest result of the week.