Weight decay

Weight decay is a regularization technique used in training neural networks that helps prevent the model from overfitting. It works by adding a penalty on the size of the weights to the loss function, encouraging the model to maintain smaller weight values. This penalty is proportional to the square of the magnitude of the weights, effectively reducing their value during training, which can lead to a more generalized model.