Gradient clipping is a technique to prevent the gradients from becoming too large, which can lead to exploding gradient problems especially in recurrent neural networks (RNNs). This can be particularly useful when you are observing NaNs during training or when the losses go to infinity.
PyTorch provides a simple utility called `torch.nn.utils.clip_grad_norm_` which can be used to clip the gradients of model parameters.
Here's how to use it:
```python
from torch.nn.utils import clip_grad_norm_
# define your model and optimizer
model = YourModel()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for inputs, targets in dataloader:
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, targets)
loss.backward()
# Clip gradients after computing the backward pass and before the optimization step
clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
```