Your Gradient Clipping Code Might Not Be Clipping the Gradient Norm

Two short pieces of code can both be called “gradient clipping” while doing different things. One caps each coordinate. The other measures the length of the entire gradient vector and, when necessary, rescales every coordinate by the same factor.

That distinction is easy to miss when reviewing generated code. A loop full of min and max calls looks reasonable, and its outputs may even be smaller than its inputs. Neither observation establishes that it implements global norm clipping.

Here is a complete numerical example and a small Python implementation you can run without a machine-learning framework.

Start with a vector you can inspect

Take a gradient [3, 4] and a maximum L2 norm of 2. Its original length is 5. For global L2 norm clipping, multiply both coordinates by 2 / 5. The result is [1.2, 1.6], whose length is 2. The vector retains its direction because both coordinates receive...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE