Reference · The Applied Higher-Order Functions catalogue

rainbow.algorithms.proofs.ahof.gradient-clipping

The Applied Higher-Order Functions cataloguesrc/rainbow/algorithms/proofs/ahof/gradient_clipping.clj1 definitions, 3 theorems

AHoF Gradient Clipping (clip-by-norm): stabilize the training of deep and recurrent networks by capping the gradient's magnitude, preventing the exploding-gradient problem where a single huge update destroys the model. When the gradient norm exceeds a threshold it is rescaled to exactly the threshold (preserving direction); otherwise it is left unchanged. The clipping decision is: clips? grad-norm threshold = decide (threshold <= grad-norm) and when clipping, the rescale factor is threshold / grad-norm. Proved native and axiom-free: the clip decision; that an oversized gradient is clipped; the certificate that a clipped gradient's norm really exceeded the threshold (native nnrat/at-most); and that a small gradient is left unclipped. The norm and the rescale are executable; what is proved is the clip decision. Requires nnrat/at-most, audit/decide-nnrat-at-most, nnrat/decide-at-most-true-sound (all in the D63 checkpoint). Scope (measured by the kernel): this module has 3 worked instances (statements about specific values). None of its theorems is a guarantee about the algorithm for all inputs; `lein verify rainbow.algorithms.proofs.ahof.gradient-clipping` prints this classification.

gradient-clipping/clips? :
  (x₁ x₂ : ℚ≥0)
  → data/Bool

gradient-clipping/large-gradient-clipped :
  → bo/is-safe 7 5 = data/true

gradient-clipping/clipped-exceeds-threshold :
  → 5 ≤ 7

gradient-clipping/small-gradient-unclipped :
  → bo/is-safe 3 5 = data/false