Reference · The Applied Higher-Order Functions catalogue

rainbow.algorithms.proofs.ahof.ppo

The Applied Higher-Order Functions cataloguesrc/rainbow/algorithms/proofs/ahof/ppo.clj3 definitions, 3 theorems

AHoF D43 Proximal Policy Optimization: a policy-gradient method that keeps updates in a trust region by CLIPPING the probability ratio r = pi_new/pi_old into [1 - eps, 1 + eps] before forming the surrogate objective. Clipping removes the incentive to move the ratio outside the interval; it is meant to give stable improvement without a hard KL constraint (an empirical motivation, not a property proved here). The clip is a clamp: ppo-clip r lo hi = max(lo, min(r, hi)) built from the decidable rational min and max (each = if a<=b then .. else .. over logic/if-data). Proved native and axiom-free: the clip; a ratio above the band clamped down to hi; a ratio below the band clamped up to lo; and an in-band ratio passed through unchanged -- all reducing by reflexivity. The advantage estimates and gradient step are executable; what is proved is the clipping clamp. Requires audit/decide-nnrat-at-most, logic/if-data (both in the D63 checkpoint). Scope (measured by the kernel): this module has 3 worked instances (statements about specific values). None of its theorems is a guarantee about the algorithm for all inputs; `lein verify rainbow.algorithms.proofs.ahof.ppo` prints this classification.

ppo/nnrat-min :
  (x₁ x₂ : ℚ≥0)
  → ℚ≥0

ppo/nnrat-max :
  (x₁ x₂ : ℚ≥0)
  → ℚ≥0

ppo/clip :
  (x₁ x₂ x₃ : ℚ≥0)
  → ℚ≥0

ppo/clips-above-to-hi :
  → ppo/clip 3/2 3/4 5/4 = 5/4

ppo/clips-below-to-lo :
  → ppo/clip 1/2 3/4 5/4 = 3/4

ppo/passes-in-band :
  → ppo/clip 1 3/4 5/4 = 1