Reference · The Applied Higher-Order Functions catalogue

rainbow.algorithms.proofs.ahof.trpo

The Applied Higher-Order Functions cataloguesrc/rainbow/algorithms/proofs/ahof/trpo.clj1 definitions, 3 theorems

AHoF D51 Trust-Region Policy Optimization: improve a policy by maximizing a surrogate advantage objective subject to a trust-region constraint that the new policy stays close to the old one in KL divergence. (TRPO's monotone-improvement bound needs a max-KL penalty term; a mean-KL constraint alone does not imply it, and it is NOT proved here.) A candidate policy update is admissible exactly when its KL divergence from the current policy is within the trust radius delta: within-trust-region? kl delta = decide (kl <= delta) over the exact-rational KL divergence. Proved native and axiom-free: the trust-region admissibility test, the certificate that the admitted worked update stays within the trust region (1/10 <= 1/4 as a native nnrat/at-most), and that an update whose KL exceeds the radius is rejected (the line search must backtrack). The surrogate advantage, the natural-gradient step, and the KL estimation are executable; what is proved is the trust-region admissibility decision. Requires nnrat/at-most, audit/decide-nnrat-at-most, nnrat/decide-at-most-true-sound (all in the D63 checkpoint). Scope (measured by the kernel): this module has 3 worked instances (statements about specific values). None of its theorems is a guarantee about the algorithm for all inputs; `lein verify rainbow.algorithms.proofs.ahof.trpo` prints this classification.

trpo/within-trust-region? :
  (x₁ x₂ : ℚ≥0)
  → data/Bool

trpo/update-admissible :
  → lis/extends? 1/10 1/4 = data/true

trpo/admitted-stays-in-region :
  → 1/10 ≤ 1/4

trpo/oversized-update-rejected :
  → lis/extends? 1/2 1/4 = data/false