Reference · The Applied Higher-Order Functions catalogue

rainbow.algorithms.proofs.ahof.gae54

The Applied Higher-Order Functions cataloguesrc/rainbow/algorithms/proofs/ahof/gae54.clj3 definitions, 4 theorems

AHoF D54 Generalized Advantage Estimation: form a low-variance advantage estimate as an exponentially-weighted sum of one-step TD residuals; the recursive form combines the current residual with the discounted future advantage: gae-advantage delta lambda-gamma next-advantage = delta + lambda-gamma * next-advantage. A residual 1/4 with decay lambda-gamma 1/2 and future advantage 1/2 gives 1/2; the one-step (lambda-gamma 0) estimate is the residual 1/4. Values are proved by cross-product equivalence. Proved native and axiom-free. Scope (measured by the kernel): this module has 3 worked instances (statements about specific values) and 1 definitional unfoldings (true by unfolding a definition). None of its theorems is a guarantee about the algorithm for all inputs; `lein verify rainbow.algorithms.proofs.ahof.gae54` prints this classification.

gae54/gae-advantage :
  (x₁ x₂ x₃ : ℚ≥0)
  → ℚ≥0

gae54/recursive-estimate :
  → ℚ≥0

gae54/one-step-estimate :
  → ℚ≥0

gae54/gae-advantage-unfold :
  (delta lg next : ℚ≥0)
  → gae54/gae-advantage delta lg next = delta + lg · next

gae54/recursive-is-half :
  → gae54/recursive-estimate = 1/2

gae54/one-step-is-residual :
  → gae54/one-step-estimate = 1/4

gae54/gae :
  → proposition/conjunction (gae54/recursive-estimate = 1/2) (gae54/one-step-estimate = 1/4)