Reference · The Applied Higher-Order Functions catalogue

rainbow.algorithms.proofs.ahof.nash-q-learning

The Applied Higher-Order Functions cataloguesrc/rainbow/algorithms/proofs/ahof/nash_q_learning.clj2 definitions, 3 theorems

AHoF D81 Nash Q-Learning (classical: learn stage-game Q-values and back up the Nash EQUILIBRIUM value of each stage game, Hu-Wellman 2003). This module does NOT model that backup: it has no strategies, no opponent payoffs and no equilibrium. It defines only a maximum selection between two supplied values: prefers-action-zero? q0 q1 = decide (q1 <= q0) nash-value q0 q1 = if (q1 <= q0) then q0 else q1 (the larger value) over exact rationals. The larger of two joint-action payoffs is not in general a Nash value: with row payoffs [[3,0],[5,1]] and column payoffs [[3,5],[0,1]] the unique equilibrium pays (1,1) while the largest individual payoff is 5. Proved native and axiom-free, as worked instances: prefers-action-zero? 3/4 1/2 is true, nash-value 3/4 1/2 = 3/4, and prefers-action-zero? 1/2 3/4 is false. The equilibrium backup and the convergence of Nash Q-learning are classical and not proved here. Requires audit/decide-nnrat-at-most, logic/if-data (both in the D63 checkpoint). Scope (measured by the kernel): this module has 3 worked instances (statements about specific values). None of its theorems is a guarantee about the algorithm for all inputs; `lein verify rainbow.algorithms.proofs.ahof.nash-q-learning` prints this classification.

nashq/prefers-action-zero? :
  (x₁ x₂ : ℚ≥0)
  → data/Bool

nashq/nash-value :
  (x₁ x₂ : ℚ≥0)
  → ℚ≥0

nashq/prefers-higher :
  → bo/is-safe 3/4 1/2 = data/true

nashq/value-is-maximum :
  → fsc/node-value 3/4 1/2 = 3/4

nashq/switches :
  → bo/is-safe 1/2 3/4 = data/false