Spurious Rewards: Rethinking Training Signals in RLVR
Spurious Rewards: Rethinking Training Signals in RLVR [130.3] 検証可能な報酬(RLVR)を用いた強化学習は,特定のモデルにおいて強い数学的推論を導出できることを示す。 例えば、RLVRはQwen2.5-Math-7BのMATH-500の性能を21.4%向上させた。 コード推論 — 実際のコード実行なしにコードで考える — は、RLVR以降、はるかに頻繁になる、独特なQwen2.5-Mathの振る舞いである。 論文参考訳（メタデータ） (Thu, 12 Jun 2025 17:49:55 GMT)
「We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain models even with spurious rewards that have little, no, or even negative correlation with the correct answer. For example, RLVR improves MATH-500 performance for Qwen2.5-Math-7B in abso- lute points by 21.4% (random reward), 13.8% (format reward), 24.1% (incorrect label), 26.0% (1-shot RL), and 27.1% (majority voting)—nearly matching the 29.1% gained with ground truth rewards.」という直観に反する結果の報告と検証。
「Our findings have three main implications: base model pretraining significantly affects RLVR outcomes; even corrupted or spurious supervision can enhance reasoning when it triggers useful existing behaviors; and effects observed in one model family may not generalize to others. Our work highlights the importance of (1) testing across multiple models with differing pretraining distributions, and (2) testing across multiple different baselines, such as format and random rewards, when evaluating reinforcement learning techniques.」としている。モデルに依存し、結果が間違っていても一定効果があるのは本当に面白い。内部知識とそれを引き出すテクニックの間にはいまだギャップがあるということだろうか。。