Why the standard reward post-training pipeline stalls when success is rare
Contemporary generative post-training leans on two assumptions: supervised pre-training covers the generation space broadly, and a reward function supplies dense enough information to sharpen quality [1]. RLHF and its variants made this pipeline the default for aligning language models, with reward models trained on human or AI preferences and optimized by policy-gradient methods [8][9]. Reward shaping work further stabilized this loop by bounding rewards and centering them against a reference, showing that the shape of the reward signal, not just its presence, governs whether training stays useful [2]. But these successes all presuppose that the base policy occasionally produces rewarded samples and that reward queries are affordable.
BaNEL isolates the harder case: base success probability so low the model may never see a positive sample, and reward oracles expensive enough that number of reward evaluations (NREs) becomes a first-class budget [1]. Under classic policy gradient, all-zero reward batches yield exactly zero gradient, so learning degenerates into brute-force sampling that cannot exceed the base success rate [1]. A constant negative baseline produces nonzero empirical gradients, but training exclusively on negatives for long horizons breaks pre-trained knowledge and collapses the model [1]. GFlowNet-style reward-based sampling also fails here: with r(x)=0 the trajectory-balance loss reduces to the empirical variance of log pθ, which assigns arbitrary mass over the sampled batch and leaves the rest uncontrolled [1]. The gap BaNEL addresses is therefore not exploration in general but learning when the only signal is the structure of failure.
BaNEL's move: treat the failure distribution as a generative model to be learned
BaNEL's core idea is to cast learning the regularities underlying failures as another in-loop generative modeling problem [1]. The algorithm samples candidates from the current proposal, fits a failure model pφ to those zero-reward samples, and forms a Bayesian posterior that reweights the proposal by the likelihood ratio pθ(x)/pφ(x), accepting only candidates that are much more likely under the prior than under the failure model [1]. When the threshold is chosen so exactly m of mf candidates are accepted, this coincides with the elite-selection step of the cross-entropy method, except that CEM ranks by reward while BaNEL ranks by the likelihood ratio because reward is always zero [1]. Sequential filtering with distillation (Algorithm 2) accumulates rejection regions across rounds and distills the filtered distribution back into the model, which the authors argue is theoretically equivalent to maintaining multiple negative models but far more efficient [1].
The design directly targets the two failure modes of prior sparse-reward methods. Count-based pseudo-counts update the density model only once per reward evaluation, and multiple updates inflate the bonus for non-novel states; RND's predictor-target MSE similarly shrinks regardless of novelty when updated repeatedly, raising NREs [1]. BaNEL instead exploits additional compute to train pφ more thoroughly at each stage, so success rate scales with compute at fixed NRE budget [1]. This is the paper's sharpest conceptual claim: failure data is not merely a negative gradient to be subtracted but a distribution whose regularities can be modeled and inverted.
What the sparse-reward experiments actually demonstrate
The authors construct sequential generation tasks with deliberately filtered reward-one samples. On MNIST 0→6, an autoregressive transformer pre-trained on 0-digits must discover 6s; the base success rate is 8e-26, and BaNEL's posterior samples shift toward 0-shaped digits with the right side removed, resembling 6s [1]. With a total NRE budget of 7500 for all methods, BaNEL's improvement factor over the base model grows with the number of epochs used to train pφ, while count-based and RND baselines cannot exploit that extra compute [1]. On an adversarial attack against a digit-addition language model, BaNEL finds carry-chain and leading-zero prompts that break the target, with rule-based attack rates of 99.02% and 99.96% respectively versus 0.04% for the pre-trained model [1]. On a challenging GSM8K subset where pre-trained models fail, BaNEL again improves success rate without any reward-one observation [1].
The comparison against novelty bonuses is the headline: BaNEL outperforms count-based and RND methods by up to several orders of magnitude in success rate while using fewer reward evaluations [1]. That framing matters because it separates two budgets that sparse-reward papers often conflate. Novelty bonuses buy exploration with frequent oracle calls; BaNEL buys it with in-loop generative modeling of failures, trading compute for oracle queries. The MNIST compute-scaling curve is the clearest evidence that this trade is real rather than incidental [1].
How BaNEL sits against curiosity, negative-RL, and preference-based alternatives
Earlier curiosity work established that agents can generate their own dense signal from novelty. Episodic Curiosity compared observations against an episodic memory using reachability rather than raw surprise, and showed that prediction-error methods like ICM can collapse into degenerate behaviors such as persistent firing in no-reward DMLab tasks [3]. BaNEL inherits the diagnosis that intrinsic bonuses do not scale with compute and must query the oracle frequently, but replaces the novelty bonus with a learned failure model [1]. The lineage is therefore continuous in motivation and discontinuous in mechanism: where EC asks how many steps it takes to reach an observation, BaNEL asks how much more likely the prior is than the failure model to produce it.
Negative-reward learning has a separate lineage. TOPR showed that REINFORCE with asymmetric tapered importance sampling can handle positive and negative examples in a unified offline framework, and that the baseline parameter effectively controls the mix of positive and negative examples in the objective [4]. PREFORL took a different route in offline RL, contrasting successful demonstrations against failures and synthetically degraded behaviors to avoid value overestimation in sparse-reward D4RL tasks [5]. Both confirm that negative examples carry usable information, but both assume some positive or preference signal exists to contrast against. BaNEL's distinctive claim is that no successful sample is needed at all, because the failure distribution itself is the learning target [1]. That is a stronger assumption than TOPR's or PREFORL's, and it is exactly what the sparse-reward experiments are designed to test.
Where the conclusion stops: task scope, oracle budgets, and unmeasured regimes
The paper's own framing limits the claim to the tested sparse-reward tasks and reported NRE budgets [1]. Nothing in the supplied evidence shows BaNEL helps when rewards are dense, when a positive sample is available for contrast, or when the reward oracle is cheap enough that novelty bonuses are not bottlenecked. The MNIST, adversarial-attack, and GSM8K-subset tasks are all sequential generation problems with deliberately filtered successes; generalization to other modalities, longer horizons, or continuous control is untested here. The compute-scaling result is also conditional: BaNEL benefits from extra epochs training pφ, so its advantage over novelty bonuses shrinks if that compute is unavailable [1].
Two broader uncertainties remain. First, reward overoptimization is a known failure mode when optimizing against imperfect proxies, with smooth scaling relationships between proxy optimization and gold-reward degradation [7]; BaNEL's failure model is not a reward model, but the same question of what happens under prolonged optimization against a learned signal is unaddressed. Second, sample-efficiency work in adjacent settings shows that high-reward oracles can raise group baselines and complicate advantage estimation [6], and reward-shaping studies show that bounded, well-shaped rewards widen the safe training window [2]. Whether BaNEL's likelihood-ratio filter interacts with those dynamics in the same way is an open question. The honest summary is that BaNEL demonstrates a new mechanism for a narrow but important regime, not a general replacement for reward supervision.
About These Sources
This research page is built on 9 studies (6 peer-reviewed, 3 preprints) — published from 2022 to 2026, 5 from 2024 or later, collectively cited 1,666 times — selected as the most relevant from 13 studies that passed quality screening, drawn from 140 papers retrieved from a database of over 500 million.
Sources used in this answer
BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewards
BaNEL post-trains generative models using only zero-reward samples by fitting an in-loop failure model and steering generation away from it, outperforming count-based and RND novelty bonuses by up to orders of magnitude in success rate on sparse-reward tasks while using fewer reward evaluations.
Reward Shaping to Mitigate Reward Hacking in RLHF
This reward-shaping study establishes that bounded RL rewards with rapid initial growth and gradual convergence stabilize PPO training and mitigate reward hacking, with PAR requiring only a single reference reward.
Episodic Curiosity through Reachability
Episodic Curiosity introduced reachability-based novelty bonuses using episodic memory, outperforming ICM in ViZDoom and DMLab navigational tasks and avoiding the degenerate firing behavior seen in prediction-error curiosity.
Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs
TOPR applies asymmetric tapered importance sampling to REINFORCE, enabling stable off-policy LLM fine-tuning with unified positive and negative examples and showing that the baseline parameter controls the effective proportion of positive examples.
Preference-based Policy Optimization from Sparse-reward Offline Dataset
PREFORL uses contrastive preference learning against failures and synthetically degraded behaviors to avoid value overestimation, achieving state-of-the-art normalized scores on sparse-reward D4RL Adroit and MuJoCo tasks.
Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs
This video-MLLM RL study examines sample efficiency and scalability, noting that a high-reward oracle raises the group baseline and complicates advantage estimation.
Scaling Laws for Reward Model Overoptimization
This scaling-law study measures reward model overoptimization, showing that gold-reward scores degrade smoothly as proxy reward optimization proceeds and that coefficients scale with reward model parameters.
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
RLAIF shows that training reward models on AI-generated preferences achieves performance comparable to RLHF across summarization and dialogue tasks, with d-RLAIF circumventing reward model training entirely.
A Survey of Reinforcement Learning from Human Feedback
This RLHF survey covers the fundamentals of learning from human feedback across control, robotics, and LLMs, examining how algorithms and human feedback interact.