Multi Turn Reinforcement Learning with Amazon Nova Forge

Introduction

Reinforcement learning (RL) can teach AI models behaviors that are difficult to capture through conventional supervised fine-tuning. Instead of learning only from labeled examples, reinforcement fine-tuning (RFT) evaluates model-generated responses and adjusts the model toward behaviors that receive higher rewards.

For multi-turn AI agents, however, defining the right reward is challenging. An agent may need to ask clarifying questions, use tools correctly, recover from mistakes, and eventually complete a task. A reward function that focuses only on the final answer can be too sparse, while poorly designed intermediate rewards can encourage unintended behavior.

Amazon Nova Forge addresses this challenge by enabling developers to create custom reward functions for multi-turn RFT and to execute the reward logic in their own environment via Bring Your Own Orchestration (BYOO).

Understanding Multi-Turn Reinforcement Fine-Tuning

Traditional supervised fine-tuning teaches a model from curated input-output examples. RFT takes a different approach: the model generates candidate responses, a reward function evaluates them, and training increases the likelihood of higher-reward behavior.

In multi-turn RFT, the model is evaluated over an entire trajectory rather than a single response. A trajectory can include several interactions, tool calls, code execution, corrections, and an outcome.

Amazon Nova Forge uses Group Relative Policy Optimization (GRPO). Multiple rollouts are generated for the same prompt, their rewards are compared, and the resulting differences guide model updates. This means the reward function must produce meaningful variation between candidate behaviors.

Designing a Composite Reward

A reliable multi-turn reward generally combines multiple signals rather than relying on one score.

Outcome Rewards

Outcome rewards measure whether the final task was completed. For example, a coding agent could receive a score based on how many hidden unit tests its final implementation passes.

These rewards directly represent the desired result but can be sparse, particularly during early training.

Behavioral Rewards

Behavioral rewards encourage useful intermediate actions. An agent might receive a reward for asking a necessary clarification before executing code or selecting an appropriate tool.

These signals help the model learn behaviors that eventually contribute to successful outcomes.

Penalties

Penalties discourage undesirable strategies such as immediately guessing, repeating similar responses, or getting stuck in a loop.

Combining outcome rewards, behavioral rewards, and penalties allows the model to learn both what to accomplish and how to approach the task.

Example: Teaching an Agent to Clarify Before Coding

AWS demonstrates this approach using a collaborative coding task. The model receives an incomplete programming request while a user simulator holds an important requirement that is revealed only when the model asks a clarification question.

The reward contains four components:

This design prevents a common problem: rewarding the desired behavior only when the final answer is already correct. If correctness is extremely difficult early in training, such a gated reward may rarely activate and provide almost no learning signal.

Running Custom Rewards with Amazon Nova Forge

For multi-turn workloads, Nova Forge supports BYOO. Instead of executing reward logic through a single Lambda invocation, the training system delegates rollouts to a customer-managed environment.

The environment can run on services such as Amazon ECS and manage the multi-turn conversation, user simulation, code execution, and verification. After completing a rollout, it returns an aggregate reward along with optional per-component metrics.

This approach is useful for workloads where conversations or evaluation processes exceed the execution limits of a single function.

Monitoring Reward Quality

A high aggregate reward does not necessarily mean that every reward component is working.

A component that returns the same value for every rollout contributes no useful variation to GRPO. Consequently, it contributes nothing to the advantage or gradient, even if that component has a high weight.

Developers should therefore monitor:

• Mean reward for each component
• Within-group reward variation
• Component contribution to the advantage
• Training transcripts
• Policy and KL behavior
• Effects of removing or changing individual reward components

Testing components independently through ablation can reveal whether a reward term is actually influencing learning.

Security When Evaluating Generated Code

When a reward function executes model-generated code, the output must be treated as untrusted.

Execution environments should be isolated and should not expose credentials or unnecessary network access. Resource limits, temporary workspaces, execution timeouts, and validation of test results should also be implemented.

The verification mechanism should prevent the model from manipulating the evaluation process itself. These safeguards are particularly important because RL intentionally encourages exploration, meaning generated outputs cannot be assumed to be safe or valid.

Conclusion

Custom reward functions are at the center of successful multi-turn reinforcement learning. Amazon Nova Forge gives developers control over how agent behavior is evaluated while providing the infrastructure needed to integrate those rewards into multi-turn RFT workflows.

The key is not simply to create a higher reward, but to design a reward that produces meaningful learning signals. Combining outcome rewards, behavioral signals, and targeted penalties, while monitoring each component independently, can help models learn the intended behavior rather than an unintended shortcut.

For enterprise AI agents, this approach provides a practical path toward training models that can reason, interact, recover, and complete complex multi-step tasks more reliably.

Drop a query if you have any questions regarding Amazon Nova Forge, and we will get back to you quickly.