Are action-based preferences obligatory? One of many key components of ACT is that the contrastive pairs spotlight variations between conversational actions. In “ACT w/ Random Actions”, we moreover study the significance of motion choice by randomly sampling each the profitable and shedding motion when setting up the choice pair, and observe this underperforms regular ACT.
Do we want on-policy sampling? In “ACT w/o on-policy sampling”, we study the significance of on-policy sampling by evaluating regular off-policy DPO on the dataset as constructed in Part 1. Whereas we do observe some enhancements over SFT (e.g., from 69.0 to 74.8 Macro F1), the general enhancements are a lot bigger when utilizing on-policy sampling as with full ACT. This can be attributable to the truth that the off-policy damaging responses should not assured to lie within the language manifold of the coverage mannequin, and distribution shift could also be too troublesome to beat with off-policy studying.
Is trajectory simulation obligatory? ACT is better-aligned with multi-turn conversations attributable to its trajectory simulation. With out multi-turn simulation, our strategy might be seen equally to on-policy DPO variants like IRPO, however with a conversation-specific reward sign which accounts for dialog actions and process heuristics. In “ACT w/ sampling w/o simulation”, we discover that this trajectory-level simulation is crucial to bettering multi-turn efficiency, particularly the coverage mannequin’s skill to purpose about its personal clarification questions.
Is ACT mannequin agnostic? The bottom mannequin in our fundamental experiments, Zephyr, is obtained by aligning Mistral. In “ACT with unaligned basis fashions” we observe a efficiency hole of 6.5 Motion F1 and 4.3 Trajectory F1 after ACT tuning for the 2 fashions. Nevertheless, our outcomes show ACT can enhance efficiency no matter pre-existing alignment with human suggestions, though it may possibly assist as an improved mannequin initialization. General, we discover that bettering base mannequin efficiency with ACT is mannequin agnostic.
