AGENT SKILL

preference-optimization

Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.

License 许可
MIT
最近更新
2026-08-19
Stars
★ 2
Forks
0

01 前言 Front Matter

description
Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.

02 资源文件 1

03 SKILL.md 正文

Preference Optimization

This skill assumes finetuning-method-selection already routed here because the data shape is preference pairs or unpaired thumbs-up/down feedback, not demonstrations (that's lora-qlora-recipes) or a verifiable reward signal (that's grpo-rlvr-training). What follows is method selection among the DPO family, the evidence for how much that selection actually matters, the production training pattern, and how to build the pairs in the first place.

Input: a routing decision (preference optimization) plus preference pairs or unpaired feedback, usually from an SFT checkpoint. Output format: a validated method choice plus a config — the kwarg values in references/method-configs.md, not free-form advice — that llm-finetuning-training-engineer consumes directly.

Method Selection

Data shape Method Key parameters
Preference pairs, default case DPO β=0.1, LR 5e-7–1e-6, 1–2 epochs
Memory-bound or no SFT checkpoint ORPO reference-free, fused SFT+preference in one loss
Unpaired thumbs-up/down KTO binary label per example, no pairing needed
Length bias observed, sweep budget available SimPO reference-free; see sweep grid below
  • DPO is the safe default. Use β=0.1 and a learning rate of 5e-7 to 1e-6 for 1–2 epochs. This LR is lower than the SFT LR that produced the checkpoint being aligned — porting an SFT- scale LR into a DPO run is the most common misconfiguration here, not an edge case.
  • ORPO routes in when memory is the constraint, or when there's no separate SFT checkpoint to start from — it's reference-free and fuses the SFT and preference objectives into one loss, skipping the separate SFT pass and the reference-model memory cost DPO carries.
  • KTO routes in when feedback is unpaired binary signal (thumbs-up/down) rather than matched preference pairs — don't force unpaired feedback into synthetic pairs to use DPO instead.
  • SimPO fixes DPO's length bias but only pays off with disciplined sweeping — its published gains are a ceiling reported under a tuned sweep, not a baseline any single config will reproduce. Route here only when there's sweep budget; use DPO instead if there isn't.
  • Classic RLHF (reward model + PPO) is retired outside frontier labs. Don't reach for it in a production pipeline — every method above is cheaper and better-supported for the same data shapes.

Worked Examples

  • "We have an SFT checkpoint and clean paired preference data, no length-bias complaints yet." → default case → DPO at β=0.1.
  • "Reviewers click thumbs-up/down per response; nothing is paired." → unpaired signal → KTO, not DPO — don't synthesize pairs to force DPO onto unpaired data.
  • "GPU budget doesn't cover a separate SFT pass plus a DPO reference model." → memory-bound, no separate checkpoint → ORPO.
  • "DPO output favors longer answers regardless of quality, and there's time to run a sweep." → length bias plus sweep budget → SimPO. Skip it if the sweep budget isn't actually there.

The Low-Leverage Truth

A 2026 240-H100-run study (arXiv 2603.19335) is the load-bearing evidence behind the table above: loss-function choice is worth roughly 1 percentage point of leverage, model scale is worth roughly 50. Zero of 20 DPO variants tested beat vanilla DPO. Rankings also invert with scale — a variant that wins in a small pilot can lose at deployment size.

Two practical consequences:

  • Don't spend a routing decision agonizing over DPO-variant bake-offs. The table above is sufficient; deeper variant selection is low-leverage compared to data quality and scale.
  • Validate at deployment scale before trusting a ranking. A method comparison run on a small pilot model doesn't transfer to the production size class — re-check the winner once scale changes.

This is also why the Method Selection table above is deliberately short: it encodes the ~1pp lever, not a ranking of DPO variants that the same study shows doesn't hold up across scale. Treat any variant-selection advice that isn't in that table — including advice that claims a specific variant "wins" — as unproven until it's been validated at the target deployment size.

Production Pattern: Iterative On-Policy DPO

A single offline DPO pass on a static preference dataset is a starting point, not the production pattern. The policy drifts away from the distribution the pairs were sampled from as training proceeds, and a static dataset goes stale against that drift. Production pipelines run DPO iteratively and on-policy instead:

  1. Sample completions from the current policy checkpoint.
  2. Score or rank the completions (reward model, judge, or task grader).
  3. Run a DPO pass using the current checkpoint as the reference model.
  4. The resulting checkpoint becomes both the new policy and the new reference for the next round.

Repeat. Each round's reference model is the prior round's output, not a fixed initial checkpoint — that's what keeps the preference signal on-policy instead of scoring against an increasingly stale distribution.

A single-pass DPO run is still a reasonable first iteration — it just isn't the whole pipeline. Plan for at least one more round once the first checkpoint exists, rather than treating pass one as the finished artifact.

Pair Construction

Build DPO/ORPO pairs from same-task passing-vs-failing trajectories — two attempts at the same underlying task, not unrelated best-and-worst examples pulled from different tasks. Within that trajectory set, select the rejected member at μ−2σ of the reward distribution, never the minimum. Naive best-vs-worst pair construction (max reward vs. absolute minimum) degrades as scale increases; the μ−2σ selection is more robust to the same scale sensitivity the low-leverage study surfaced above.

sorted_by_reward = sort(trajectories, key=reward)
chosen   = sorted_by_reward[-1]                # highest reward
mu, sigma = mean(rewards), stdev(rewards)
rejected = closest(sorted_by_reward, mu - 2 * sigma)
# NOT sorted_by_reward[0] — the absolute minimum
# is the naive best-vs-worst construction that
# degrades as scale increases.

For the mechanics of turning graded traces into these pairs — including rejection sampling and judge-scored delta selection — see trace-to-training-data.

References

Complete TRL config blocks per method — DPOConfig, ORPOConfig, KTOConfig, and the SimPO sweep grid — plus Unsloth wrappers and a catastrophic-forgetting note live in references/method-configs.md. Those configs use the same current-TRL API conventions established in lora-qlora-recipes's references/unsloth-trl-mapping.md (processing_class, not tokenizer=).

references/method-configs.md also carries the catastrophic-forgetting note: a too-high learning rate is the usual cause when a preference-tuned checkpoint loses general capability, and the fix is almost always to drop the LR toward the low end of the range in the Method Selection table above before reaching for any other remediation.

Related skills: finetuning-method-selection routes here once preference pairs or unpaired feedback exist; lora-qlora-recipes produces the SFT checkpoint DPO/KTO/SimPO align (ORPO's fused path can skip it); trace-to-training-data converts passing/failing trajectories into the pairs this skill's Pair Construction section consumes.

04 同仓库更多 Skills

Reihantt6
Reihantt6/sns-myagent

youtube-transcript

Extract transcripts from YouTube videos. Use when the user asks for a transcript, subtitles, or captions of a YouTube video and provides a YouTube URL (youtube.com/watch?v=, youtu.be/, or similar). Supports output with or without timestamps.

★ 2
Reihantt6
Reihantt6/sns-myagent

youtube-thumbnail-design

YouTube thumbnail design with specific dimensions, contrast rules, and mobile preview optimization. Covers safe zones, text placement, face expression psychology, and A/B testing. Use for: YouTube thumbnails, video cover images, click-through optimization. Triggers: youtube thumbnail, thumbnail design, video thumbnail, click through rate, ctr optimization, youtube cover, video cover image, thumbnail maker, thumbnail tips, youtube design, video preview image

★ 2
Reihantt6
Reihantt6/sns-myagent

youtube-short-maker

Make YouTube Shorts that hold attention with Pexo — vertical 9:16, fast-paced, hook in the first 3 seconds. Describe your topic and Pexo writes the script, generates the shots, picks the models, and assembles a finished Short with music and captions. Use for YouTube Shorts: "youtube short", "make a youtube short", "shorts video", "short-form video". NOT for long-form or landscape video.

★ 2
Reihantt6
Reihantt6/sns-myagent

xcode-project-setup

Safely modifies Xcode projects (.pbxproj) to add Swift Packages and link files. Use this skill whenever an iOS project needs dependencies installed (e.g. Firebase, Alamofire).

★ 2
Reihantt6
Reihantt6/sns-myagent

x402

Set up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key. Two setups it works out up front — "just use it" (set up a wallet so you or Claude Code can run cloud browser tasks paid from the wallet — Claude writes and runs throwaway scripts, nothing touches your codebase) or "build it in" (install the SDK and write the key + code into your project). Walks through wallet setup, funding, .env, and a ~$1 test run. Use when the user asks about x402, pay-per-use, USDC payments, or wants Browser Use Cloud without an API key. For the free-tier signup (reverse-CAPTCHA → API key), use `browser-use cloud signup` or the `cloud` skill instead.

★ 2
Reihantt6
Reihantt6/sns-myagent

x-twitter-scraper

Build GitHub Copilot workflows with Xquik X API SDKs, REST endpoints, hosted Apify Actor runs, MCP tools, TweetClaw OpenClaw plugin installs, signed webhooks, tweet search, user lookup, follower exports, media actions, and agent automation.

★ 2