This class includes only the parameters that are specific to ORPO training. For a full list of training arguments,
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
max_length (`int` or `None`, *optional*, defaults to `1024`):
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
to use the default data collator.
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
Maximum length of the prompt. This argument is required if you want to use the default data collator.
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
Maximum length of the completion. This argument is required if you want to use the default data collator
and your model is an encoder-decoder.
beta (`float`, *optional*, defaults to `0.1`):
Parameter controlling the relative ratio loss weight in the ORPO loss. In the
[paper](https://huggingface.co/papers/2403.07691), it is denoted by λ. In the
[code](https://github.com/xfactlab/orpo), it is denoted by `alpha`.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model.
label_pad_token_id (`int`, *optional*, defaults to `-100`):
Label pad token id. This argument is required if you want to use the default data collator.
padding_value (`int` or `None`, *optional*, defaults to `None`):
Padding value to use. If `None`, the padding value of the tokenizer is used.
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
This argument is required if you want to use the default data collator.
generate_during_eval (`bool`, *optional*, defaults to `False`):
If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
you need to specify if the model returned by the callable is an encoder-decoder model.
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
string.
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrIz8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnorJéréúç-Cëâ6
“d|“d|“d|“d|“d|“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,|“d-| “d.|!“d/|"“d0|#“d1|$“d2|%“d3|&“d4|'“d5|(“d6|)“d7|*“d8|+“d9|,“d:|-“d;|.“d<|/“d=|0“d>|1“d?|2“d@|3“dA|4“dB|5“dC|6“dD|7“dE|8“dF|9“dG|:“dH|;“dI|<“dJ|=“dK|>“dL|?“dM|@“dN|A“dO|B“dP|C“dQ|D“dR|E“dS|F“dT|G“dU|H“dV|I“dW|J“dX|K“dY|L“dZ|M“d[|N“d\|O“d]|P“d^|Q“d_|R“d`|S“da|T“db|U“dc|V“dd|W“de|X“df|Y“dg|Z“dh|[“di|\“dj|]“dk|^“dl|_“dm|`“dn|a“do|b“dp|c“dq|d“dr|e“ds|f“dt|g“du|h“dv|i“dw|j“dx|k“dy|l“dz|m“d{|n“d||o“d}|p“d~|q“d|r“d€|s“d�|t“d‚|u“dƒ|v“d„|w“d…|x“d†|y“d‡|z“dˆ|{“d‰||“dŠ|}“d‹|~“dŒ|“d�|€“dŽ|�“d�|‚“d�|ƒ“d‘|„“d’|…“d“|†“d”|‡“d•|ˆ“d–|‰“d—|Š“d˜|‹“d™|Œ“|�¤Ž|�|_|Ž|_|�|_ dS)›NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rOza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r}r~Úunsloth_training_checkpointsror)Ú cpu_countrJrpÚ
}||d|j…||<qâqÍ|jdk�r dD]}|||jd…||<qùqÍtd|j›�ƒ‚q͈ˆfD]#}t|dƒ||jk�r8dD]}||d|j|j…||<�q&�q‡fdd„dDƒ}‡fdd„dDƒ}|ddd…|d<|jgtˆdƒ|ddtˆdƒ…<|ddd…|d<|jgtˆdƒ|ddtˆdƒ…<|||dœ¡D]\}}|¡D]\}}|dk�r£�q˜|||›|›�<�q˜�q�n[|j|d|jdd�‰|j|d|jdd�‰|j|d|jdd�}ˆd|d<ˆd|d<|d|d<|d |d!<|du�rt|d"ƒ�r|jt|d¡d#�|d$<|jt|d¡d#�|d%<tƒ�rF|D]4}d|v�s|j�r |j}n|d&¡�r*|j}n|d'¡�r2d(}|||g|jt||ƒ||<�q|S))a/Tokenize a single row from a ORPO specific dataset.
At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
we truncate the chosen/rejected.
We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
r‚ÚchosenÚrejectedz prompt should be an str but got FrxcSsi|]
<dictcomp>ßóz4_UnslothORPOTrainer.tokenize_row.<locals>.<dictcomp>z chosen should be an str but got z"rejected should be an str but got r|NcSsg|]\}}||k‘qSrere)r‘ÚaÚbrererfÚ
<listcomp>÷sz4_UnslothORPOTrainer.tokenize_row.<locals>.<listcomp>rOzdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.rzÚ
z#_UnslothORPOTrainer.odds_ratio_lossr\rŸÚaverage_log_probcCs|jdd…|jkrtdƒ‚|s)|dd…dd…f¡}|dd…dd…dd…f}||k}t||kd|¡}t||ƒ}|rI||d¡|d¡S||d¡S)aŽCompute the log probabilities of the given labels under the given logits.
Args:
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
labels:
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
ignored. Shape: (batch_size, sequence_length)
average_log_prob:
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
log probabilities of the (non-masked) tokens.
label_pad_token_id: The label pad token id.
is_encoder_decoder: Whether the model is an encoder-decoder model.
Returns:
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
given logits.
NrIzKLogits (batch and sequence length dim) and labels must have the same shape.rOr)rRr[Úcloner;Úwherer9r®)r\rŸrßrrÚ loss_maskrdrererfÚget_batch_logpsÆs
z#_UnslothORPOTrainer.get_batch_logpscsŠˆj|ˆjˆjˆjˆjjd�}|djd}ˆjr"dˆ|d¡ini}ˆjr+d|d<||df|d d
dœ|¤Ž}|j }‡fdd
„}ˆjrM|d
¡} n|d
¡} |d }
t|
dk| ˆj¡} ||d|…| d|…ƒ}ˆj
||ddˆjˆjd�}|d|…}
||d…}ˆjs¦|d|…dd…dd…f}||d…dd…dd…f}n|d|…}||d…}ˆjr¾|
|||||jfS|
||||fS)zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.
We do this to avoid doing two forward passes, because it's faster for FSDP.
|7}||fS)zXCompute the ORPO loss and other metrics for the given batch of inputs for train or test.NérýÚeval_r‚zrewards/chosenzrewards/rejectedzrewards/accuracieszrewards/marginszlogps/rejectedzlogps/chosenzlogits/rejectedz
||j|jj ƒ}|jj|dd�}|S)zRGenerate samples from the model and reference model for the given batch of inputs.r|r}T)rzr{rÚ do_samplerMN)Úskip_special_tokens)r`r"rXrÄr«r2Úgeneraterr5rMr4Úbatch_decode)r!r0rµÚgenerate_context_managerÚ
¡�$|�|j||dd�\}}Wdƒn1sDwYWdƒn1sSwY|j|dd�|rh|¡ddfS|d|ddœ}‡fd d
„|¡Dƒ} tj| |jj d�} tj| jd|jj d�}
|¡| |
fS)
Na!prediction_step is only implemented for DPODataCollatorWithPadding, and you passed a datacollator that is different than DPODataCollatorWithPadding - you might see unexpected behavior. Alternatively, you can implement your own prediction_step method if you are using a custom data collatorrgÚkeys_to_ignore_at_inferencerýrúeval_logits/chosenúeval_logits/rejected)r!r"csg|]
Nz]model.config.decoder_start_token_id has to be defined. It is usually set to the pad_token_id.rI)rO.rNrO).rz,model.config.pad_token_id has to be defined.r‹)rhr[r*r;ÚfullrRrÍÚ new_zerosràrMÚmasked_fill_)r!rzÚshifted_input_idsrererfrîs
ÿ
z _UnslothORPOTrainer._shift_rightcsL|jjdurt|jjƒj}n |jjd¡d}|j|d�tƒ||¡dS)Nú/rI)Ú
