si`ddlmZmZddlmZmZmZmZddlZeGddZ dS)) dataclassfield)AnyCallableDictListNceZdZUdZeed<edZee ed<dee e e fde e e j ffdZd S) SentenceTransformerDataCollatora!Collator for a SentenceTransformers model. This encodes the text columns to {column}_input_ids and {column}_attention_mask columns. This works with the two text dataset that is used as the example in the training overview: https://www.sbert.net/docs/training/overview.html tokenize_fnc ddgS)NlabelscorerV/var/www/icac/venv/lib/python3.11/site-packages/sentence_transformers/data_collator.pyz(SentenceTransformerDataCollator.s GWCUr)default_factoryvalid_label_columnsfeaturesreturnct|d}i}d|vr&|d|dd|d<|jD]@|vr:t jfd|D|d<|nA|D]G|fd|D}|D]\}}||d|<H|S)Nr dataset_namec g|] }| Srr).0row label_columns r z.s.U.U.USs.%s)J)J)J##f+)J)J)Jr_)listkeysremovertorchtensorr items) selfrcolumnsbatch tokenizedkeyvaluerrs @@r__call__z(SentenceTransformerDataCollator.__call__s6x{''))** W $ $ NN> * * *$,QK$?E. !!4  Lw&&!&.U.U.U.UH.U.U.U!V!Vg|,,,'  1 1F(()J)J)J)J)J)J)JKKI'oo// 1 1 U+0''#''(( 1 rN)__name__ __module__ __qualname____doc__r__annotations__rrrstrrrr$Tensorr-rrrr r s %*U;U;U%V%V%VcVVVd38n!5$sEL?P:Qrr ) dataclassesrrtypingrrrrr$r rrrr7s((((((((,,,,,,,,,,,,             r