siddlZddlZddlmZmZmZmZmZmZddl m cm Z ddl mZm Z ddlmZddlmZddlmZGddZGd d e jZdS) N)AnyDictIterableListOptionalUnion)Tensornn)SentenceTransformer)CachedGISTEmbedLoss)"CachedMultipleNegativesRankingLossc^eZdZd dZd dZdedefdZdeeefdeeeffdZ dS) ForwardDecoratorreturnNcL||_d|_g|_d|_d|_dSNr)fndimcache cache_dimidx)selfrs ^/var/www/icac/venv/lib/python3.11/site-packages/sentence_transformers/losses/MatryoshkaLoss.py__init__zForwardDecorator.__init__s) c"||_d|_dSr)rr)rrs rset_dimzForwardDecorator.set_dimsrtensorc|jd}|j|krtd|jd||dd|jf}tj|dd}|S)Nz Dimension zL in matryoshka_dims cannot be greater than the model's embedding dimension: .)pr)shaper ValueErrorF normalize)rr tensor_dims rshrinkzForwardDecorator.shrinkst\"% 8j TXs} ZtxZ(Vqb111 rfeaturesch|j|j|jkr<||}|j||j|_n|j|j}||d|d<||d|d<|xjdz c_|S)Ntoken_embeddingssentence_embedding)rrrrappendrr()rr)outputs r__call__zForwardDecorator.__call__$s > !T^tx%?%?WWX&&F J  f % % %!XDNNZ)F%)[[8J1K%L%L!"'+{{6:N3O'P'P#$ A  r)rN) __name__ __module__ __qualname__rrr r(rstrr0rrrr sV c6k!2 tCK7H      rrc eZdZ ddedejdeedeee e efdeddf fd Z d e e eefd edefd Zde eeffd ZedefdZxZS)MatryoshkaLossNr modellossmatryoshka_dimsmatryoshka_weightsn_dims_per_steprct||_||_t |t rt jddt |trt jdd|dgt|z}t||}tt|dd \|_ |_ ||_dS) a The MatryoshkaLoss can be seen as a loss *modifier* that allows you to use other loss functions at various different embedding dimensions. This is useful for when you want to train a model where users have the option to lower the embedding dimension to improve their embedding comparison speed and costs. Args: model: SentenceTransformer model loss: The loss function to be used, e.g. :class:`MultipleNegativesRankingLoss`, :class:`CoSENTLoss`, etc. matryoshka_dims: A list of embedding dimensions to be used for the loss function, e.g. [768, 512, 256, 128, 64]. matryoshka_weights: A list of weights to be used for the loss function, e.g. [1, 1, 1, 1, 1]. If None, then the weights will be set to 1 for all dimensions. n_dims_per_step: The number of dimensions to use per step. If -1, then all dimensions are used. If > 0, then a random sample of n_dims_per_step dimensions are used per step. The default value is -1. References: - The concept was introduced in this paper: https://arxiv.org/abs/2205.13147 - `Matryoshka Embeddings <../../examples/training/matryoshka/README.html>`_ Requirements: 1. The base loss cannot be :class:`CachedMultipleNegativesRankingLoss` or :class:`CachedGISTEmbedLoss`. Relations: - :class:`Matryoshka2dLoss` uses this loss in combination with :class:`AdaptiveLayerLoss` which allows for layer reduction for faster inference. Input: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | any | any | +---------------------------------------+--------+ Example: :: from sentence_transformers import SentenceTransformer, losses, InputExample from torch.utils.data import DataLoader model = SentenceTransformer("microsoft/mpnet-base") train_examples = [ InputExample(texts=['Anchor 1', 'Positive 1']), InputExample(texts=['Anchor 2', 'Positive 2']), ] train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32) train_loss = losses.MultipleNegativesRankingLoss(model=model) train_loss = losses.MatryoshkaLoss(model, train_loss, [768, 512, 256, 128, 64]) model.fit( [(train_dataloader, train_loss)], epochs=10, ) zIMatryoshkaLoss is not compatible with CachedMultipleNegativesRankingLoss.r!) stacklevelz:MatryoshkaLoss is not compatible with CachedGISTEmbedLoss.Nr-c|dSrr5)xs rz)MatryoshkaLoss.__init__..s `abc`drT)keyreverse)superrr8r9 isinstancer warningswarnr lenzipsortedr:r;r<)rr8r9r:r;r< dims_weights __class__s rrzMatryoshkaLoss.__init__4sB   d> ? ? u Mers t t t t d/ 0 0 f MVcd e e e e  %"#s?';';!; ?,>?? 8;VLVdVdnr=s=s=s8t5d5.rsentence_featureslabelsc|jj} t|}||j_tt |j}|jdkr2|jt |krtj||j}d}|D]M}|j|}|j |} | ||| | ||zz }N ||j_n#||j_wxYw|S)Nrg) r8forwardrrangerHr:r<randomsampler;rr9) rrMrNoriginal_forwarddecorated_forward dim_indicesr9rrweights rrPzMatryoshkaLoss.forwards:- 2 01A B B !2DJ D$8 9 9::K#a''D,@3{CSCS,S,S$mK9MNN D" F F*3/05!))#...+ris ================555555PPPPPPnnnnnn########LxxxxxRYxxxxxr