si'2ddlZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl Z ddlmZmZddlmZddlmZerddlmZejeZGdd eZdS) N) nullcontext) TYPE_CHECKINGCallableDictOptional)average_precision_score ndcg_score)SentenceEvaluator)cos_sim)SentenceTransformerceZdZdZdddedddddf ded ed ed ee j e j ge j fd ed edede ede effd Z ddddededede eeff dZdZdZdZxZS)RerankingEvaluatora This class evaluates a SentenceTransformer model for the task of re-ranking. Given a query and a list of documents, it computes the score [query, doc_i] for all possible documents and sorts them in decreasing order. Then, MRR@10, NDCG@10 and MAP is compute to measure the quality of the ranking. Args: samples (list): A list of dictionaries, where each dictionary represents a sample and has the following keys: - 'query': The search query. - 'positive': A list of positive (relevant) documents. - 'negative': A list of negative (irrelevant) documents. at_k (int, optional): Only consider the top k most similar documents to each query for the evaluation. Defaults to 10. name (str, optional): Name of the evaluator. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. similarity_fct (Callable[[torch.Tensor, torch.Tensor], torch.Tensor], optional): Similarity function between sentence embeddings. By default, cosine similarity. Defaults to cos_sim. batch_size (int, optional): Batch size to compute sentence embeddings. Defaults to 64. show_progress_bar (bool, optional): Show progress bar when computing embeddings. Defaults to False. use_batched_encoding (bool, optional): Whether or not to encode queries and documents in batches for greater speed, or 1-by-1 to save memory. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. mrr_at_k (Optional[int], optional): Deprecated parameter. Please use `at_k` instead. Defaults to None. T@FNat_kname write_csvsimilarity_fct batch_sizeshow_progress_baruse_batched_encoding truncate_dimmrr_at_kc t||_||_| &td| d| |_n||_||_||_||_ ||_ | |_ t|jtr+t|j|_d|jD|_d|rd|zndzd|jdz|_d d d d |jd |jg|_||_d|_dS)Nz?The `mrr_at_k` parameter has been deprecated; please use `at_k=z ` instead.cvg|]6}t|ddkt|ddk4|7S)positivernegativelen.0samples f/var/www/icac/venv/lib/python3.11/site-packages/sentence_transformers/evaluation/RerankingEvaluator.py z/RerankingEvaluator.__init__..MsQ   VJ5G1H1H11L1LQTU[\fUgQhQhklQlQlFQlQlQlr_rz _results_@z.csvepochstepsMAPzMRR@{}zNDCG@{}map)super__init__samplesrloggerwarningrrrrrr isinstancedictlistvaluescsv_fileformat csv_headersrprimary_metric) selfr.rrrrrrrrr __class__s r$r-zRerankingEvaluator.__init__,sV     NNq]eqqq r r r DIIDI,$!2$8!( dlD ) ) 7 3 3 5 566DL  !%    -d0Jd KNj[_[dNjNjNjj    OODI & &   TY ' '  ##r&modelr output_pathr(r)returnc |dkr|dkrd|}n d|d|d}nd}|j|d|jd z }td |jd |d ||}|d }|d}|d} d|jD} d|jD} tdt|jtj | tj | tj | tj | tj | tj | td|dztd|j |dztd|j | dz||j rtj||j} tj| } t'| d| rdndd5}t)j|}| s||j|||||| gdddn #1swxYwYd |d|j |d|j | i}|||j}||||S)a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. r;z after epoch z in epoch z after z stepsrNz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:r+mrrndcgc8g|]}t|dS)rrr!s r$r%z/RerankingEvaluator.__call__..}%LLLVVJ/00LLLr&c8g|]}t|dS)rrr!s r$r%z/RerankingEvaluator.__call__..~rEr&zmQueries: {} Positives: Min {:.1f}, Mean {:.1f}, Max {:.1f} Negatives: Min {:.1f}, Mean {:.1f}, Max {:.1f}z MAP: {:.2f}dzMRR@{}: {:.2f}zNDCG@{}: {:.2f}awzutf-8)newlinemodeencodingzmrr@zndcg@)rr/inforcompute_metricesr.r6r npminmeanmaxrrospathjoinr5isfileopencsvwriterwriterowr7prefix_name_to_metrics store_metrics_in_model_card_data)r9r<r=r(r)out_txtscoresmean_apmean_mrr mean_ndcg num_positives num_negativescsv_pathoutput_file_existsfrYmetricss r$__call__zRerankingEvaluator.__call__\sS B;;{{1%11BuBBUBBBG   ( =):=== =G dtyddZadddeee&&u---%=6N MLt|LLL LLt|LLL  } E EDL!!}%% &&}%%}%% &&}%%     M((377888 $++DIx#~FFGGG %,,TY CHHIII  "t~ "w||K??H!#!9!9 h8J1SPS^efff NjkA)6OOD$4555w) LMMM  N N N N N N N N N N N N N N N 7 49   DI    --gtyAA --eW===sA J11J58J5cd|jr||n||S)a Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )rcompute_metrices_batchedcompute_metrices_individual)r9r<s r$rNz#RerankingEvaluator.compute_metricess8( 9D ) )% 0 0 011%88 r&cXg}g}g}|jtn||j5|d|jDd|j|j}g}|jD]8}||d||d9||d|j|j}dddn #1swxYwYd\} } |jD]w} || } | dz } t| d} t| d}|| | | z|z}| | |zz } | d ks|d kr_| | |}t|j dkr|d }tj | }| }dg| zd g|zz}d }t|d |jD]\}}||r d|dzz }n|||t%|g|g|j |t'||yt)j|}t)j|}t)j|}|||d S) aE Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Ncg|] }|d S)queryr!s r$r%z?RerankingEvaluator.compute_metrices_batched..s<<**GO,, xCCCsBCC!Ccg}g}g}tj|j|j dD]}|d}t|d}t|d}t |dkst |dkr\||z} dgt |zdgt |zz} |jt n||j5||gd |j d } || d |j d } dddn #1swxYwY| | | } t | j dkr| d} tj | }| } d}t!|d|jD]\}}| |r d|dzz }n|||t'| g| g|j |t)| | t+j|}t+j|}t+j|}|||d S)aO Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Samples)disabledescrnrrrrrNTFrprsru)tqdmr.rr3r rrrvrwrrryrzr{r|r}r~rrr rrOrQ)r9r<rrrrrnrrdocsrrrrrrrrr_r`ras r$rkz.RerankingEvaluator.compute_metrices_individuals  $,D**GO,, xCCCs >D**D. 1D. )Nr;r;)__name__ __module__ __qualname____doc__r intstrboolrrzTensorrr-rfloatrhrNrjrk __classcell__)r:s@r$rrsx2OV"'%)&*"&.$.$.$ .$  .$ !%, !=u|!KL .$.$ .$#.$sm.$3-.$.$.$.$.$.$bdfEE*E9<ELOE]`E c5j EEEEN   HDHDHDT:D:D:D:D:D:D:Dr&r)rXloggingrS contextlibrtypingrrrrnumpyrOrzrsklearn.metricsrr 2sentence_transformers.evaluation.SentenceEvaluatorr sentence_transformers.utilr )sentence_transformers.SentenceTransformerr getLoggerrr/rror&r$rs&  """""":::::::::::: ????????PPPPPP......NMMMMMM  8 $ $bDbDbDbDbD*bDbDbDbDbDr&