si PddlZddlmZmZmZmZerddlmZGddZdS)N) TYPE_CHECKINGAnyDictUnion)SentenceTransformerceZdZdZdZ ddddeded ed eee eefff d Z d e eefd ed e eeffdZ ddd e ee fd dfdZ ed efdZdS)SentenceEvaluatorzl Base class for all evaluators Extend this class and implement __call__ for custom evaluators. c"d|_d|_dS)a Base class for all evaluators. Notably, this class introduces the ``greater_is_better`` and ``primary_metric`` attributes. The former is a boolean indicating whether a higher evaluation score is better, which is used for choosing the best checkpoint if ``load_best_model_at_end`` is set to ``True`` in the training arguments. The latter is a string indicating the primary metric for the evaluator. This has to be defined whenever the evaluator returns a dictionary of metrics, and the primary metric is the key pointing to the primary metric, i.e. the one that is used for model selection and/or logging. TN)greater_is_betterprimary_metric)selfs e/var/www/icac/venv/lib/python3.11/site-packages/sentence_transformers/evaluation/SentenceEvaluator.py__init__zSentenceEvaluator.__init__s"&"Nmodelr output_pathepochstepsreturncdS)a This is called during training to evaluate the model. It returns a score for the evaluation with a higher score indicating a better result. Args: model: the model to evaluate output_path: path where predictions and metrics are written to epoch: the epoch where the evaluation takes place. This is used for the file prefixes. If this is -1, then we assume evaluation on test data. steps: the steps in the current epoch at time of the evaluation. This is used for the file prefixes. If this is -1, then we assume evaluation at the end of the epoch. Returns: Either a score for the evaluation with a higher score indicating a better result, or a dictionary with scores. If the latter is chosen, then `evaluator.primary_metric` must be defined N)r rrrrs r__call__zSentenceEvaluator.__call__s 2 rmetricsnamecs|Sfd|D}t|dr/|jdzsdz|jz|_|S)Nc(i|]\}}dz|z|S)_r).0keyvaluers r z.:s(MMMzsE4#:#UMMMrr r)itemshasattrr startswith)r rrs `rprefix_name_to_metricsz(SentenceEvaluator.prefix_name_to_metrics7s~ NMMMMW]]__MMM 4) * * C43F3Q3QRVY\R\3]3] C"&*t/B"BD rc<|j||dS)N)model_card_dataset_evaluation_metrics)r rrs r store_metrics_in_model_card_dataz2SentenceEvaluator.store_metrics_in_model_card_data?s! 44T7CCCCCrc|jj} |d}|d|}n#t$rYnwxYwt jdd|S)z Returns a human-readable description of the evaluator: BinaryClassificationEvaluator -> Binary Classification 1. Remove "Evaluator" from the class name 2. Add a space before every capital letter EvaluatorNz([a-z])([A-Z])z \g<1> \g<2>) __class____name__index IndexErrorresub)r class_namer/s r descriptionzSentenceEvaluator.descriptionBsm^,  $$[11E#FUF+JJ    D v' CCCs . ;;)Nrr)r. __module__ __qualname____doc__rstrintrfloatrrr&rr*propertyr4rrrr r s3 # # #df  * 9< LO ]` ud3:&& '    6d3:.>cdSVX]S]N^D6KDVZ[^`c[cVdDimDDDDDSDDDXDDDrr ) r1typingrrrr)sentence_transformers.SentenceTransformerrr rrrr>s 222222222222NMMMMMMIDIDIDIDIDIDIDIDIDIDr