[i tddlZddlZddlmZddlmZddlZddlmZm Z ddl m Z m Z m Z gdZde dee fd Zde d eedee fd Zd e defd Ze jfd e dedede fdZd e defdZdd e dedzde fdZeded e dede fdZd e defdZd e dejde fdZdS)N)Any) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dps)apply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipesgraphreturnc:t|tSN)_get_all_graph_pipes_helperset)rs R/var/www/icac/venv/lib/python3.11/site-packages/torch/utils/data/graph_settings.pyrrs &ucee 4 44id_cachecg}|D]Z\}\}}||vr |||||t ||[|Sr)itemsaddappendextendr)rrresultsdp_iddatapipe sub_graphs rrrs!G(- II$$) H    Ux   29hGGHHHH Nrrc~t|tp(t|dotj|jS)Nr ) isinstancerhasattrinspectismethodr rs r_is_sharding_datapiper')s> h 5 6 6 *++ 6  X4 5 5rnum_of_instances instance_idcVt|}ddfd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. Nrcp|D]\}}d}t|ry|td|d|tj|j}t |jdkr|n| |}||}||dS)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr' RuntimeErrorr$ signaturer len parameters) r prev_applieddpr appliedsig_helperr)r(r-s rr7zapply_sharding.._helper=s"\\^^ ( (MB G$R(( +&d7Cdd_add '(9::s~&&**%%&6 DDDD%%(+n&& GIw ' ' ' '' ( (rr)rN)r )rr(r)r-rr7s ``` @rr r 0s[  " "E((((((((((, GENNN Orct|doAt|do1tj|jotj|jS)N set_shuffleset_seed)r#r$r%r9r:r&s r_is_shuffle_datapiper;XsT-(( 0 Hj ) ) 0  X1 2 2 0  X. / / rshufflec||St|}t|}d|D}|s/|r-tjdd|}|g}|D]}|||S)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) Nc0g|]}t||S)r;).0pipes r z*apply_shuffle_settings..qs&JJJ$/CD/I/IJJJJrz`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.) stacklevel)r rwarningswarnr<r9)rr<r all_pipes shufflersshufflers rr r as  " "E#E**IJJ)JJJI       [    ##%%  &&W%%%% Orz`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)categoryrngc"t||Sr)r )rrKs rr r s Xs + ++rcTt|dotj|jS)Nr:)r#r$r%r:r&s r_is_random_datapiperNs% 8Z ( ( PW-=h>O-P-PPrct|}t|}t}g}|D]Z}t||vrt |r7|||t|[|D]j}ttj dtj  | }| |k|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds r?)dtype) generator)r rridrNrrinttorchemptyint64random_itemr:)rrKrrGcacherandom_datapipesrA random_seeds rr r s  " "E#E**I EEE   d88u    t $ $  # #D ) ) ) IIbhh    ## K%+ . . . 6 6 6 E E J J L L   k"""" Orr)r$rEtypingrtyping_extensionsrrT(torch.utils.data.datapipes.iter.shardingrrtorch.utils.data.graphrrr __all__listrrrSrboolr'DEFAULTr r;r FutureWarningr rN Generatorr r?rrrfsc(((((( IHHHHHHHHH   5}5h5555   $'H  (^    H'. %%%%%  %%%%P8Xt xD .  ,,,,,,  ,Q(QtQQQQu8r