9i>fddlmZ ddlZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z ddlmZmZmZmZddlmZmZmZmZddlZddlmZd d lmZd d lmZerddlZdd lm Z d d l!m"Z"d dl#m$Z$d dlm%Z%ddZ&ej'e(Z)ej*+e(dZ,Gdde-Z.Gdde.Z/Gdde.Z0Gdde.Z1Gdde1Z2Gdd e1Z3Gd!d"e1Z4Gd#d$e1Z5Gd%d&e.Z6Gd'd(e.Z7Gd)d*e.Z8Gd+d,e.Z9Gd-d.e.Z:Gd/d0e:Z;Gd1d2e.Z<Gd3d4e<Z=Gd5d6e.Z>Gd7d8e<Z?Gd9d:e<Z@Gd;de<ZBGd?d@e?ZCGdAdBe ZDGdCdDe<ZEGdEdFe.ZFGdGdHe<ZGGdIdJe.ZHGdKdLe.ZIGdMdNe.ZJGdOdPeKZLGdQdReKZMGdSdTe.ZNGdUdVe.ZOGdWdXeOZPGdYdZeOZQGd[d\eQZRGd]d^eQZSGd_d`eOZTGdadbeOZUGdcddeOZVGdedfeOZWGdgdheOZXeYePeZeQe[eRe\eTe]eSe^eUe-eVe_eWe`eXi ZaddlZbdddmdduZcddxZdejejfjgejejfjhejejfjiejejfjjejejfjkfZlddZmddZne d ddZoddZpeqZrerddddZsGddZtdddZuddZvddZw dddZxddZyddZzddZ{ dddZ|ddZ}ddZ~ddZ dddZdS)) annotationsN)autoEnum) lru_cache)Path) extract_stack format_exc format_list StackSummary)AnyNoReturnOptional TYPE_CHECKING)get_file_path_2)config)counters) CompileId)DynamoTracerOutput)InstructionTranslatorBase)DynamoFrameType case_namestrreturnc4d|ddzS)NzhFor more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#_-)replace)rs D/var/www/icac/venv/lib/python3.11/site-packages/torch/_dynamo/exc.pyexportdb_error_messager 9s% I   C % % & graph_breaksc eZdZdfd ZxZS)TorchDynamoExceptionargsr kwargsrNonecHtj|i|d|_dSN)super__init___torch_dynamo_tracer_outputselfr%r& __class__s rr+zTorchDynamoException.__init__Fs-$)&)))IM(((r!r%r r&r rr'__name__ __module__ __qualname__r+ __classcell__r/s@rr$r$EsGNNNNNNNNNNr!r$ceZdZdS)InternalTorchDynamoErrorNr2r3r4r!rr8r8KDr!r8ceZdZdS)ResumePrologueTracingErrorNr9r:r!rr=r=Or;r!r=c2eZdZUded<ddd fd ZxZS) RestartAnalysis Optional[str]restart_reasonN)rAr%r rr'cB||_tj|dSr))rAr*r+)r.rAr%r/s rr+zRestartAnalysis.__init__Vs%,$r!)r%r rAr@rr'r2r3r4__annotations__r+r5r6s@rr?r?SsX!!!!CG            r!r?ceZdZdS)SpeculationRestartAnalysisNr9r:r!rrFrF[r;r!rFceZdZdS)UnspecializeRestartAnalysisNr9r:r!rrHrH_r;r!rHceZdZdS) CompileCollectiveRestartAnalysisNr9r:r!rrJrJcr;r!rJceZdZdS)TensorifyScalarRestartAnalysisNr9r:r!rrLrLgr;r!rLceZdZdS) SkipFrameNr9r:r!rrNrNkr;r!rNceZdZdS)TorchRuntimeErrorNr9r:r!rrPrPor;r!rPc eZdZdfd ZxZS)InvalidBackendnamerrr'cRtd|ddS)NzInvalid backend: z=, see `torch._dynamo.list_backends()` for available backends.)r*r+)r.rSr/s rr+zInvalidBackend.__init__ts7  e e e e     r!)rSrrr'r1r6s@rrRrRss=          r!rRc eZdZdfd ZxZS) ResetRequiredrr'cnttjddS)Nz Must call `torch._dynamo.reset()` before changing backends. Detected two calls to `torch.compile()` with a different backend compiler arguments. )r*r+textwrapdedent)r.r/s rr+zResetRequired.__init__{s=  O       r!rr'r1r6s@rrVrVzs=          r!rVc(eZdZd fd Zd d ZxZS) ShortenTracebackr%r first_useful_frameOptional[types.FrameType]r&rr'cHtj|i|||_dSr))r*r+r])r.r]r%r&r/s rr+zShortenTraceback.__init__s/ $)&)))"4r! typing.Selfc|j}|j| tjr|S|j|jur!|j}| Jd|j|ju!||S)Nz#internal error, please report a bug) __traceback__r]rverbosetb_frametb_nextwith_traceback)r.tbs rremove_dynamo_framesz%ShortenTraceback.remove_dynamo_framessq    " *bjFNjKk!888B>>#H>>>k!888""2&&&r!)r%r r]r^r&r rr')rr`)r2r3r4r+rhr5r6s@rr\r\sQ555555 ''''''''r!r\c eZdZd fd ZxZS) BackendCompilerFailed backend_fnr inner_exception Exceptionr]r^rr'ct|dd|_||_d|jdt|jd|}t ||dS)Nr2?zbackend=z raised: z: )r])getattr backend_namerltyper2r*r+)r.rkrlr]msgr/s rr+zBackendCompilerFailed.__init__sq $J C@@.k*kk_8M8M8VkkZikk 1CDDDDDr!)rkr rlrmr]r^rr'r1r6s@rrjrjsG E E E E E E E E E Er!rjc:eZdZddddfd Zdd ZdddZxZS) UnsupportedN)r real_stackrsrrr@rvNone | StackSummaryrr'ct||s#tjj}||_||_d|_| ||_ dSr)) r*r+torch_guardsTracingContextrrvrscategory add_to_statsr)r.rsrrvr/s rr+zUnsupported.__init__sn  F5CCEEJ$'+  (1r!c|jJt|j|jxxdzcc<t|j|jdkrt|j|j=dSdS)Nrrr|rrsr.s rremove_from_statszUnsupported.remove_from_statssm}((()))Q.))) DM "48 , 1 1'111 2 1r! unimplementedr|cT||_t||jxxdz cc<dS)Nrr)r.r|s rr}zUnsupported.add_to_statss2  48$$$)$$$$$r!)rsrrr@rvrwrr'rZ)r)r|rrr')r2r3r4r+rr}r5r6s@rrurus| $(*. 22222222 2222 *********r!ruceZdZdS)$UnknownPropertiesDuringBackwardTraceNr9r:r!rrrr;r!rceZdZdS)RecompileErrorNr9r:r!rrrr;r!rceZdZdS)ArgsMismatchErrorNr9r:r!rrrr;r!rceZdZdS)AttributeMutationErrorNr9r:r!rrrr;r!rceZdZdS)InfiniteGeneratorErrorNr9r:r!rrrDr!rceZdZdS)SideEffectsErrorNr9r:r!rrrr;r!rceZdZdZdS)CondOpArgsMismatchErrorz? Internal error from cond() due to arguments mismatch. N)r2r3r4__doc__r:r!rrrsr!rceZdZeZeZeZeZeZeZ eZ eZ dS) UserErrorTypeN) r2r3r4rDYNAMIC_CONTROL_FLOW ANTI_PATTERNSTANDARD_LIBRARYCONSTRAINT_VIOLATION DYNAMIC_DIM INVALID_INPUTINVALID_OUTPUT*UNSUPPORTED_ALIASED_MUTATED_DYNAMIC_INPUTSr:r!rrrsl466466Ltvv466$&&KDFFMTVVN15...r!rc$eZdZ d d fd ZxZS) UserErrorN error_typerrsrrr@rr'c|It|tsJ|dr|dz }n|dz }|t|z }t |||_||_dS)aG Type of errors that would be valid in Eager, but not supported in TorchDynamo. The error message should tell user about next actions. error_type: Type of user error msg: Actionable error message case_name: (Optional) Unique name (snake case) for the usage example in exportdb. N.  ) isinstancerendswithr r*r+rmessage)r.rrsrr/s rr+zUserError.__init__s  i-- - - -||C   s t  ))44 4C $ r!r))rrrsrrr@rr'r1r6s@rrrsENRr!rceZdZdS)SkipCodeRecursiveExceptionNr9r:r!rrrr;r!rceZdZdS)RecompileLimitExceededNr9r:r!rrrr;r!rceZdZddZdS)StepUnsupportedrr'cVtjj|_dSr))ryrzr{rrvrs rr+zStepUnsupported.__init__ s-6DDFFr!NrZ)r2r3r4r+r:r!rrr s.GGGGGGr!rceZdZdS)UnsafeScriptObjectErrorNr9r:r!rrrr;r!rc"eZdZd d fd ZxZS) UncapturedHigherOrderOpErrorNrsrrvOptional[StackSummary]rr'ct|||_||n"tjj|_dSr))r*r+rsryrzr{rrv)r.rsrvr/s rr+z%UncapturedHigherOrderOpError.__init__sP % J-;;== r!r))rsrrvrrr'r1r6s@rrrsB           r!rceZdZdS)IncorrectUsageNr9r:r!rrrr;r!rceZdZdS)FailOnRecompileLimitHitNr9r:r!rrr&r;r!rceZdZdS) PackageErrorNr9r:r!rrr*r;r!rc eZdZdfd ZxZS)ObservedExceptionr%r r&rr'ctj|i|tjj|_dSr))r*r+ryrzr{rrvr-s rr+zObservedException.__init__0s;$)&)))(- (D(R(R(T(Tr!r0r1r6s@rrr.sGUUUUUUUUUUr!rc,eZdZUded<d fd ZxZS) ObservedUserStopIterationz Optional[Any]valuer%r r&rr'ctdt|dkr|d|_dSd|_dS)Nzunhandled `raise StopIteration`r)r*r+lenrr-s rr+z"ObservedUserStopIteration.__init__;sE :;;; t99q==aDJJJDJJJr!r0rCr6s@rrr5sKr!rceZdZdS)ObservedLookupErrorNr9r:r!rrrCrr!rceZdZdS)ObservedIndexErrorNr9r:r!rrrHrr!rceZdZdS)ObservedKeyErrorNr9r:r!rrrMrr!rceZdZdS)ObservedGeneratorExitNr9r:r!rrrRr;r!rceZdZdS)ObservedAttributeErrorNr9r:r!rrrVrr!rceZdZdS)ObservedRuntimeErrorNr9r:r!rrr[rr!rceZdZdS)ObservedNotImplementedErrorNr9r:r!rrr`r;r!rceZdZdS)ObservedTypeErrorNr9r:r!rrrdrr!rexc_typetype[Exception]type[ObservedException]c|tvrAt|dt|}td|dtfit|<t|S)Nr2ObservedError)observed_exception_maprprrrr)rrSs rget_dynamo_observed_exceptionrvs]---xS]];;+/ "t " " "%6$8", , x( "( ++r!)r%r&txrr%Optional[list[Any]]r&Optional[dict[str, Any]]r cddlm}||||pg|pi}|j|t |}|r|||)Nr)BuiltinVariable) variablesr call_function exn_vt_stackset_current_exceptionr)rrr%r&r exception_vt raised_excs rraise_observed_exceptionrs+*****#?8,,::2tzr6z,unimplemented_with_warning..s/   r!cSr)r:)graph_break_msgsrrz,unimplemented_with_warning..s?r!) metadata_fn payload_fnz%sT)rrrrfrom_exc log_warning)format_error_msg_verbosery_loggingtrace_structuredgraph_breaks_logdebugr)rrrrrr_unimplementedrs @runimplemented_with_warningrs /q$77O N##  +*** $4111"NN r!ctj|d}dd|D}tj|d}|d|d|d|d}|S)N rc3jK|].}dtj|dzV/dS)z Hint: rN)rXindentlstrip).0hints r z-format_graph_break_message..sP@D X_T62299;;;r!z Explanation: z Developer debug context: )rXr r join)rrrr hints_strrss rformat_graph_break_messagers /+v66==??K HMIogv..5577G         $    C Jr!)maxsizedict[str, Any]c ttj}t dt |d}t |5}tj|}dddn #1swxYwYn,#t$rt di}YnwxYwi}| D]\}}|D] }|||d<|S)z Loads the gb_type to gb_id map from the graph break registry from JSON file with caching. Includes historical gb_type (mapping behavior of duplicate gb_types with different gb_ids is undefined). zgraph_break_registry.jsonNz!Error accessing the registry fileGb_type) r__file__resolveparentrropenjsonloadrmlog exceptionitems) script_dir registry_pathfregistrymappingkventrys r_load_gb_type_to_gb_id_mapr's6 (^^++--4 ' J!<  -  $Ay||H $ $ $ $ $ $ $ $ $ $ $ $ $ $ $  9:::G  **1 * *E()GE)$ % % * Ns6ABA;/ B;A??BA?B&B0/B0r@cpd}t}||vr!|d||ddSdS)z Retrieves the GBID documentation link for a given graph break type. Args: gb_type: The graph break type to look up. Returns: A string containing the documentation URL if found, otherwise None. z;https://meta-pytorch.github.io/compile-graph-break-site/gb/gbGBz.htmlN)r'r )rGRAPH_BREAK_SITE_URLgb_type_to_gb_id_maps rget_gbid_documentation_linkr-s[ F677&&&# X X';G'D'K'KD'Q'Q X X X  4r!F)rrrrboolct||||}t|}|r|d|z }|rt||tur+d}t |dr|j}t|||t|)a Called within dynamo to cause a graph break. Args: gb_type: Context-free graph break type. It should be a short string without any information specific to the tracing context (i.e. no dynamically-generated strings) context: Developer context for the graph break. It can contain tracing context/dynamic strings. explanation: User-facing context-dependent explanation for the graph break. Can be dynamic. hints: List of user-facing hints for the graph break. z9 For more details about this graph break, please visit: Nrv)rv)rr-rwarning_NOTHINGhasattrrvru) rrrrrrrsdocumentation_linkpast_real_stacks rrr+s& %Wg{E J JC4W==a `L^``` Cx 8\ * * 2&1O#/:::H c  r!c&eZdZd dZd dZd dZd S) KeyErrorMsgrr rr'c||_dSr))r)r.rs rr+zKeyErrorMsg.__init__Rs  r!rc*t|jSr))rrrs r__str__zKeyErrorMsg.__str__Us4:r!c*|Sr))r9rs r__repr__zKeyErrorMsg.__repr__Xs||~~r!N)rr rr')rr)r2r3r4r+r9r;r:r!rr6r6QsPr!r6rexcrsexportc|ddl}d|_t|}|Nt|dkr;|d|_|dd||z }t jr&t|dr|d|j d|j dz }t j st|d r|d z }t|d r^t|j d rIt|j d r!|d|j j d|j j dz }n|d|j j dz }t|jdkrdnt|jd}t!|t"r*t%||zf|jddz|_dS||z}|f|jddz|_dS)Nrz from user code: rrecord_filenamez! Last frame execution written to zD. To run only this frame while debugging, run torch._dynamo.replay('z'). rvz Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo" rl minifier_path buck_commandz Minifier script written to zW. Run this buck command to find the smallest traced graph which reproduces this error: rzQ. Run this script to find the smallest traced graph which reproduces this error. r) tracebackinnermost_user_frame_summaryget_real_stackrrr rreplay_record_enabledr2r@rcrlrArBr%rrKeyErrorr6)r<rsr=rCrvold_msgnew_msgs raugment_exc_messagerJ\s,'+C$$$J#j//A"5"5+5b>( Rbggi.C.CJ.O.O&P&PRRR # 5F(G(G   31D 3 3+ 3 3 3  > gc<88   J  s%&& 7 _,,  3& 7 7  U0C0QUU030C0PUUU CC _0C0Q___ C MMQ&&bbC ,<, !  en , , E 5> ) ) J *4 BB %    r!rScpddlm}tj|d|}|r|dSdS)Nr)TORCH_DYNAMO_RESUME_IN_PREFIXz _(\w+)_at_\d+)resume_executionrarematchgroup)rSrards rremove_resume_prefixrfsL?????? H5FFF M ME {{1~~ 4r!ct}|D]U}|j t|j}|r |r|dj|kr ||d<||_@||V|S)a1 When we graph break, we create a resume function and make a regular Python call to it, which gets intercepted by Dynamo. This behavior is normally shown in the traceback, which can be confusing to a user. So we can filter out resume frames for better traceback clarity. Example: File "..." line 3, in f File "..." line 5, in torch_dynamo_resume_in_f_at_80 File "..." line 10, in torch_dynamo_resume_in_f_at_120 becomes File "..." line 10, in f Nr?)r rNrfrSr^)rY new_stackrTrSs rcollapse_resume_framesris(I$$ > ! #EJ//  $ $)B-"4"<"<!IbMEJJ   U # # # # r!r@cd|jd|jd|jd}|dz }|tz }t ||}|4|dz }|dt |z }|dz }|dz }|S) NWON'T CONVERT r line rz.========== TorchDynamo Stack Trace ========== zY ========== The above exception occurred while processing the following code ========== rz ==========)co_name co_filenameco_firstlinenor rErr )r<rr@rTrsrvs rrrs XWW(8WW@SWWW CCC:<<CU++J    rww{:../// t  x Jr!c rt|dddt|dddt|dddS) Nrmz z (rnrlror))rp)rs rformat_frame_inforrsX 4K 0 0 6 6 D- 5 5 6 6.22 6 6 6r!Optional[types.CodeType]reasonc>|t|}d|d|Sd|S)Nz6torch.compile intentionally decided to skip the frame z! and fall back to eager. Reason: zVtorch.compile intentionally decided to skip the frame and fall back to eager. Reason: rr)rrt frame_infos rformat_skip_frame_messagerxsO &t,,  Z         r! frame_summaryc0t|}d|d|S)NzxSkipping frame because there is a graph break in a for/while loop torch.compile intentionally decided to skip the frame zd and fall back to eager. Reason: Skipping frame because there is a graph break in a for/while loop. rv)rryrws rformat_loop_skip_frame_messager{(s5"4((J AK     r!c tjrt||||Sd|jd|jd|jdt S)Nrkrrlz due to: )rrcrrmrnror )r<rr@rTs rformat_error_msgr}2sk ~K'T?EJJJ 8DL 8 84+; 8 8  8 8)3 8 88r!)rrrr)rrrr) rrrrr%rr&rrr )rr rr')rrmrrrrrrrrrrrr ) rrrrrrrrrr)rr)rrrr@)rrrrrrrrrr rr.rr )rF)r<rmrsrr=r.rr')rrmrKrrrL)rr r))r<rmrTrUrr)rYr rr )rSrrr@)NN) r<rmrrr@r@rTrUrr)rrrr)rrsrtrrr)rrryrrr) __future__rrloggingrcrXtypingenumrr functoolsrpathlibrrCrr r r r r rr torch._guardsrytorch._utils_internalrrrutilsrtypesr output_graphrsymbolic_convertrrr getLoggerr2rrgetArtifactLoggerr RuntimeErrorr$r8r=r?rFrHrJrLrNrPrRrVr\rjrurrrrrrrrrrrrrrrmrrrrrrrrrrrrr StopIteration LookupError IndexError GeneratorExitrGAttributeErrorNotImplementedError TypeErrorrrrr _subclasses fake_tensorDataDependentOutputExceptionDynamicOutputShapeExceptionUnsupportedOperatorExceptionUnsupportedFakeTensorException$UnsupportedMutationAliasingException!exceptions_allowed_to_be_fallbackrrr'r-objectr1rr6rJrPrSrErRrfrirrrrxr{r}r:r!rrs """"""2   JJJJJJJJJJJJ999999999999111111'LLL''''''000000;;;;;;&&&&&&g!!>33HnMMNNNNN<NNN      3        !5        *                /                _        $        ,        )         (    '''''+'''" E E E E E, E E E*****&***8     ;        )                [        [        {   / 88888D888 0     !5        [   GGGGG*GGG      2        #7        Y        i        '   UUUUU,UUU      1        +        ,        *        -        .        ,        "3        )   ,$"( *&4   ,,,,!%'+ (....< !> != !> !@ !F %!####L, 120 688!!!!!!L,-,-,-,-,-^    )))) 8<CCCCCB     J&*'+ 6     &*'+ 8 8 8 8 8 8 8r!