[iM#jUdZddlZddlmZddlmZmZddlmZddl Z ddl m Z edZ edZ iZee jjefed <ehd Zd ed ed ee e fdeee e fgee e fffdZ d9ddddde jde jde jdee jdededede jfdZedde d9ddddde jde jde jdee jdededede jfdZdeededefdZd e jd!edede jfd"Zd#e jd$e jd%ed&edeed'ede jfd(Zd%ed&eddfd)Z d#e jd$e jd%ed&edeede jf d*Z! d:dddddd+dd,d#e jd$e jd-e jd.ee jd/ee jd0ee jd1ed2ed3ed'edeed4ed5eede"e je je je jffd6Z#ed7de# d:dddddd+dd,d#e jd$e jd-e jd.ee jd/ee jd0ee jd1ed2ed3ed'edeed4ed5eede"e je je je jffd8Z$dS);zImplementations of ONNX operators as native Torch ops. NOTE: Fake implementations: Refer to https://docs.pytorch.org/docs/stable/library.html#torch.library.register_fake for more details on how to create fake kernels. N)Callable)OptionalTypeVar) ParamSpec)_dtype_mappings_P_RONNX_ATEN_DECOMP_TABLE> op_type opset_version fake_implreturncdtttfdtttfffd }|S)zDDecorator to register an ONNX operator with a custom implementation.funcrcd}tjdd|d|}|tt t tjj|<||S)Nopsetzonnx::.) mutates_args)torchlibrary custom_opr getattropsonnx register_fake)roverloadtorch_oprrrs G/var/www/icac/venv/lib/python3.11/site-packages/torch/onnx/ops/_impl.py decoratorz_onnx_op..decorator's*=**=** )W ) )x ) )+    wwuy~w'G'GRRS y))))rrr )rrrr$s``` r#_onnx_opr&"sU R( Xb"f-=         r%F) interleaved num_headsrotary_embedding_dimx cos_cache sin_cache position_idsr'r(r)c*|S)zFFake implementation for RotaryEmbedding-23 for torch.compile purposes.)clone)r*r+r,r-r'r(r)s r#_rotary_embedding_23_fake_implr05s 7799r%RotaryEmbeddingch|jt}ddtjdkfdtjjdkfdtjjdkfdtjdkodkfd nGtjd kod kfd |d krtj|d }nJ|d krDtj|dkfdd}||z} || g} tj|| }tjt|jd kd|jd } |dkr| }|ddddddd|f} |dddddd|df} |dzntjjdkojdkfdtjjdkojdkfdtjjdkfdtjjdkfdtjdtjd|r+| dddddddddf} | dddddddddf}ntj| dd\} }| z|zz }| z|zz}|r]tj|d}tj|d}tj ||fd}tj|| j} ntj ||fd} tj | | fd}|d krtj|Stj|d S)z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rNcdjS)Nz6position_ids must be 2D when provided. Received shape shape)r-sr#z%rotary_embedding_23..Zsa\M_aar%c*ddjdS)Nz6position_ids first dim (batch) must match x.shape[0] ( ). Received rr7) batch_sizer-sr#r9z%rotary_embedding_23..^s"|Z||eqewxyez||r%r c*ddjdS)Nz;position_ids second dim (sequence) must match x.shape[-2] (r;r r7)r-sequence_lengthsr#r9z%rotary_embedding_23..bs>GRaGGo{pBCDpEGGr%c(djdjS)NzWcos_cache/sin_cache must be 2D when position_ids is provided. Received cos_cache shape , sin_cache shape r7r+r,sr#r9z%rotary_embedding_23..f-](1]]KT?]]r%c(djdjS)Nz[cos_cache/sin_cache must be 3D when position_ids is not provided. Received cos_cache shape r@r7rAsr#r9z%rotary_embedding_23..lrBr%)rr5r rCcdS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape r) input_shapesr#r9z%rotary_embedding_23..ysobmoor%cdS)Nzx should be a 4D tensor by nowrrr%r#r9z%rotary_embedding_23..s,Lr%c&djdddS)Nzcos has shape  but expected (batch=, seq=, ...)r7)r<cosr>sr#r9z%rotary_embedding_23..&jjjjjSbjjjr%c&djdddS)Nzsin has shape rJrKrLr7)r<r>sinsr#r9z%rotary_embedding_23..rNr%c,djdddS)NzLast dimension of cos cache (rQ') should match rotary_embedding_dim/2 ().r7)rMrotary_embedding_dim_halfsr#r9z%rotary_embedding_23..4D " DDfDDDr%c,djdddS)NzLast dimension of sin cache (rQrSrTr7)rUrPsr#r9z%rotary_embedding_23..rVr%dim) r8lenr_checkrYpermutereshape unsqueezechunkcat)r*r+r,r-r'r(r) input_rank hidden_size head_size new_shapex_rotate x_not_rotatex1x2realimagx_rotate_concatoutputr<rMrGrUr>rPs ``` @@@@@@r#rotary_embedding_23rmCsv'K[!!JQJ!"oO     ! # a a a a      q !Z / | | | | |      q !_ 4 G G G G G     MMOOq 9Y]]__%9 ] ] ] ] ]     MMOOq 9Y]]__%9 ] ] ] ] ]   Q M!\ * * q N o o o o   "!n 9, )YG M!Y ' ' LQW"$L$LMMM Iq  (AAAqqq////0HQQQ11123334L 4 9     L ! "Fsy|'Fjjjjjj L ! "Fsy|'Fjjjjjj L " 22 D D D D D L " 22 D D D D D / Q  C / Q  C 2 aaaAAAqt!tm $ aaaAAAqt!tm $Xqb111B 8cBh D 8cBh D3tR((tR(()T4Lb999=(.AA9dD\r222 Y,/R 8 8 8FQ}V[111 = . ..r%scalercc8||ndtj|z S)z/Get the scale factor for attention computation.Ng?)mathsqrt)rnrcs r#_get_scale_factorrrs %55C$)I2F2F,FGr%tensorr<c|jd|jd}}||z}|||||ddS)z1Reshape 3D tensor to 4D for multi-head attention.r r5)r8view transpose contiguous)rsr<r(r>rbrcs r#_reshape_3d_to_4drxsS$*<?FLO[Oy(I JIFF 1a r%QKcurrent_q_num_headscurrent_kv_num_headsqk_matmul_output_modec |dkrt|||||Stjtj||ddS)z1Get QK output tensor based on the specified mode.rr4rQ)_compute_qk_output_for_mode_0r zeros_likematmulrv)ryrzr{r|rnr}s r#_get_qk_output_for_aten_spdars[!!, q%';U    Q B0C0C D DEEEr%cJtjzdkfddS)z-Validate Group Query Attention configuration.rcdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQAr)r|r{sr#r9z-_validate_gqa_configuration..sy 3yyZnyyyr%N)rr[)r{r|s``r#_validate_gqa_configurationrsA L22a7yyyyyr%c|}||kr||z}||d}t||jd}tj|}||z} ||z} t j| | ddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.r rXrCr4rQ)repeat_interleaverrr8rprqrrrv) ryrzr{r|rnK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaleds r#rrsH222+/CC &&}!&<<$UAGAJ77L<((J:~H*$H <("4"4R"<"< = ==r%) is_causal kv_num_heads q_num_headsr}rnsoftcapsoftmax_precisionV attn_maskpast_key past_valuerrrrrc^|jd} t|jdkrr|jd}|j}|.| ||jd|jdz|jd|zf}n| ||jd|jd|zf}|}| |||df}n|jd}|j}|A|jd|jd|jd|jdz|jdf}n|j}|}|jd|jd|jd|df}tj||j|j}tj||j|j}tj||j|j}tj||j|j}||||fS)z@Fake implementation for Attention-23 for torch.compile purposes.rrCr Nr5dtypedevice)r8rZremptyrr)ryrzrrrrrrrr}rnrrr<q_sequence_length output_shapepresent_key_shapepresent_value_shapeqk_output_shaperl present_key present_value qk_outputs r#_attention_23_fake_implrs"J 17||qGAJw   q!AGAJ. l* !    l* !  0    a  GAJw     q!AGAJ. !  !" / GAJ GAJ GAJ a  [QWQX F F FF+/qwqxPPPKK 31718TTTM O1718LLLI ; y 88r% Attentionc  d\} }}t|j}|jd}t|jdkr`tj|dko|dkd|jd}t |||}t |||}t |||}tjt|jdko/t|jdkot|jdkd|j|}t | |} |tj||g| n|}|tj||g| n|}||}}|j| }|j| }|j|}|j|}| d ko"| dko| duo|dup|jtj k}t|||rStj j ||||d || t||k }t||||| | }nY||kr3||z}||| }||| }tj|||j|j }|rutj|dud tjtj||tj |j }||t+d}|?|jtj kr%||t+d}n||z}t | |jd}t-j|} || z}!|| z}"tj|!|"dd}#|#}|#|z}$| dkr|$}| d kr| tj|$| z z}$| dkr|$}| x| t6vrX|$j}%|$t:j| }$tj|$d }&|&|%}&n-tj|$d }&ntj|$d }&| dkr|&}tj|&|}|dkr=|dd !||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)r r5rCrrCcdS)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrr%r#r9zattention_23..sQr%r rEcdS)Nz'Q, K, and V should be 4D tensors by nowrrr%r#r9zattention_23..s9r%NrXr)r dropout_prrn enable_gqarcdS)Nz'Cannot use both is_causal and attn_maskrrr%r#r9zattention_23..s+Tr%z-infr4rQr5)"rZr8rr[rxrrr`r/rboolrnn functionalscaled_dot_product_attentionrrzerosrtrilones masked_fillfloatrprqrrvtanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorONNX_DTYPE_TO_TORCH_DTYPEsoftmaxrwru)'ryrzrrrrrrrr}rnrr num_head_dim sequence_dimhead_diminput_shape_lenr<r q_head_sizerrr{r|kv_sequence_length can_use_sdparlrr attn_bias causal_maskrrrrqk_matmul_output qk_with_biasoriginal_dtype qk_softmaxs' r# attention_23rcs&,3(L,!'llOJ 17||q 1  2!2 Q Q   GAJ a[ 9 9 a\ : : a\ : : L AG Ec!'lla/ECLLA4E99 '(#K e[ 1 1E    8Q-\2222 WWYY  !  :q/|4444 WWYY  qA',/7<0 -. 3 A !Q & A  % A$  ?)/UZ"?  35IJJJk-$AA #';;B   1   !    "6 6 6/3GGM##M|#DDA##M|#DDAK 1    K LT!#T#T    * %&*8 K"--{lE&MMJJI  %*,,%119*eFmmLL & 1 ) ;; Y|,, z>z>!<(2D2DR2L2LMM% ()3 A % %$I S=="UZ w0F%G%GGL A % %$I  ( $QQQ!-!3+#=>OP   #]:: "]rs5 $$$$$$$$$$$$$$'''''' ******Yt__ WT]]ACUZ2H<=BBB09 11- !$19"b&1A xB (2r6"223.,0   !    | | | 5<(      \      R!?@@ ,0 D/  !D/D/D/ |D/|D/|D/5<( D/  D/D/D/ \D/D/D/A@D/NHXe_HHHHHH  L &) 69  \    F |F |FF F E? F  F \FFFF$47 > |> |>> > E? >  \ >>>>4)-'+)- Q9!"!'+Q9Q9Q9 |Q9 |Q9 |Q9 % Q9 u|$ Q9 & Q9Q9Q9Q9Q9 E?Q9Q9 }Q9 5<u|U\ ABQ9Q9Q9Q9h +r233 )-'+)- 9!"!'+999 |9 |9 |9 % 9 u|$ 9 & 99999 E?99 }9 5<u|U\ AB99943999r%