[i.UdZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z m Z dd lmZe rdd lmZddlZdd lmZd gZdad ed<eGddZedWdZ dXdYdZedZdZd[dZ d\d]d'Zd^d0Zd_d7Ze d8Z d`d;Z! dadbdBZ"dcdDZ#dddddddEdddIZ$ddddJdedMZ% dfddPdgdQZ&ddPdhdTZ'ej(dUeVdS)izUBER PROTOTYPE!!!) annotationsN) dataclass)cache)Any TYPE_CHECKING) TypeVarTupleUnpack) _registry) ModuleType)Libraryregister_flash_attention_fa4 str | None_FA4_MODULE_PATHc"eZdZUded<ddZdS) _FA4HandlezLibrary | NonelibraryreturnNonecd|_dSN)r)selfs J/var/www/icac/venv/lib/python3.11/site-packages/torch/nn/attention/_fa4.pyremovez_FA4Handle.remove"s  N)rr)__name__ __module__ __qualname____annotations__rrrrrs6rrdevice torch.devicerintcJtj|\}}|Sr)torchcudaget_device_capability)r!major_s r_get_device_majorr*&s z//77HE1 Lrflash_attn.cute.interface module_pathstrcZt|}|attS)z Register FA4 flash attention kernels with the PyTorch dispatcher. Args: module_path: Python module path to the FA4 implementation. )_fa4_import_modulerr_fa4_register_kernels)r,r)s rrr,s, ;''A" +-- . ..rr ctj|}t|drt|dstd|d|S)N_flash_attn_fwd_flash_attn_bwdzModule 'z' does not expose FA4 kernels) importlib import_modulehasattr RuntimeError)r,modules rr/r/;sX  $[ 1 1F 6, - -RWVEV5W5WRPkPPPQQQ Mrr ctddd}|dtd|dtd|dtd|dt d|S)NatenIMPLCUDA_flash_attention_forward_flash_attention_backward#_scaled_dot_product_flash_attention,_scaled_dot_product_flash_attention_backward)r impl!_fa4_flash_attention_forward_impl"_fa4_flash_attention_backward_impl4_fa4_scaled_dot_product_flash_attention_forward_impl5_fa4_scaled_dot_product_flash_attention_backward_impl)libs rr0r0Cs &&& ) )CHH ')JFSSSHH (*LfUUUHH-< HH6= Jrr query torch.Tensortensorstuple[torch.Tensor, ...] cum_seq_qtorch.Tensor | None require_fp32$tuple[tuple[str, torch.Tensor], ...]ctd|DsdStd|DdkrdS|jtjtjfvrdS|D]!\}}|jtjkr|dcS"||dkrd S||d krd Stj sd St|j d vrdSdS)Nc3$K|] }|jV dSr)is_cuda.0ts r z,_fa4_common_support_error..Zs$**Qqy******rzinputs must be CUDA tensorsch|] }|j Sr )r!rRs r z,_fa4_common_support_error..\s & & &AH & & &rr inputs must share devicez'query dtype must be float16 or bfloat16z dtype must be float32zdense query must be 4Dzragged query must be 3DzCUDA not available) z+FA4 requires compute capability 9.0 or 10.0) alllendtyper%float16bfloat16float32dimr& is_availabler*r!)rGrIrKrMnametensors r_fa4_common_support_errorrgTs! **'*** * *-,, & &g & & &''1,,)) {5=%.99988$33 f <5= ( (222 2 2 2 )UYY[[A--''!1!1(( : " " $ $$##&&g55<< 4rkeyvalue dropout_pfloatreturn_debug_maskbool alibi_slopes seqused_kc|dkrdS|rdS|dS| |jtjkrdS|jsdSt ||||f|}| |dkrdS|SdS) Ndropout_p must be 0zreturn_debug_mask must be Falsezalibi_slopes not supportedzseqused_k must be int32zseqused_k must be CUDArXz(query, key, value must be on same device)r_r%int32rQrg) rGrhrirjrlrnrorKerrors r_fa4_forward_support_errorrunsC$$100++ ?ek ) ),,  ,++ % U  E   . . .== 4rgrad_outout logsumexpwindow_size_left int | Nonewindow_size_rightc b|dkrdS|| dSt|||||||f|d|ff} | | SdS)Nrqrrz windowed attention not supportedrx)rM)rg) rvrGrhrirwrxrjrKryr{rts r_fa4_backward_support_errorr}snC$$#'8'D11 % 5#uc95"I.0    E   4rTs Unpack[Ts]tuple[Unpack[Ts]]c4td|DS)Nc3BK|]}|ddVdS)r N) transposerRs rrUz#_transpose_dense..s044qQ""444444r)tuple)rIs r_transpose_densers 44G444 4 44rcu_seq_qcu_seq_kscale float | None is_causal!tuple[torch.Tensor, torch.Tensor]c ttdtt} ||||d||| | ndd} | | | d<| j|||fi| \} } | | fS)NFA4 not registeredT) softmax_scalecausalryr{ return_lse cu_seqlens_q cu_seqlens_krorw)rr7r/ contiguousr2)rGrhrirrrrryr{rorwr8kwargslses r_fa4_run_forwardrs/000  0 1 1F,.  /8/DY))+++$  F u %v%eS%BB6BBHC    r/tuple[torch.Tensor, torch.Tensor, torch.Tensor]c ttdtt} | |||||||| || \} } } | | | fS)Nr)rrrr)rr7r/r3r)rvrGrhrirwrxrrrrr8dqdkdvs r_fa4_run_backwardrs/000  0 1 1F''    (  JBB r2:r)rryr{rornrw cum_seq_kmax_qmax_kc  vt||||| || |}|td|t|||||| || | | | \}}tjdtj|j}tjdtj|j}tjd|j|j}|||||fS)Nz)FA4 flash_attention forward unsupported: )r)r_r!r r) rur7rr%zerosuint64r!emptyr_)rGrhrirKrrrrjrrlrryr{rornrwrtr rng_state philox_offset debug_masks rrBrBs& '     E NuNNOOO       HC D U\JJJIK%,u|LLLMQek%,GGGJ Y z 99r)rryr{runusedc t||||||| ||| }|td|t|||||||||| \}}}|||fS)Nz*FA4 flash_attention backward unsupported: )r}r7r)rvrGrhrirwrxrKrrrrjrrrrryr{rtrrrs rrCrC"s( (      E OOOPPP"       JBB r2:rrqFrct|||||ddd}|td|t|||\}} } tj|} | dd} |d} | d}t|| | dd| ||||||  \}}}}}|d}|d}| |dd|||||f S)NzFA4 SDPA forward unsupported: r r)rrw)rur7rr% empty_likersizerB)rGrhrirjrrlrrtqkvout_bhsdout_bshd max_q_flash max_k_flashr)rrrrrrs rrDrDSs. '        E CECCDDDuc511GAq! &&H!!!Q''H&&))K&&))K3T       4 4 40AsI}j JJqMME HHQKKE       r philox_seedrcXt||||||| ddd }|td|t|||||\}}}}}|d}|d} t ||||||dd|| | | | | |\}}}t|||\}}}|||fS)NzFA4 SDPA backward unsupported: rr)r}r7rrrC)rvrGrhrirwrxrKrrrrjrrrrrtrrrogorrrs rrErEs$ (         E DUDDEEE%eS%hGGNAq!Q JJqMME HHQKKE3         JBB"""b"--JBB r2:rFA4) register_fn)r!r"rr#)r+)r,r-rr)r,r-rr )rr )r ) rGrHrIrJrKrLrMrNrr)rGrHrhrHrirHrjrkrlrmrnrLrorLrKrLrr)rvrHrGrHrhrHrirHrwrHrxrHrjrkrKrLryrzr{rzrr)rIrrrr)rGrHrhrHrirHrrLrrLrrrrmryrzr{rzrorLrwrLrr)rvrHrGrHrhrHrirHrwrHrxrHrrLrrLrrrrmrr) rGrHrhrHrirHrKrLrrLrr#rr#rjrkrrmrlrmrrryrzr{rzrorLrnrLrwrL)"rvrHrGrHrhrHrirHrwrHrxrHrKrLrrLrr#rr#rjrkrrmrrHrrHrrryrzr{rz)rqFF)rGrHrhrHrirHrjrkrrmrlrmrr)rvrHrGrHrhrHrirHrwrHrxrHrKrLrrLrr#rr#rjrkrrmrrHrrHrr))__doc__ __future__rr4 dataclassesr functoolsrtypingrrtyping_extensionsrr r typesr r% torch.libraryr __all__rrrr*rr/r0rgrur}r~rrrrBrCrDrEregister_flash_attention_implr rrrs#"""""!!!!!!%%%%%%%%22222222!      !!!!!!#  $####   3 / / / / /*:< 4B6\$5555 $!!!!!BT#'$(%)(,##/:/:/:/:/:/:D#'$(%......j# :::::::Z!555555p( ';WXXXXXXr