[i;ddlZddlmZddlZddlmZmZddlmZm Z ddl m Z ddl m ZddlmZgd ZGd d eZGd d eZ eeeeefZGddeZGddeZGddeZdS)N)Union)SizeTensor) functionalinit) Parameter) CrossMapLRN2d)Module)LocalResponseNormr LayerNorm GroupNormRMSNormc eZdZUdZgdZeed<eed<eed<eed< ddedededed d f fd Zd e d e fdZ dZ xZ S)r aApplies local response normalization over an input signal. The input signal is composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. .. math:: b_{c} = a_{c}\left(k + \frac{\alpha}{n} \sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta} Args: size: amount of neighbouring channels used for normalization alpha: multiplicative factor. Default: 0.0001 beta: exponent. Default: 0.75 k: additive factor. Default: 1 Shape: - Input: :math:`(N, C, *)` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> lrn = nn.LocalResponseNorm(2) >>> signal_2d = torch.randn(32, 5, 24, 24) >>> signal_4d = torch.randn(16, 5, 7, 7, 7, 7) >>> output_2d = lrn(signal_2d) >>> output_4d = lrn(signal_4d) )sizealphabetakrrrr-C6???returnNct||_||_||_||_dSNsuper__init__rrrrselfrrrr __class__s Q/var/www/icac/venv/lib/python3.11/site-packages/torch/nn/modules/normalization.pyrzLocalResponseNorm.__init__5;    inputcZtj||j|j|j|jSz( Runs the forward pass. )Flocal_response_normrrrrrr$s r!forwardzLocalResponseNorm.forward>s%$UDItz49dfUUUr#c&djdi|jS@ Return the extra representation of the module. z){size}, alpha={alpha}, beta={beta}, k={k}format__dict__rs r! extra_reprzLocalResponseNorm.extra_reprD!B:ARRDMRRRr#)rrr) __name__ __module__ __qualname____doc__ __constants__int__annotations__floatrrr*r3 __classcell__r s@r!r r s:322M III LLL KKK HHHNQ %49EJ VVVVVVV SSSSSSSr#r c ~eZdZUeed<eed<eed<eed< ddededededd f fd Zd edefd Zde fd Z xZ S)r rrrrrrr rNct||_||_||_||_dSrrrs r!rzCrossMapLRN2d.__init__Qr"r#r$cZtj||j|j|j|jSr&)_cross_map_lrn2dapplyrrrrr)s r!r*zCrossMapLRN2d.forwardZs% %eTY DItvVVVr#c&djdi|jSr,r/r2s r!r3zCrossMapLRN2d.extra_repr`r4r#)rrr ) r5r6r7r:r;r<rrr*strr3r=r>s@r!r r Ks III LLL KKK HHHNO %49EJ WVWWWWW SCSSSSSSSSr#r c eZdZUdZgdZeedfed<eed<e ed< dde dede d e d d f fd Z dd Z de d e fdZd efdZxZS)r a Applies Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Layer Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated over the last `D` dimensions, where `D` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. ``input.mean((-2, -1))``). :math:`\gamma` and :math:`\beta` are learnable affine transform parameters of :attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``. The variance is calculated via the biased estimator, equivalent to `torch.var(input, correction=0)`. .. note:: Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the :attr:`affine` option, Layer Normalization applies per-element scale and bias with :attr:`elementwise_affine`. This layer uses statistics computed from input data in both training and evaluation modes. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps: a value added to the denominator for numerical stability. Default: 1e-5 elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True``. bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`elementwise_affine` is ``True``). Default: ``True``. Attributes: weight: the learnable weights of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 1. bias: the learnable bias of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 0. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> # NLP Example >>> batch, sentence_length, embedding_dim = 20, 5, 10 >>> embedding = torch.randn(batch, sentence_length, embedding_dim) >>> layer_norm = nn.LayerNorm(embedding_dim) >>> # Activate module >>> layer_norm(embedding) >>> >>> # Image Example >>> N, C, H, W = 20, 5, 10, 10 >>> input = torch.randn(N, C, H, W) >>> # Normalize over the last three dimensions (i.e. the channel and spatial dimensions) >>> # as shown in the image below >>> layer_norm = nn.LayerNorm([C, H, W]) >>> output = layer_norm(input) .. image:: ../_static/img/nn/layer_norm.jpg :scale: 50 % normalized_shapeepselementwise_affine.rHrIrJh㈵>TNbiasrc6||d}tt|tjr|f}t ||_||_||_|jrlttj |jfi||_ |r*ttj |jfi||_ nC|ddn,|dd|dd|dS)NdevicedtyperLweight)rr isinstancenumbersIntegraltuplerHrIrJrtorchemptyrQrLregister_parameterreset_parameters) rrHrIrJrLrOrPfactory_kwargsr s r!rzLayerNorm.__init__s/%+U;;  &(8 9 9 3 02  %&6 7 7"4  " 2# D1DD^DDDK 6%K 5HHHH ''5555  # #Hd 3 3 3  # #FD 1 1 1 r#c|jr;tj|j|jtj|jdSdSdSr)rJrones_rQrLzeros_r2s r!rYzLayerNorm.reset_parameterssO  " ' Jt{ # # #y$ DI&&&&& ' '$$r#r$cZtj||j|j|j|jSr)r' layer_normrHrQrLrIr)s r!r*zLayerNorm.forwards*| 4($+ty$(   r#c&djdi|jS)NF{normalized_shape}, eps={eps}, elementwise_affine={elementwise_affine}r.r/r2s r!r3zLayerNorm.extra_reprs1 = 66< N N?C} N N r#)rKTTNNrN)r5r6r7r8r9rUr:r;r<bool_shape_trrYrr*rEr3r=r>s@r!r r jsKKZFEEMCHo%%% JJJ #'  "  !           B''''  V      C        r#r c eZdZUdZgdZeed<eed<eed<eed< ddedededed d f fd Z dd Z d e d e fdZ d e fdZxZS)raApplies Group Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Group Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The input channels are separated into :attr:`num_groups` groups, each containing ``num_channels / num_groups`` channels. :attr:`num_channels` must be divisible by :attr:`num_groups`. The mean and standard-deviation are calculated separately over each group. :math:`\gamma` and :math:`\beta` are learnable per-channel affine transform parameter vectors of size :attr:`num_channels` if :attr:`affine` is ``True``. The variance is calculated via the biased estimator, equivalent to `torch.var(input, correction=0)`. This layer uses statistics computed from input data in both training and evaluation modes. Args: num_groups (int): number of groups to separate the channels into num_channels (int): number of channels expected in input eps: a value added to the denominator for numerical stability. Default: 1e-5 affine: a boolean value that when set to ``True``, this module has learnable per-channel affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True``. Shape: - Input: :math:`(N, C, *)` where :math:`C=\text{num\_channels}` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> input = torch.randn(20, 6, 10, 10) >>> # Separate 6 channels into 3 groups >>> m = nn.GroupNorm(3, 6) >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm) >>> m = nn.GroupNorm(6, 6) >>> # Put all 6 channels into a single group (equivalent with LayerNorm) >>> m = nn.GroupNorm(1, 6) >>> # Activating the module >>> output = m(input) ) num_groups num_channelsrIaffinerfrgrIrhrKTNrc||d}t||zdkrtd|d|d||_||_||_||_|jrIttj |fi||_ ttj |fi||_ n,| dd| dd| dS)NrNrznum_channels (z#) must be divisible by num_groups ()rQrL)rr ValueErrorrfrgrIrhrrVrWrQrLrXrY) rrfrgrIrhrOrPrZr s r!rzGroupNorm.__init__$s%+U;;  * $ ) )___R\___ %( ; 2#EK $O$O$O$OPPDK!%+l"M"Mn"M"MNNDII  # #Hd 3 3 3  # #FD 1 1 1 r#c||jr4tj|jtj|jdSdSr)rhrr\rQr]rLr2s r!rYzGroupNorm.reset_parametersAs@ ; # Jt{ # # # K " " " " " # #r#r$cZtj||j|j|j|jSr)r' group_normrfrQrLrIr)s r!r*zGroupNorm.forwardFs"|E4?DKDHUUUr#c&djdi|jS)Nz8{num_groups}, {num_channels}, eps={eps}, affine={affine}r.r/r2s r!r3zGroupNorm.extra_reprIs)PIP  m   r#)rKTNNrb)r5r6r7r8r9r:r;r<rcrrYrr*rEr3r=r>s@r!rrs++ZDCCMOOO JJJ LLL                :#### VVVVVVV C        r#rc eZdZUdZgdZeedfed<edzed<e ed< dde dedzde d dffd Z dd Z d e jd e jfd Zd efdZxZS)raApplies Root Mean Square Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Root Mean Square Layer Normalization `__ .. math:: y_i = \frac{x_i}{\mathrm{RMS}(x)} * \gamma_i, \quad \text{where} \quad \text{RMS}(x) = \sqrt{\epsilon + \frac{1}{n} \sum_{i=1}^{n} x_i^2} The RMS is taken over the last ``D`` dimensions, where ``D`` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the RMS is computed over the last 2 dimensions of the input. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps: a value added to the denominator for numerical stability. Default: ``torch.finfo(x.dtype).eps`` elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights). Default: ``True``. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> rms_norm = nn.RMSNorm([2, 3]) >>> input = torch.randn(2, 2, 3) >>> rms_norm(input) rG.rHNrIrJTrc||d}tt|tjr|f}t ||_||_||_|jr*ttj |jfi||_ n| dd|dS)NrNrQ)rrrRrSrTrUrHrIrJrrVrWrQrXrY)rrHrIrJrOrPrZr s r!rzRMSNorm.__init__}s%+U;;  &(8 9 9 3 02  %&6 7 7"4  " 4# D1DD^DDDKK  # #Hd 3 3 3 r#cJ|jrtj|jdSdS)zS Resets parameters based on their initialization used in __init__. N)rJrr\rQr2s r!rYzRMSNorm.reset_parameterss1  " $ Jt{ # # # # # $ $r#xcNtj||j|j|jSr&)r'rms_normrHrQrI)rrss r!r*zRMSNorm.forwards!z!T2DKJJJr#c&djdi|jS)r-rar.r/r2s r!r3zRMSNorm.extra_reprs1  = 66< N N?C} N N r#)NTNNrb)r5r6r7r8r9rUr:r;r<rcrdrrYrVrr*rEr3r=r>s@r!rrOs &&PFEEMCHo%%%  !#'   " T\ !         0$$$$KK%,KKKK  C        r#r)rStypingrrVrrtorch.nnrr'rtorch.nn.parameterr _functionsr rBmoduler __all__r r:listrdr rrr.r#r!r~s ********((((((999999 V U U7S7S7S7S7S7S7S7StSSSSSFSSS8 d3i% &C C C C C C C C L\ \ \ \ \ \ \ \ ~Z Z Z Z Z fZ Z Z Z Z r#