FPhEdZddlZddlZddlmZddlmcmZddlm Z m Z ddl m Z ddl mZmZmZdZGdd ej$ZGd d eZGd d ej$ZGddej$ZGddej$ZGddej$ZGddej$ZGddej$ZGddej$ZGddej$Zy)zTransformer modules.N) constant_xavier_uniform_)Conv) _get_clonesinverse_sigmoid#multi_scale_deformable_attn_pytorch) TransformerEncoderLayerTransformerLayerTransformerBlockMLPBlock LayerNorm2dAIFIDeformableTransformerDecoder!DeformableTransformerDecoderLayer MSDeformAttnMLPcleZdZdZdddej dffd Zd dZd dZd d Z d d Z xZ S) r z2Defines a single layer of the transformer encoder.Fct|ddlm}|s t dt j |||d|_t j|||_ t j|||_ t j||_ t j||_ t j||_t j||_t j||_||_||_y)zAInitialize the TransformerEncoderLayer with specified parameters.) TORCH_1_9z]TransformerEncoderLayer() requires torch>=1.9 to use nn.MultiheadAttention(batch_first=True).T)dropout batch_firstN)super__init__utils.torch_utilsrModuleNotFoundErrornnMultiheadAttentionmaLinearfc1fc2 LayerNormnorm1norm2Dropoutrdropout1dropout2actnormalize_before) selfc1cm num_headsrr-r.r __class__s mC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\ultralytics/nn/modules/transformer.pyrz TransformerEncoderLayer.__init__s 2%oq q''IwTXY99R$99R$\\"% \\"% zz'*  7+  7+  0c||S||zS)z2Add position embeddings to the tensor if provided.)r/tensorposs r4with_pos_embedz&TransformerEncoderLayer.with_pos_embed*sv6&3,6r5c l|j||x}}|j|||||d}||j|z}|j|}|j |j |j |j|}||j|z}|j|S)z.Performs forward pass with post-normalization.value attn_maskkey_padding_maskr) r:r#r+r(r&rr-r%r,r))r/srcsrc_masksrc_key_padding_maskr9qksrc2s r4 forward_postz$TransformerEncoderLayer.forward_post.s##C--Awwq!3(MawbcdeDMM$''jjoxx TXXdhhsm%<=>DMM$''zz#r5c l|j|}|j||x}}|j|||||d}||j|z}|j |}|j |j |j|j|}||j|zS)z-Performs forward pass with pre-normalization.r<r) r(r:r#r+r)r&rr-r%r,)r/r@rArBr9rErCrDs r4 forward_prez#TransformerEncoderLayer.forward_pre8szz###D#..Awwq!48NbwcdefDMM$''zz#xx TXXdhhtn%=>?T]]4(((r5cj|jr|j||||S|j||||S)z8Forward propagates the input through the encoder module.)r.rHrF)r/r@rArBr9s r4forwardzTransformerEncoderLayer.forwardBs;  ##C3GM M  h0DcJJr5NNNN) __name__ __module__ __qualname____doc__r!GELUrr:rFrHrJ __classcell__r3s@r4r r s4<"a'"'')^c1*7)Kr5r cheZdZdZdddej dffd ZfdZed dZ xZ S) rz#Defines the AIFI transformer layer.rrrFc.t|||||||y)z7Initialize the AIFI instance with specified parameters.N)rr)r/r0r1r2rr-r.r3s r4rz AIFI.__init__Ls RGS:JKr5cd|jdd\}}}|j|||}t| |j dj ddd|j |j|j}|j dddjd|||gjS)z,Forward pass for the AIFI transformer layer.rNr)devicedtype)r9) shape"build_2d_sincos_position_embeddingrrJflattenpermutetorXrYview contiguous)r/xchw pos_embedr3s r4rJz AIFI.forwardPs''!"+1a;;Aq!D GOAIIaL00Aq9y||STS[S[cdcjcj|?kO lyyAq!&&Aq!}5@@BBr5ctjt|tj}tjt|tj}tj||d\}}|dzdk(sJd|dz}tj|tj|z }d||zz }|j d|d z}|j d|d z} tj tj|tj|tj| tj| gd d S) z)Builds 2D sine-cosine position embedding.rYij)indexingrzHEmbed dimension must be divisible by 4 for 2D sin-cos position embeddingg?.NNr) torcharangeintfloat32meshgridr]catsincos) rerd embed_dim temperaturegrid_wgrid_hpos_dimomegaout_wout_hs r4r\z'AIFI.build_2d_sincos_position_embeddingXsc!fEMM:c!fEMM:F1}! W V W!q. WEMM:WDkU*+ +eDk9 +eDk9yy%))E*EIIe,NPUPYPYZ_P`acdefjkkr5)g@) rMrNrOrPr!rQrrJ staticmethodr\rRrSs@r4rrIs<-"a \aLCllr5rc(eZdZdZfdZdZxZS)r zeTransformer layer https://arxiv.org/abs/2010.11929 (LayerNorm layers removed for better performance).c|t|tj||d|_tj||d|_tj||d|_tj|||_tj||d|_ tj||d|_ y)z]Initializes a self-attention mechanism using linear transformations and multi-head attention.F)bias)rur2N) rrr!r$rCrDvr"r#r%r&)r/rcr2r3s r4rzTransformerLayer.__init__ms 1ae,1ae,1ae,''!yI99Q.99Q.r5c|j|j||j||j|d|z}|j |j ||zS)z?Apply a transformer block to the input x and return the output.r)r#rCrDrr&r%r/rbs r4rJzTransformerLayer.forwardwsR GGDFF1Itvvay$&&) 4Q 7! ;xx $q((r5rMrNrOrPrrJrRrSs@r4r r jso/)r5r c(eZdZdZfdZdZxZS)r z4Vision Transformer https://arxiv.org/abs/2010.11929.ct|d|_|k7rt||_t j |_t jfdt|D|_ |_ y)zaInitialize a Transformer module with position embedding and specified number of heads and layers.Nc36K|]}tywrK)r ).0_c2r2s r4 z,TransformerBlock.__init__..s!]K\a"22y"AK\s) rrconvrr!r$linear Sequentialrangetrr)r/r0rr2 num_layersr3s `` r4rzTransformerBlock.__init__s^  8R DIiiB' --!]5Q[K\!]^r5cB|j|j|}|j\}}}}|jdjddd}|j ||j |zjdddj ||j||S)z;Forward propagates the input through the bottleneck module.rWrr)rr[r]r^rrreshaper)r/rbbrrerdps r4rJzTransformerBlock.forwards 99  ! AWW 1a IIaL Aq )wwq4;;q>)*221a;CCAtwwPQSTUUr5rrSs@r4r r }s>Vr5r creZdZdZej ffd ZdejdejfdZ xZ S)r z6Implements a single block of a multi-layer perceptron.ct|tj|||_tj|||_||_y)zcInitialize the MLPBlock with specified embedding dimension, MLP dimension, and activation function.N)rrr!r$lin1lin2r-)r/ embedding_dimmlp_dimr-r3s r4rzMLPBlock.__init__s; IImW5 IIg}5 5r5rbreturnc`|j|j|j|S)zForward pass for the MLPBlock.)rr-rrs r4rJzMLPBlock.forwards"yy$))A,/00r5) rMrNrOrPr!rQrrmTensorrJrRrSs@r4r r s-@357711%,,1r5r c(eZdZdZfdZdZxZS)rz=Implements a simple multi-layer perceptron (also called FFN).ct|||_|g|dz z}tjdt |g|z||gzD|_y)zXInitialize the MLP with specified input, hidden, output dimensions and number of layers.rc3NK|]\}}tj||ywrK)r!r$)rnrDs r4rzMLP.__init__..s!#g@f1BIIaO@fs#%N)rrrr! ModuleListziplayers)r/ input_dim hidden_dim output_dimrrdr3s r4rz MLP.__init__sS $ LJN +mm#gYKRSOUVZdYeUe@f#gg r5ct|jD]:\}}||jdz krtj||n||}<|S)z Forward pass for the entire MLP.r) enumeraterrFrelu)r/rbilayers r4rJz MLP.forwardsF!$++.HAu$%!(;$;uQx qA/r5rrSs@r4rrsGhr5rc*eZdZdZdfd ZdZxZS)raP 2D Layer Normalization module inspired by Detectron2 and ConvNeXt implementations. Original implementation at https://github.com/facebookresearch/detectron2/blob/main/detectron2/layers/batch_norm.py https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ct|tjt j ||_tjt j||_||_ y)z1Initialize LayerNorm2d with the given parameters.N) rrr! Parameterrmonesweightzerosreps)r/ num_channelsrr3s r4rzLayerNorm2d.__init__sG ll5::l#;< LL\!:; r5c|jdd}||z jdjdd}||z tj||jzz }|j ddddf|z|j ddddfzS)z0Perform forward pass for 2D layer normalization.rTkeepdimrWN)meanpowrmsqrtrrr)r/rbuss r4rJzLayerNorm2d.forwards FF1dF # UKKN  4  0 UejjTXX. .{{1dD=)A- !T4-0HHHr5)gư>rrSs@r4rrsIr5rc2eZdZdZdfd ZdZddZxZS)rz Multi-Scale Deformable Attention Module based on Deformable-DETR and PaddleDetection implementations. https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/ops/modules/ms_deform_attn.py ct|||zdk7rtd|d|||z}||z|k(sJdd|_||_||_||_||_tj|||z|zdz|_ tj|||z|z|_ tj|||_ tj|||_ |jy)z2Initialize MSDeformAttn with the given parameters.rz.d_model must be divisible by n_heads, but got z and z(`d_model` must be divisible by `n_heads`@rWN)rr ValueError im2col_stepd_modeln_levelsn_headsn_pointsr!r$sampling_offsetsattention_weights value_proj output_proj_reset_parameters)r/rrrr _d_per_headr3s r4rzMSDeformAttn.__init__s  W  !MgYV[\c[def f( W$/[1[[/      " '7X3E3PST3T U!#7Gh4F4Q!R))GW599Wg6  r5cHt|jjjdt j |j tjdtjz|j z z}t j|j|jgd}||jjdddz j|j ddd j!d|j"|j$d}t'|j$D]}|d d d d |d d fxx|dzzcc<t j(5t+j,|jd|j_d d d t|j0jjdt|j0j.jdt3|j4jjt|j4j.jdt3|j6jjt|j6j.jdy #1swYxYw) zReset module parameters.rrhg@rZTrrrrWN)rrrdatarmrnrrpmathpistackrtrsabsmaxr`repeatrrrno_gradr!rrrrrr)r/thetas grid_initrs r4rzMSDeformAttn._reset_parameterss$''..33R8dll%--@C$''MTXT`T`D`aKKvzz| .s5 11Q4!A$; srlrrZ)rYrXrkg?z5Last dim of reference_points must be 2 or 4, but got .)r[sumr masked_fillfloatr`rrrrrrrsoftmaxrm as_tensorrYrXfliprr r)r/query refer_bboxr= value_shapes value_maskbslen_qlen_vrr num_pointsoffset_normalizeraddsampling_locationsoutputs r4rJzMSDeformAttn.forwardsH"KKO E A5 55>>>&  !%%j&;U1XFE 2udllDLLDLL4PQ007<>r5$,,X\XeXehlhuhuXuvII&7<AA"eT\\[_[h[hjnjwjwx%%b) ? % EKKX]XdXd e j jkm n "%6tT4DRS7S%TTC!+Aq$4,B!Cc!I  1_"T]]2Z1dAtUVUW@W5XX[^^C!+Aq$4!,C!Ds!J TU_T``abc c4ULJ\^op''r5)r}rkrrkrK)rMrNrOrPrrrJrRrSs@r4rrs !.2$((r5rcneZdZdZddddej ddffd ZedZd Z d d Z xZ S) raK Deformable Transformer Decoder Layer inspired by PaddleDetection and Deformable-DETR implementations. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py https://github.com/fundamentalvision/Deformable-DETR/blob/main/models/deformable_transformer.py r}rirrkcht|tj||||_tj ||_tj||_t|||||_ tj ||_ tj||_ tj|||_||_tj ||_tj|||_tj ||_tj||_y)zKInitialize the DeformableTransformerDecoderLayer with the given parameters.)rN)rrr!r" self_attnr*r+r'r(r cross_attnr,r)r$linear1r-dropout3linear2dropout4norm3) r/rrd_ffnrr-rrr3s r4rz*DeformableTransformerDecoderLayer.__init__.s ..wQ 7+ \\'* 'w'8L 7+ \\'* yy%0  7+ yy0  7+ \\'* r5c||S||zS)z;Add positional embeddings to the input tensor, if provided.r7)r8r9s r4r:z0DeformableTransformerDecoderLayer.with_pos_embedDsv6&3,6r5c |j|j|j|j|}||j |z}|j |S)zHPerform forward pass through the Feed-Forward Network part of the layer.)rrr-rrr)r/tgttgt2s r4 forward_ffnz-DeformableTransformerDecoderLayer.forward_ffnIsL||DMM$((4<<3D*EFGDMM$''zz#r5c|j||x}} |j|jdd| jdd|jdd|djdd} ||j| z}|j |}|j |j|||j d|||} ||j| z}|j|}|j|S)z:Perform the forward pass through the entire decoder layer.rr)r>rW) r:r transposer+r(r unsqueezer,r)r) r/embedrfeatsshapes padding_maskr> query_posrCrDrs r4rJz)DeformableTransformerDecoderLayer.forwardOs##E955AnnQ[[A. Aq0A5??STVWCX'022355>Yq!_  c** 5!ood11%CZEYEYZ[E\^cek*, c** 5!&&r5rL) rMrNrOrPr!ReLUrr~r:rrJrRrSs@r4rr&sF #AT272779_`kl+,77 'r5rc0eZdZdZdfd Z ddZxZS)rz Implementation of Deformable Transformer Decoder based on PaddleDetection. https://github.com/PaddlePaddle/PaddleDetection/blob/develop/ppdet/modeling/transformers/deformable_transformer.py ct|t|||_||_||_|dk\r||_y||z|_y)zFInitialize the DeformableTransformerDecoder with the given parameters.rN)rrrrrreval_idx)r/r decoder_layerrr r3s r4rz%DeformableTransformerDecoder.__init__jsB !-< $$$,M zH7L r5c |} g} g} d} |j}t|jD]\}}|| |||| |||} ||| }tj|t |z}|j rb| j ||| |dk(r| j |nm| j tj|t | zn<||jk(r-| j ||| | j |n#|} |j r|jn|}tj| tj| fS)z4Perform the forward pass through the entire decoder.Nr) sigmoidrrrmrtrainingappendr detachr)r/rrrr bbox_head score_headpos_mlpr>rr dec_bboxesdec_clslast_refined_bboxrrbbox refined_bboxs r4rJz$DeformableTransformerDecoder.forwardrsG  '') !$++.HAu6:uflIW^_iWjkF9Q<'D == 0K)KLL}}}z!}V456%%l3%%emmD?K\;]4]&^_dmm#}z!}V45!!,/ , 26--,,.\J%/({{:& G(<<rs 4TT g4Kbii4Knl "lB)ryy)&VryyV, 1ryy 1"))"I"))I0X(299X(v:' :'z4=2994=r5