Ph'&ddlmZmZmZmZddlZddlmcmcm Z ddl m Z m Z mZmZmZddlmZeeeeeffZdedefdZdeedeefdZd ej.d e d e dej.fd ZGd dej2j4Zy))castDictListTupleN)_Partial DTensorSpec Placement ReplicateShard) DeviceMeshvalreturnc|\}\}}|js|jr|jr| S|js|jr|jr|Sy)z[ Replicate from inner to outer dimension. Shard from outer to inner dimension. r) is_replicate is_partialis_shard)r icurrenttargets qC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/_tensor/redistribute.py_replicate_then_shardrsd A!2!2!4':J:J:Lr    G$6$6$8foo>Ocddlm}|t}|t}g}|D]\}\}}|jr%|t t |j xxdz cc<|jr%|t t |j xxdz cc<t|t rt|t rt|j |j k7s||j ||j k7r<|j||tff|j|t|ff|j|||ff|S)aB Decompose Si -> Sj into Si -> R -> Sj There's 2 ways a shardings can differ within a mesh dimension: 1) sharding on different tensor dimensions, e.g. Shard(0) -> Shard(1) 2) different sub-shards of a repeated shard ("mis-aligned sharding") (Shard(0), Shard(0)) -> (Replicate(), Shard(0)) Here the Shard(0) -> Shard(0) for mesh dimension 2 is actually a reshard, because in the first case it's a sub-sharding of an already tensor dimension 0, and in the second case, it's the first sharding on tensor dimension 0. r) defaultdict) collectionsrintrrr dim isinstanceappendr )r rrepeat_dim_currentrepeat_dim_targetoutputrrrs r_decompose_reshardr$!s()4S)9(3C(8#%F # GV     tE7377 8A = 8 ??  d5&155 6! ; 6 w &65) vzz)%gkk26G 6SS MM1w 45 6 MM1y{F34 5 MM1w/0 1%!$( Mr local_tensor current_spec target_specc|j|jk7r tdd}|j}|j}tt t ||}t |}|jt|j}|D]\}\} } |j} |j|} | |cS| | k(r|}:| jr| jr%tt| } | j|||}nl| j!r0tt"| }|j%||j&||}n,t)d|d|d| j!rtt"| }| jr%tt| } | j+||||}n| jr/|j-|| dd \}}|| |j/}n| j!s Jd | tt"| }|j0|j0k7rCtd | jr(| jr|| z }nt)d|d|d|J|}|Jd |S) z This redistribute the local tensor (torch.Tensor) from the current DTensorSpec to the target DTensorSpec, which involves the necessary collective calls to transform the local shard of the DTensor from its current spec to the target spec. z)Cross device mesh comm not supported yet!N)key)mesh_dimzredistribute from z to z not supported yetF) with_padding contiguousz,Current placement should be shard but found z+Changing sharding dim is not supported yet!zredistribute failed!)meshNotImplementedError placementslist enumeratezipr$sortrget_coordinatesizerrrr _to_replicaterr _to_replicate_tensorshape RuntimeError _to_shard _split_tensorcloner)r%r&r'new_local_tensorcurrent_placementstarget_placementssorted_placements device_meshrrr my_coordinate num_chunks partial_speccurrent_placementtarget_placementshards_ shard_specs rredistribute_local_tensorrJKsK,,,!"MNN%00#..Ys+=?P'QRS*+<=45##K 1 GV#224  %%q%1    f +      !!##Hg6 #/#=#= +q$ !!#$($8!#4#I#I ,"4"4k1$ #();(>%5%9%99-E   ##%$0*#< "();(  rc f|j}|j}g}t|j|jD]P\}}|j s*|j r|j t @|j |Rt|jt||j}|j}t|||} tj| |j|j|j|j |j"|j%} | ddfSrP)r&rUr2r/rr r rr-tuplerQrVrJrWrXr8rRrSrT) rY grad_output previous_specr&r?rrr'r%r#output_dtensors rbackwardzRedistribute.backwards(( #(( -/"<#:#:Mrrs** //5sE)Y"6778 ~ # 'D0'T.5I'Td,,ddd \\ dNI 5>>**I r