Ph%TUddlmZmZddlmZddlZddlmZmZm Z m Z ddl Z ddl m Z mZmZmZddlmZddlmcmcmcmZddlmZe rdd lmZGd d eZeGd d eZGddeZiZ ee!eeeffe"d<dZ#defdZ$dZ%eGddeZ&dZ'y))ABCabstractmethod) dataclassN)CallableDictList TYPE_CHECKING) check_tensorget_chunked_dim_sizeget_split_size(validate_non_overlapping_shards_metadata) ShardMetadata)_decorator_func) ShardedTensorceZdZdZy) PlacementSpecz Base class representing the placement of an entity. Subclasses of this class can be used to specify customized placements which might not be covered by existing APIs. N)__name__ __module__ __qualname____doc__uC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/_shard/sharding_spec/api.pyrrs   rrcJeZdZUdZej j ed<dZy)DevicePlacementSpecz Associates placement of an entity with a single device. Args: device(:class:`torch.distributed._remote_device`): The device to place the entity on. devicect|jtjjs/tjj |j|_yyN) isinstancertorch distributed_remote_device)selfs r __post_init__z!DevicePlacementSpec.__post_init__,s>$++u'8'8'G'GH++::4;;GDKIrN) rrrrr!r"r#__annotations__r%rrrrr!s!    , ,,HrrceZdZdZedej dejdejfdZ ed dejde dd fd Z y) ShardingSpecz: Base class representing sharding specifications. tensor_sizestensor_propertiesreturncy)a Given a global tensor size, define how to shard a tensor like this shape across ranks, return ShardedTensorMetadata Args: tensor_sizes (:class:`torch.Size`): The tensor shape to shard on, a `torch.Size` object that represents the tensor shape to be sharded according to the ShardingSpec. tensor_properties(:class:`torch.distributed._shard.sharded_tensor.TensorProperties): Tensor properties used to create a ShardedTensor. Returns: A :class:`ShardedTensorMetadata` object that encodes the information about the layout of the ShardedTensor and its properties. Nrr$r)r*s rbuild_metadatazShardingSpec.build_metadata4rNtensorsrc_rankrcy)a Given a global tensor on src_rank, shard this tensor across ranks within the process group, return a ShardedTensor. Args: tensor (:class:`torch.Tensor`): Tensor needs to be sharded. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Returns: A :class:`ShardedTensor` sharded from the given tensor. Nrr$r0r1 process_groups rshardzShardingSpec.shardGr/rrN)rrrrrr!Sizesharded_tensor_metaTensorPropertiesShardedTensorMetadatar.Tensorintr5rrrr(r(0ss %*ZZ *=*N*N 0EE  $ ELL C Tc  rr(_CUSTOM_SHARDING_SPEC_OPScVt|j}|tvxr |t|vS)zQ Returns whether or not the ShardingSpec has a custom op implementation. )typerr=) sharding_specop class_names r_has_custom_oprC\s1m$11J 2 2 br=VWa=b7bbrrAct|j}t||std|d|t||}|||||S)zA Calls the custom op for this ShardingSpec if it exists. z Custom op: z not registered for )r?rrC RuntimeErrorr=)r@rAtypesargskwargsr4rBfuncs r_dispatch_custom_oprJcsWm$11J - ,[,@ MNN $Z 0 4D tV] 33rc|j}|tvr it|<tjt|t|S)z Decorator to allow custom registration of ops. Args: sharding_spec_class(type): The ShardingSpec for which we need to add this custom op. func(Callable): The op to override (ex: torch.bmm) )rAop_table)rr= functoolspartialr)sharding_spec_classrIrBs rcustom_sharding_spec_oprPmsD%11J2202!*-    *:6 rceZdZUdZeeed<dZdejde jde jfdZ d d ejd edd fd Zy)EnumerableShardingSpeca@ This is a type of PlacementSpec that allows users to specify a generic sharding scheme by enumerating exactly how each shard is laid out. Args: shards(List[ShardMetadata]): List of :class:`ShardMetadata` objects representing each shard. Note that none of the shards should overlap. shardsc `t|jdk(rtd|jd}|jD]X}|dk7r<|t|jk7r$td|dt|jt|j}Zt |jy)NrzEmpty shard list provided: z%Found inconsistent ranks for shards: z and )lenrS ValueError shard_offsetsr)r$rankr5s rr%z$EnumerableShardingSpec.__post_init__s t{{ q :4;;-HI I[[Erzdc%*=*=&>> #HeTWX]XkXkTlSm!noou**+D! 1=rr)r*r+cpt|j|tj|j||Sr)r rSr8r:r-s rr.z%EnumerableShardingSpec.build_metadatas2 T[[,/"88 KK    rNr0r1rctd)Nz1EnumerableShardingSpec.shard not implemented yet!)NotImplementedErrorr3s rr5zEnumerableShardingSpec.shards!"UVVrr6)rrrrrrr&r%r!r7r8r9r:r.r;r<r5rrrrRrR~sk   >  %*ZZ  *=*N*N  0EE  WELLWCWTcWrrRc 8g}d}g}g}|D]}|j|j|j}|jt||j|jt |Dcgc] \}}|dk7s |} }}t | dk(rt | dk7rd}n|s| d}|| dk7sd}n|tt||dD cgc]\} } |  }} } ddl m } | ||} t|D cgc]} | | c} }t|}t |}t||}ttt |Dcgc]}t|||c}}||k(r| St|Scc}}wcc} } wcc} wcc}w)a Infer the sharding spec from the metadata of each shard of a ShardedTensor. If the tensor is sharded only on one dimension, we can then verify whether it's a ChunkShardingSpec or not. The way to verify it is to first get the total length and perform a chunk sharding with the given placements to see if we can have the same chunk size as the given shards_metadata. If not, we assume it's enum sharded. Args: shards_metadata (List[ShardMetadata]): List of Metadata of local shards. Returns: A :class:`torch.distributed._shard.sharding_spec.ShardingSpec` object of sharding spec for one sharded tensor. Nrr c |dS)Nrr)es rz;_infer_sharding_spec_from_shards_metadata..s STUVSWr)key)ChunkShardingSpec)dim placements)append placementrXsum shard_sizes enumeraterVsortedzipchunk_sharding_specrbr ranger rR)shards_metadatardchunk_sharding_dimchunk_offset_listshard_size_listshard_metadata local_offsetsidxr_ shard_dims_xrb chunk_specrhshard_total_lengthchunks split_sizechunk_shard_sizess r)_infer_sharding_spec_from_shards_metadatar}sJO).223&44   ]!34~99:(1-(@K(@fc1AFc(@ K z?a   z?a !%  "!+A  :a= 0!%  +*.%!%6 !CX X$!QAX   ;&"! _M_a 23_MN  -Z#$6? "!Z1 1C%%7SI1   + +  !/ 22SL( N  s8 FF! F  FF)(abcrr dataclassesrrMtypingrrrr r! _internalsr r r r!torch.distributed._shard.metadatar0torch.distributed._shard.sharded_tensor.metadatar"_shardsharded_tensormetadatar8*torch.distributed._shard.op_registry_utilsr'torch.distributed._shard.sharded_tensorrrrr(r=strr&rCrJrPrRr}rrrrs##!66  <NNFF C   H- H  H' 3' TBD4T(H*<%= =>Cc484" 'W\'W 'WTB3r