PhA] UddlZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddgZ e s:ddl Z GddZdZee j d _ee j d _ydd lmZmZmZmZmZmZmZmZmZej:eZer dd l m!Z!Gd dZ$e$Z%e$e&d<dde'fdZ(GddZddde'dee)dfdeee'dfdefdZy#e"$rejGd Y\wxYw)N)DictListOptionalTuple TYPE_CHECKINGUnion) is_availableinit_device_mesh DeviceMeshc eZdZy)_DeviceMeshStubN)__name__ __module__ __qualname__hC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/device_mesh.pyr r s rr cyNrrrr_init_device_mesh_stubrs rztorch.distributed.device_mesh) _find_pg_by_ranks_and_tag_get_default_group_get_group_tagget_rankget_world_sizeinit_process_groupis_initialized new_group ProcessGroup) ArrayLikezCDeviceMesh requires numpy >= 1.21 to be installed for type checkingceZdZddZddZdddededdfd Zdddedfd Z dddeefd Z e d edefd Z e d edefdZ y)_MeshEnvreturnNc g|_i|_yr) mesh_stackchild_to_parent_mappingselfs r__init__z_MeshEnv.__init__:s02DOIKD (rr cft|jdk(r td|jdS)Nrz#No device mesh is currently active!)lenr% RuntimeErrorr's rget_current_meshz_MeshEnv.get_current_mesh>s.4??#q("#HII??2& &r device_meshmesh_dim mesh_dim_namecH|j}|jjd|jd|jj |}|D]#}t |j ||fd}||vs"|}%|j|g_||j|<|S)Nr+Fmesh_dim_names_init_process_groups) rmeshswapdimsreshapesizer device_type_dim_group_infosr&) r(r/r0r1cur_rankpg_ranks_by_dimmesh_1dsub_mesh res_sub_meshs rcreate_child_meshz_MeshEnv.create_child_meshCs #++-H)..77HEMMK$$))(3O+%++$1#3).  w&#+L+.9-I-I(-S,TL )9DD ( ( 6 rc:|jj|dSr)r&get)r(r/s rget_parent_meshz_MeshEnv.get_parent_mesh\s//33KF Frc|j|}|j}|r9|r7t|dk(sJd|d}|jr|j|Sy)z Return the index of the mesh dim in the parent mesh. The device_mesh passed in needs to be sliced out from a parent mesh. z%The child mesh can only be a 1D mesh.rN)rDr4r,_get_mesh_dim_by_name)r(r/ parent_meshchild_mesh_dim_nameschild_mesh_dim_names rget_parent_mesh_dimz_MeshEnv.get_parent_mesh_dim_sm ..{;K#.#=#= 3,-2;:;2&:1&=#--&<<=PQQrr:c4t|jSr)_get_device_handle device_countr:s rnum_devices_per_hostz_MeshEnv.num_devices_per_hostos%k2??A ArcBttj|zSr)rr"rPrOs r num_hostsz_MeshEnv.num_hostsss"#x'D'D['QQ Qrr#Nr#r )rrrr)r.intstrrArrDrK staticmethodrPrRrrrr"r"9s L '  + 7: KN  2 G| G@V G < HSM   Bc Bc B  B  R3 R3 R  Rrr"_mesh_resourcesr:c$tt|dS)a: Get the module corresponding to the device_type which is cuda or cuda-like device. For example, when the device_type is cuda, the module `torch.cuda` is returned. Return None when there is no corresponding module for device_type, otherwise return the corresponding module. N)getattrtorchrOs rrMrM{suk400rc eZdZUdZeed<ejed<ee edfed<ddddede ejd fdee edfd e d df d Z d Z dZd dZd!dZd efdZdZded e fdZded dfdZ d"dee eefd e eeeffdZd"deed efdZed efdZed e edffdZd efdZd"dee eefd efdZd eeefdZ ded efdZ!y)#r a DeviceMesh represents a mesh of devices, where layout of devices could be represented as a n-d dimension array, and each value of the n-d dimensional array is the global id of the default process group ranks. DeviceMesh could be used to describe the layout of devices across the cluster, and serves as a proxy for communication among the device lists within the cluster. DeviceMesh can be used as a context manager. .. note:: DeviceMesh follows SPMD programming model, which means the same PyTorch Python program is running on all processes/ranks in the cluster. Therefore, users need to make sure the `mesh` array (which describes the layout of devices) should be identical across all ranks. Inconsistent `mesh` will lead to silent hang. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh (ndarray): A multi-dimensional array or an integer tensor describing the layout of devices, where the IDs are global IDs of the default process group. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. A reduction over the first dimension of mesh will reduce across columns (0, 4), .. and (3, 7), a reduction over the second dimension of mesh reduces across rows (0, 1, 2, 3) and (4, 5, 6, 7). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r:r6.r4NTr3r r5r#c||_t|tjr|j n$tj |tj |_||_t|jjj|_ t|j|jjt|f|_|dk7r$|j#|r|j%yyy)N)dtypexla)r: isinstancer[TensordetachtensorrUr6r4tupleflattentolist_flatten_mesh_listhashshapeid_hash_get_or_create_default_groupr5)r(r:r6r4r5s rr)zDeviceMesh.__init__s +D dELL1 \\$eii8 I #1D ',DII,=,=,?,F,F,H&ID #t66 DRSDJe#113'--/( $rc t}|s tt}|jj |kDr't d|jj dt |j}|sZ|rX|j}||kDr'||zdk7rt d|d|d|jd|jt|z|jtk(j}|jddvsJ|jddkDr|djnd|_tS) Nz=Mesh should not be bigger than default world size, but found z ranks!rz8DeviceMesh only support homogeneous hardware, but found z ranks and  z devices!)rrF)rrrr6numelr-rMr:rN set_devicernonzeror9rf_coordinate_on_dimr)r(default_initialized world_size device_handlerP rank_coordss rrlz'DeviceMesh._get_or_create_default_groupsO"0"2 &"$')Jyy :-"STXT]T]TcTcTeSffmn/t/?/?@M&=(5'A'A'C$!55"%99Q>&R%,k2F1GqIYIYHZZce((6J)JK 99 2;;=K##A&&0 00+6+;+;A+>+B A%%'  #&' 'rc g}|jjdk(rk|jjtk(rF|j t t tttf||_yt|jjD]}|jjd|jd|jj|}|D]w}|j}t|}|j|vs2t||kDrt!d|jd|d|j t ||fy||_y)NrFr+)rankszFEach device mesh dimension should get only one process group, but got z in !)r6ndimrorappendrrlistranger7r8r9rfrrr,r-r;)r(dim_group_infosdimr=dim_meshsubgroup_ranks dim_groups rr5zDeviceMesh._init_process_groupssM<>Oyy~~"tyy'8N#345@%4D !1!0C'+ii&8&8S&A&I&IDIINN3/'O %4)1):%.N$C ==?n<"?3c9&2&lmqmzmzl{|**8)9%<'"!",22!/ !:N K%410%4D !rcDtjj||Sr)rXr%r{r's r __enter__zDeviceMesh.__enter__s  & & - -d 3Krc@tjjyr)rXr%pop)r(exc_type exc_value exc_tracebacks r__exit__zDeviceMesh.__exit__"s  & & * * ,rc>d|jjdS)Nz DeviceMesh())r6rfr's r__repr__zDeviceMesh.__repr__&s !1!1!3 4A6 6rc|jSr)rkr's r__hash__zDeviceMesh.__hash__)s :: rotherc t|tsyt|jt|jk(ry|jj|jjk(xr|j |j k(S)NFT)r`r rjr6rirg)r(rs r__eq__zDeviceMesh.__eq__,sceZ0$))}5::. 5::#3#33H++u/G/GG rr1c|jjdkr#td|jjd|j|}tj |||}|S)a Slice the current DeviceMesh based on the mesh_dim_name given to create a child DeviceMesh. Args: mesh_dim_name (str): the name of the mesh dimension of the parent DeviceMesh to create a child DeviceMesh for. Returns: A :class:`DeviceMesh` object The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh["tp"] on rank 0, 1, 2, 3 would return a 1D child DeviceMesh:([0, 1, 2, 3]). Calling mesh["tp"] on rank 4, 5, 6, 7 would return a 1D child DeviceMesh:([4, 5, 6, 7]). Calling mesh["dp"] on rank 0, 4 would return a 1D child DeviceMesh:([0, 4]). Calling mesh["dp"] on rank 1, 5 would return a 1D child DeviceMesh:([1, 5]). Calling mesh["dp"] on rank 2, 6 would return a 1D child DeviceMesh:([2, 6]). Calling mesh["dp"] on rank 3, 7 would return a 1D child DeviceMesh:([3, 7]). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rFzCannot slice a DeviceMesh with z dimension.)r6rzr-rGrXrA)r(r1r0submeshs r __getitem__zDeviceMesh.__getitem__6sa8yy~~""5diinn5E[Q11-@H%77h VGNrr0ct|ds td|jjdk(rt |j dS|6t |tr|j|}t |j |Sg}t|jjD]&}|jt |j |(|S)a Returns a list of ProcessGroups corresponding to the mesh dimensions, or returns a single ProcessGroup if mesh_dim is specified or the given mesh has only one mesh dimension. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: A list of :class:`ProcessGroup` object when `mesh_dim` is not specified for a DeviceMesh with more than 1 dimension; otherwise, returns a single :class:`ProcessGroup` object. r;z*DeviceMesh process groups not initialized!rFr) hasattrr-r6rzrr;r`rVrGr}r{)r(r0 dim_groupsith_dims r get_groupzDeviceMesh.get_group[s"4!34"#OPPyy~~"0$2G2G2JKK#h,#99(CH0$2G2G2QRR $TYY^^4G%%143H3H3QR 5"!rcp||jjS|jj|Sr)r6ror9)r(r0s rr9zDeviceMesh.size~s*(0(8499??$ VdiinnX>V Vrc.|jjSr)r6rzr's rrzzDeviceMesh.ndims99>> !rc@t|jjSr)rdr6rir's rrizDeviceMesh.shapes) )rctS)z: Returns the current global rank. )rr's rrzDeviceMesh.get_ranks : rc|jdkDr&|$td|jjdd|d}|j|}t |t sJdt |S)a{ Returns the local rank of the given mesh_dim of the DeviceMesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: An integer denotes the local rank. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 0, 1, 2, 3 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 4, 5, 6, 7 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 0, 4 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 1, 5 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 2, 6 would return 2. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 3, 7 would return 3. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rFzFound the DeviceMesh have z dimensionszJOptional kwarg `mesh_dim` needs to be specified when device_mesh.ndim > 1.rz1We expect ProcessGroup before calling `get_rank`!)rzr-r6rr`rr)r(r0mesh_dim_groups rget_local_rankzDeviceMesh.get_local_ranks8yy1}!1"00@ L`!!^^H5N  CB CN+ +rc6|jr |jSdS)z Return the relative indices of this rank relative to all dimensions of the mesh. If this rank is not part of the mesh, return None. N)rrr's rget_coordinatezDeviceMesh.get_coordinates /3.E.E4** O4 Orc|jt|jdk(r td||jvrtd|dd|j|jj|S)NrzNo `mesh_dim_names` found.zMesh dimension 'z' does not exist.zAvailable mesh dimensions are: )r4r,KeyErrorindex)r(r1s rrGz DeviceMesh._get_mesh_dim_by_names""*c$2E2E.F!.K0D$7$77&}o5FG5d6I6I5JK&&,,]; ;rrTrSr)"rrr__doc__rV__annotations__r[rarrrboolr)rlr5rrrrobjectrrrUrrrr9propertyrzrirrrrGrrrr r s% Nll sCx119=)-  0 0 k12 0 %U38_5  0 #'  0 0>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> mesh_1d = init_device_mesh("cuda", mesh_shape=(8,)) >>> mesh_2d = init_device_mesh("cuda", mesh_shape=(2, 8), mesh_dim_names=("dp", "tp")) z"Each mesh_dim_name must be unique.z/Found repeated mesh_dim_name in mesh_dim_names z6mesh_shape and mesh_dim_names should have same length!zFound len(mesh_dim_names): z and len(mesh_shape):.)r:r6r4) r,setr-r[arangemathprodviewr )r:rr4r6r/s rr r sP  %3~&'3~+>>"8EnEUV :#n"55"L1#n2E1FF[\_`j\k[llmn ||DIIj1277 C #) r)cuda)*loggingrtypingrrrrrrr[torch.distributedr __all__sysr rmodulesr r "torch.distributed.distributed_c10drrrrrrrrr getLoggerrlogger numpy.typingr ImportErrorwarningr"rXrrVrMrUrrrrs0 DD * | ,~   ?NCKK/0;0KK'   W  x (F  . >R>R@!) OX*11F<F