o H&iA] @shUddlZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddgZ e sGddl Z GdddZdd Zee jd _ee jd _dSdd lmZmZmZmZmZmZmZmZmZeeZer{zdd l m!Z!Wne"yze#d YnwGdddZ$e$Z%e$e&d<dde'fddZ(GdddZddde'dee)dfdeee'dfdefddZdS)N)DictListOptionalTuple TYPE_CHECKINGUnion) is_availableinit_device_mesh DeviceMeshc@s eZdZdS)_DeviceMeshStubN)__name__ __module__ __qualname__rrHC:\wamp64\www\opt\env\Lib\site-packages\torch/distributed/device_mesh.pyr sr cCsdSNrrrrr_init_device_mesh_stubsrztorch.distributed.device_mesh) _find_pg_by_ranks_and_tag_get_default_group_get_group_tagget_rankget_world_sizeinit_process_groupis_initialized new_group ProcessGroup) ArrayLikezCDeviceMesh requires numpy >= 1.21 to be installed for type checkingc@seZdZdddZdddZddd ed eddfd d Zdddedfd dZ dddeefddZ e dedefddZ e dedefddZ dS)_MeshEnvreturnNcCsg|_i|_dSr) mesh_stackchild_to_parent_mappingselfrrr__init__:s z_MeshEnv.__init__r cCs t|jdkr td|jdS)Nrz#No device mesh is currently active!)lenr RuntimeErrorr!rrrget_current_mesh>s z_MeshEnv.get_current_mesh device_meshmesh_dim mesh_dim_namec Csl|}|jd|d|j|}|D]}t|j||fdd}||vr'|}q|j|g|_||j|<|S)Nr$Fmesh_dim_names_init_process_groups) rmeshswapdimsreshapesizer device_type_dim_group_infosr ) r"r(r)r*Zcur_rankpg_ranks_by_dimZmesh_1dZsub_meshZ res_sub_meshrrrcreate_child_meshCs"  z_MeshEnv.create_child_meshcCs|j|dSr)r get)r"r(rrrget_parent_mesh\sz_MeshEnv.get_parent_meshcCsH||}|j}|r"|r"t|dksJd|d}|jr"||SdS)z Return the index of the mesh dim in the parent mesh. The device_mesh passed in needs to be sliced out from a parent mesh. z%The child mesh can only be a 1D mesh.rN)r7r,r%_get_mesh_dim_by_name)r"r(Z parent_meshZchild_mesh_dim_namesZchild_mesh_dim_namerrrget_parent_mesh_dim_s  z_MeshEnv.get_parent_mesh_dimr2cCs t|Sr)_get_device_handle device_countr2rrrnum_devices_per_hosto z_MeshEnv.num_devices_per_hostcCstt|Sr)rrr>r=rrr num_hostsssz_MeshEnv.num_hostsrNrr )r r rr#r'intstrr5rr7r: staticmethodr>r@rrrrr9s$   r_mesh_resourcescudar2cCs tt|dS)a: Get the module corresponding to the device_type which is cuda or cuda-like device. For example, when the device_type is cuda, the module `torch.cuda` is returned. Return None when there is no corresponding module for device_type, otherwise return the corresponding module. N)getattrtorchr=rrrr;{s r;c @seZdZUdZeed<ejed<ee edfed<ddddede ejd fdee edfd e d df d d Z ddZ ddZd1ddZd2ddZd efddZddZded e fddZded dfddZ d3d ee eefd e eeeffd!d"Zd3d eed efd#d$Zed efd%d&Zed e edffd'd(Zd efd)d*Zd3d ee eefd efd+d,Zd eeefd-d.Z ded efd/d0Z!dS)4r a DeviceMesh represents a mesh of devices, where layout of devices could be represented as a n-d dimension array, and each value of the n-d dimensional array is the global id of the default process group ranks. DeviceMesh could be used to describe the layout of devices across the cluster, and serves as a proxy for communication among the device lists within the cluster. DeviceMesh can be used as a context manager. .. note:: DeviceMesh follows SPMD programming model, which means the same PyTorch Python program is running on all processes/ranks in the cluster. Therefore, users need to make sure the `mesh` array (which describes the layout of devices) should be identical across all ranks. Inconsistent `mesh` will lead to silent hang. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh (ndarray): A multi-dimensional array or an integer tensor describing the layout of devices, where the IDs are global IDs of the default process group. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. A reduction over the first dimension of mesh will reduce across columns (0, 4), .. and (3, 7), a reduction over the second dimension of mesh reduces across rows (0, 1, 2, 3) and (4, 5, 6, 7). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r2r..r,NTr+rr-rcCs||_t|tjr |ntj|tjd|_||_t |j |_ t |j |jjt|f|_|dkr@||rB|dSdSdS)N)ZdtypeZxla)r2 isinstancerITensordetachZtensorrCr.r,tupleflattentolist_flatten_mesh_listhashshapeid_hash_get_or_create_default_groupr-)r"r2r.r,r-rrrr#s   zDeviceMesh.__init__cCst}|stt}|j|krtd|jdt|j}|sK|rK|}||krC||dkrCtd|d|d|jd| t ||jt k }| ddvs\J| ddkri|d nd|_tS) Nz=Mesh should not be bigger than default world size, but found z ranks!rz8DeviceMesh only support homogeneous hardware, but found z ranks and  z devices!)rr8)rrrr.numelr&r;r2r<Z set_devicerZnonzeror1rO_coordinate_on_dimr)r"Zdefault_initializedZ world_sizeZ device_handler>Z rank_coordsrrrrUs8  z'DeviceMesh._get_or_create_default_groupcCsg}|jjdkr |jtkr |tttttfnGt|jjD]@}|j d| d|j |}|D],}| }t |d}||vret||kr\td|jd|d|t||fq9q&||_dS)Nr8r$)ZrankszFEach device mesh dimension should get only one process group, but got z in !)r.ndimrWrappendrrlistranger/r0r1rOrrr%r&r3)r"Zdim_group_infosdimr4Zdim_meshZsubgroup_ranksZ dim_grouprrrr-s6        zDeviceMesh._init_process_groupscCstj||Sr)rFrr[r!rrr __enter__s zDeviceMesh.__enter__cCstjdSr)rFrpop)r"exc_type exc_value exc_tracebackrrr__exit__"szDeviceMesh.__exit__cCsd|jdS)Nz DeviceMesh())r.rOr!rrr__repr__&szDeviceMesh.__repr__cCs|jSr)rTr!rrr__hash__)szDeviceMesh.__hash__othercCsBt|tsdSt|jt|jkrdS|jj|jjko |j|jkS)NFT)rJr rSr.rRrP)r"rhrrr__eq__,s  zDeviceMesh.__eq__r*cCs<|jjdkrtd|jjd||}t|||}|S)a Slice the current DeviceMesh based on the mesh_dim_name given to create a child DeviceMesh. Args: mesh_dim_name (str): the name of the mesh dimension of the parent DeviceMesh to create a child DeviceMesh for. Returns: A :class:`DeviceMesh` object The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh["tp"] on rank 0, 1, 2, 3 would return a 1D child DeviceMesh:([0, 1, 2, 3]). Calling mesh["tp"] on rank 4, 5, 6, 7 would return a 1D child DeviceMesh:([4, 5, 6, 7]). Calling mesh["dp"] on rank 0, 4 would return a 1D child DeviceMesh:([0, 4]). Calling mesh["dp"] on rank 1, 5 would return a 1D child DeviceMesh:([1, 5]). Calling mesh["dp"] on rank 2, 6 would return a 1D child DeviceMesh:([2, 6]). Calling mesh["dp"] on rank 3, 7 would return a 1D child DeviceMesh:([3, 7]). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r8zCannot slice a DeviceMesh with z dimension.)r.rZr&r9rFr5)r"r*r)Zsubmeshrrr __getitem__6s  zDeviceMesh.__getitem__r)cCst|ds td|jjdkrt|jdS|dur+t|tr$||}t|j|Sg}t |jjD] }| t|j|q3|S)a Returns a list of ProcessGroups corresponding to the mesh dimensions, or returns a single ProcessGroup if mesh_dim is specified or the given mesh has only one mesh dimension. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: A list of :class:`ProcessGroup` object when `mesh_dim` is not specified for a DeviceMesh with more than 1 dimension; otherwise, returns a single :class:`ProcessGroup` object. r3z*DeviceMesh process groups not initialized!r8rN) hasattrr&r.rZrr3rJrDr9r]r[)r"r)Z dim_groupsZith_dimrrr get_group[s     zDeviceMesh.get_groupcCs|dur |jS|j|Sr)r.rWr1)r"r)rrrr1~szDeviceMesh.sizecCs|jjSr)r.rZr!rrrrZszDeviceMesh.ndimcCs t|jjSr)rMr.rRr!rrrrRr?zDeviceMesh.shapecCstS)z: Returns the current global rank. )rr!rrrrszDeviceMesh.get_rankcCsX|jdkr|durtd|jjdd|durd}||}t|ts(Jdt|S)a{ Returns the local rank of the given mesh_dim of the DeviceMesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: An integer denotes the local rank. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 0, 1, 2, 3 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 4, 5, 6, 7 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 0, 4 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 1, 5 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 2, 6 would return 2. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 3, 7 would return 3. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r8NzFound the DeviceMesh have z dimensionszJOptional kwarg `mesh_dim` needs to be specified when device_mesh.ndim > 1.rz1We expect ProcessGroup before calling `get_rank`!)rZr&r.rlrJrr)r"r)Zmesh_dim_grouprrrget_local_ranks zDeviceMesh.get_local_rankcCs|jr|jSdS)z Return the relative indices of this rank relative to all dimensions of the mesh. If this rank is not part of the mesh, return None. N)rXr!rrrget_coordinateszDeviceMesh.get_coordinatecCsP|jdus t|jdkrtd||jvr"td|dd|j|j|S)NrzNo `mesh_dim_names` found.zMesh dimension 'z' does not exist.zAvailable mesh dimensions are: )r,r%KeyErrorindex)r"r*rrrr9s    z DeviceMesh._get_mesh_dim_by_namerBrAr)"r r r__doc__rD__annotations__rIrKrrrboolr#rUr-r_rdrfrgobjectrirjrCrrrlr1propertyrZrRrrmrnr9rrrrr sR '   # +  & # *)r, mesh_shape.r,rcCs|dur.tt|t|krtdd|t|t|kr.tddt|dt|dtt||}t|||d}|S) aG Initializes a `DeviceMesh` based on `device_type`, `mesh_shape`, and `mesh_dim_names` parameters. This creates a DeviceMesh with an n-dimensional array layout, where `n` is the length of `mesh_shape`. If `mesh_dim_names` is provided, each dimension is labeled as `mesh_dim_names[i]`. .. note:: `init_device_mesh` follows SPMD programming model, meaning the same PyTorch Python program runs on all processes/ranks in the cluster. Ensure `mesh_shape` (the dimensions of the nD array describing device layout) is identical across all ranks. Inconsistent `mesh_shape` may lead to hanging. .. note:: If no process group is found, init_device_mesh will initialize distributed process group/groups required for distributed communications behind the scene. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". mesh_shape (Tuple[int]): A tuple defining the dimensions of the multi-dimensional array describing the layout of devices. mesh_dim_names (Tuple[str], optional): A tuple of mesh dimension names to assign to each dimension of the multi-dimensional array describing the layout of devices. Its length must match the length of `mesh_shape`. Each string in `mesh_dim_names` must be unique. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> mesh_1d = init_device_mesh("cuda", mesh_shape=(8,)) >>> mesh_2d = init_device_mesh("cuda", mesh_shape=(2, 8), mesh_dim_names=("dp", "tp")) Nz"Each mesh_dim_name must be unique.z/Found repeated mesh_dim_name in mesh_dim_names z6mesh_shape and mesh_dim_names should have same length!zFound len(mesh_dim_names): z and len(mesh_shape):.)r2r.r,) r%setr&rIZarangemathprodviewr )r2rvr,r.r(rrrr s$()rG)*loggingrytypingrrrrrrrIZtorch.distributedr__all__sysr rmodulesr r Z"torch.distributed.distributed_c10drrrrrrrrr getLoggerr loggerZ numpy.typingr ImportErrorwarningrrFrrrDr;rCrrrrsT    ,  @ N