o H&i@sddlZddlZddlmZddlZddlZddlZdefddZer-ej s-e dej j Z ej jZej jZej jZerddlmZmZmZmZmZmZmZmZmZmZm Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dde.fd d Z/ej0d krdd lm1Z1m2Z2d dl3Td dl3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:d dl;m;Z;mm?Z?e,dSGdddZ@e@ejAd_dS)N)EnumreturncCs ttjdS)a Return ``True`` if the distributed package is available. Otherwise, ``torch.distributed`` does not expose any other APIs. Currently, ``torch.distributed`` is available on Linux, MacOS and Windows. Set ``USE_DISTRIBUTED=1`` to enable it when building PyTorch from source. Currently, the default value is ``USE_DISTRIBUTED=1`` for Linux and Windows, ``USE_DISTRIBUTED=0`` for MacOS. _c10d_init)hasattrtorch_CrrEC:\wamp64\www\opt\env\Lib\site-packages\torch/distributed/__init__.py is_available s r z&Failed to initialize torch.distributed)Store FileStoreTCPStore ProcessGroupBackend PrefixStoreReducerLoggerBuiltinCommHookType GradBucketWork_DEFAULT_FIRST_BUCKET_BYTES_register_comm_hook_register_builtin_comm_hook_broadcast_coalesced"_compute_bucket_assignment_by_size_verify_params_across_processes_test_python_store DebugLevelget_debug_levelset_debug_levelset_debug_level_from_env_make_nccl_premul_sumrankcCst|kr;d}ttjtjrtj}tdt_ztd|ddW|dur-tj|t_n |dur:tj|t_wt dS)a Set a breakpoint, but only on a single rank. All other ranks will wait for you to be done with the breakpoint before continuing. This calls ``breakpoint()`` under the hood, so you can customize it using the normal facilities, e.g., ``PYTHONBREAKPOINT`` environment variable. Args: rank (int): Which rank to break on. Default: ``0`` NrzS !!! ATTENTION !!! Type 'up' to get to the frame that called dist.breakpoint(rank=z) )header) Zget_rank isinstancesysstdinio TextIOWrapperopen breakpointcloseZbarrier)r"Z old_stdinrrr r*;s&     r*win32) HashStore_round_robin_process_groups)*)_all_gather_base_reduce_scatter_base_create_process_group_wrapper_rank_not_in_group_coalescing_manager_CoalescingManager_get_process_group_name) rendezvous_create_store_from_optionsregister_rendezvous_handler)_remote_devicec@s eZdZdS)_ProcessGroupStubN)__name__ __module__ __qualname__rrrr r<sr<ztorch.distributed)r)Bosr%enumrpdbr'rboolr rr RuntimeErrorZ _DistErrorZ DistErrorZ_DistBackendErrorZDistBackendErrorZ_DistNetworkErrorZDistNetworkErrorZ_DistStoreErrorZDistStoreErrorZtorch._C._distributed_c10dr r r rrZ_BackendrrrrrrZ_Workrrrrrrrrrrr r!intr*platformr-r.Zdistributed_c10dr1r2r3r4r5r6r7r8r9r:Z remote_devicer;r<modulesrrrr s2 d $