o IÆ&iAã@sÀddlZddlmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZGd d „d e ƒZd d „Zdd„ZGdd„deƒZGdd„deƒZGdd„deƒZGdd„deƒZdS)éN)ÚDictÚListé)Ú variables)Ú unimplemented)Úistypeé)ÚVariableTracker©ÚConstantVariablecs(eZdZ‡fdd„Zedd„ƒZ‡ZS)ÚDistributedVariablec s*tƒjdi|¤Žt ¡stdƒdSdS)Nz+torch.distributed package is not available!©)ÚsuperÚ__init__r Ú is_availabler)ÚselfÚkwargs©Ú __class__r úNC:\wamp64\www\opt\env\Lib\site-packages\torch/_dynamo/variables/distributed.pyr s ÿzDistributedVariable.__init__cCs tj ¡S©N)ÚtorchÚ distributedrr r r rrs z DistributedVariable.is_available)Ú__name__Ú __module__Ú __qualname__rÚ staticmethodrÚ __classcell__r r rrr s r cCs,t ¡sdSddlm}t |¡o||juS)NFr)ÚDTensor)r rZtorch.distributed._tensorrÚinspectÚ isfunctionZ from_local)Úvaluerr r rÚ is_from_locals r"cCs6t ¡sdSddlm}m}||g}t |¡o||vS)NFr)Ú_get_group_tagÚget_process_group_ranks)r rZ"torch.distributed.distributed_c10dr#r$rr )r!r#r$Zconstant_processgroup_functionsr r rÚis_constant_pg_functions sþr%csBeZdZ‡fdd„Zedd„ƒZ     d ‡fd d „ Z‡ZS)ÚPlacementClassVariablec ótƒjdi|¤Ž||_dS©Nr ©rrr!©rr!rrr rr2ó zPlacementClassVariable.__init__cCs.t ¡sdSddlm}t|ƒtuot||ƒS©NFr)Ú Placement)r rÚ)torch.distributed._tensor.placement_typesr-ÚtypeÚ issubclass©r!r-r r rÚis_placement_type6s z(PlacementClassVariable.is_placement_typeÚargsúList[VariableTracker]rúDict[str, VariableTracker]Úreturnr csft |jdd¡tjfvr+|jr+t |j¡}t|ƒ}t |jdd¡r+| |d||¡|Stƒ  |||¡S)NÚ__new__r) rÚgetattr_staticr!Úobjectr7ÚsourceÚPlacementVariableÚ call_methodrÚ call_function)rÚtxr3rZnew_objÚvarrr rr=@sÿ z$PlacementClassVariable.call_function©r3r4rr5r6r )rrrrrr2r=rr r rrr&1s   ÿÿþr&csJeZdZ‡fdd„Zedd„ƒZdd„Z   d‡fd d„ Z‡ZS)r;c r'r(r)r*rr rrSr+zPlacementVariable.__init__cCó"t ¡sdSddlm}t||ƒSr,)r rr.r-Ú isinstancer1r r rÚ is_placementWó  zPlacementVariable.is_placementcCó|jSr©r!©rr r rÚas_python_constantaóz$PlacementVariable.as_python_constantr3r4rr5r6r c sÊddlm}ddg}||vr\zt|jƒ}t |dd¡dus!Jdƒ‚t ||¡}Wn ty3d}Ynw|tjur>|  d¡Sdd„|Dƒ}d d „|  ¡Dƒ}||jg|¢Ri|¤Ž|St ƒ  ||||¡S) Nrr rÚ __setattr__Ú __getattr__zno custom getattr allowed!cSsg|]}| ¡‘qSr ©rH)Ú.0Úxr r rÚ |sz1PlacementVariable.call_method..cSsi|] \}}|| ¡“qSr rL)rMÚkÚvr r rÚ }sz1PlacementVariable.call_method..) Úr r/r!rr8ÚAttributeErrorr9rÚcreateÚitemsrr<) rr>Únamer3rr Úallowed_methodsZ value_typeÚmethodrr rr<ds&  ÿ ÿ  zPlacementVariable.call_methodr@) rrrrrrCrHr<rr r rrr;Rs   üûúr;csFeZdZ‡fdd„Zedd„ƒZdd„Zdedef‡fd d „ Z ‡Z S) ÚDeviceMeshVariablec r'r(r)r*rr rr…r+zDeviceMeshVariable.__init__cCrA)NFr)Ú DeviceMesh)r rZtorch.distributed.device_meshr[r)r!r[r r rÚis_device_mesh‰rDz!DeviceMeshVariable.is_device_meshcCrErrFrGr r rrH“rIz%DeviceMeshVariable.as_python_constantrWr6cs$|dkr t |jj¡Stƒ ||¡S)NÚndim)r rUr!r]rÚ var_getattr©rr>rWrr rr^–szDeviceMeshVariable.var_getattr) rrrrrr\rHÚstrr r^rr r rrrZ„s    rZcsbeZdZdZ‡fdd„Zdd„Zdd„Z  d‡fdd„ Z‡fdd„Ze dd„ƒZ ‡Z S)ÚProcessGroupVariablea£ We don't want a ProcessGroup object to end up in our output graph. But it's common for dynamo to intercept a PG that is then used to get info like rank() or world_size(), as well as passed to utility functions in distributed_c10d which desugar it into plain types like a ranklist and tag. For convenience and proper guarding, we construct a variable type. TODO: make it possible to use ProcessGroupVariable as input to simple functions like _expand_group without dynamo complaining about making a proxy for it. It is not a tensor-like type, and we don't want a proxy- but dynamo assumes torch library functions are dealing with tensor-like types and would have proxies for their args. TODO: should we make this inherit VT instead of UDOV? Do we want any of the default behaviors or just graph-break whenever one of our special cases is not hit? c r'r(r)r*rr rr¯r+zProcessGroupVariable.__init__cCrErrFrGr r rrH³rIz'ProcessGroupVariable.as_python_constantcCs t|jƒSr)r/r!rGr r rÚ python_type¶s z ProcessGroupVariable.python_typer3r4rr5r6r csF|dkr tj |j ¡¡S|dkrtj |j ¡¡Stƒ ||||¡S)NÚrankÚsize)rr rUr!rcrdrr<)rr>rWr3rrr rr<¹s z ProcessGroupVariable.call_methodcs,ˆdvrt ‡‡‡fdd„¡Stƒ ˆˆ¡S)N)rcrdcsˆ ˆˆ||¡Sr)r<)r3r©rWrr>r rÚÊsz2ProcessGroupVariable.var_getattr..)rZLambdaVariablerr^r_rrerr^Çs ÿz ProcessGroupVariable.var_getattrcCs2t ¡sdSddlm}ddlm}t|||fƒS)NFr)Ú ProcessGroup)ÚFakeProcessGroup)r rZtorch._C._distributed_c10drgZ+torch.testing._internal.distributed.fake_pgrhr)r!rgrhr r rÚis_process_groupÏs   z%ProcessGroupVariable.is_process_groupr@) rrrÚ__doc__rrHrbr<r^rrirr r rrraœs üûú ra)rÚtypingrrrrSrÚexcrÚutilsrÚbaser Zconstantr r r"r%r&r;rZrar r r rÚs      !2