o <&i|@s UdZddlZddlZddlZddlZddlZddlmZmZm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZm ZmZmZmZmZmZgdZedZedZedZed Z d Z!d Z"e Z#e Z$e e$geee e#ffZ%e ee e#ge$fZ&e Z'e e#ge'fZ(e e'ge#fZ)e d ee*ge*fZ+e e*geee e#e*ffZ,Gd ddeZ-e.Z/iZ0eee e-fe1d<GdddeZ2iZ3eee e2fe1d<iZ4ee*ee fe1d<dddddee de%de&dee*dee(dee)ddfddZ5  ddddddee de%de&dee+dee,dee*dee(dee)ddfd d!Z6dddddee de%de&dee*dee(dee)ddfd"d#Z7d$ee e fdeee e#ffd%d&Z8d'ee d(e#dee e ffd)d*Z9d$ee deee e#ffd+d,Z:d'ee d(e#dee fd-d.Z;d$ee d/fdeee e#ffd0d1Zd'ee d(e#defd6d7Z?d(e#de'fd8d9Z@d:e'de#fd;d<ZAd$ee e fdeee e#ffd=d>ZBd'ee d(e#dee e ffd?d@ZCeBZDeCZEd$ee e fdeee e#ffdAdBZFd'ee d(e#dee e ffdCdDZGd(e#de'fdEdFZHd:e'de#fdGdHZIdIee deee e#ffdJdKZJd'ee d(e#dee fdLdMZKe7eLee?e@eAdRdSe7e eBeCdTdOe7eeFeGdUeHeIde7eeJeKdVdOdWe deOfdXdYZPdWe de fdZd[ZQdWe$deOfd\d]ZRejSGd^d d ZTGd_d`d`eTZUeUZVdWe$daee deTfdbdcZWdWe$deee eTffdddeZXdaee dfeTde$fdgdhZYdWe$daee ddfdidjZZdWe$dee fdkdlZ[dWe$deTfdmdnZ\doe d/e fdWe$de$fdpdqZ]doe d/e fdWe$de$fdrdsZ^eeeeefZ_eeeeeeefZ`eee eee d/ffZae eeefge fZbe eeeefge fZce ege fZde e ge fZee ege e ge ffZfedte_eefdefebeee ffdudvZgedte`eeefdefeceeee ffdwdvZgedteedefedee ffdxdvZgedteadefeee fdydvZgdteadefeee fdzdvZgedteedoedee fdWe$de$fd{d|Zhedte_eefdoebeee fdWe$de$fd}d|Zhedte`eeefdoeceeee fdWe$de$fd~d|Zhdteadoeee dWe$de$fdd|Zhedteedoedee fdWe$de$fddZiedte_eefdoebeee fdWe$de$fddZiedte`eeefdoeceeee fdWe$de$fddZidteadoeee dWe$de$fddZide e geOfdWe$deOfddZjde e geOfdWe$deOfddZkedteededeeOfdWe$deOfddZledte_eefdebeeeOfdWe$deOfddZledte`eeefdeceeeeOfdWe$deOfddZldteadeeeOdWe$deOfddZledteededeeOfdWe$deOfddZmedte_eefdebeeeOfdWe$deOfddZmedte`eeefdeceeeeOfdWe$deOfddZmdteadeeeOdWe$deOfddZmdWe$dfeTdeee fddZnejSGdddZoGdddeZpiZqeerepfe1d<dfeTdeofddZsde'deTfddZtepeseteqd <ddfeTdeerde*fddZude*deTfddZvGdddZwdfeTde*fddZxdfeTde*fddZyde*deTfddZzde$de$dee fddZ{dS)a Contains utility functions for working with nested python data structures. A *pytree* is Python nested data structure. It is a tree in the sense that nodes are Python collections (e.g., list, tuple, dict) and the leaves are Python values. Furthermore, a pytree should not contain reference cycles. pytrees are useful for working with nested collections of Tensors. For example, one can use `tree_map` to map a function over all Tensors inside some nested collection of Tensors and `tree_leaves` to get a flat list of all Tensors inside some nested collection. pytrees are helpful for implementing nested collection support for PyTorch APIs. This pytree implementation is not very performant due to Python overhead To improve the performance we can move parts of the implementation to C++. N) defaultdictdeque namedtuple OrderedDict)AnyCallablecast DefaultDictDequeDictIterableList NamedTupleOptionalroverloadTupleTypeTypeVarUnion)PyTreeContext FlattenFunc UnflattenFuncDumpableContextToDumpableContextFnFromDumpableContextFnTreeSpecLeafSpecregister_pytree_node tree_flattentree_unflatten tree_leavestree_structuretree_map tree_map_ tree_map_onlytree_map_only_tree_alltree_any tree_all_only tree_any_onlytreespec_dumpstreespec_loadstreespec_pprintTSURNO_SERIALIZED_TYPE_NAME_FOUNDrc@s*eZdZUeeed<eed<eed<dS)NodeDeftype flatten_fn unflatten_fnN)__name__ __module__ __qualname__rr__annotations__rrr<r<>C:\wamp64\www\opt\env\Lib\site-packages\torch/utils/_pytree.pyr4ds  r4SUPPORTED_NODESc@s:eZdZUeeed<eed<eeed<ee ed<dS)_SerializeNodeDeftypserialized_type_nameto_dumpable_contextfrom_dumpable_contextN) r8r9r:rrr;strrrrr<r<r<r=r?us  r?SUPPORTED_SERIALIZED_TYPESSERIALIZED_TYPE_TO_PYTHON_TYPErArBrCclsr6r7rArBrCreturncCst|tvrt|dWdn1swYt||||||dzddlm}Wn ty9YdSw|j||||||ddS)azRegister a container-like type as pytree node. Args: cls: the type to register flatten_fn: A callable that takes a pytree and returns a flattened representation of the pytree and additional context to represent the flattened pytree. unflatten_fn: A callable that takes a flattened version of the pytree, additional context, and returns an unflattened pytree. serialized_type_name: A keyword argument used to specify the fully qualified name used when serializing the tree spec. to_dumpable_context: An optional keyword argument to custom specify how to convert the context of the pytree to a custom json dumpable representation. This is used for json serialization, which is being used in torch.export right now. from_dumpable_context: An optional keyword argument to custom specify how to convert the custom json dumpable representation of the context back to the original context. This is used for json deserialization, which is being used in torch.export right now. z& is already registered as pytree node.NrGr2) _cxx_pytree)_NODE_REGISTRY_LOCKr> ValueError_private_register_pytree_noderJ ImportError)rHr6r7rArBrCZcxxr<r<r=rs4   r to_str_fnmaybe_from_str_fncCs@tjddd|dus|durtdt||||||ddS)aRegister a container-like type as pytree node for the Python pytree only. Args: cls: the type to register flatten_fn: A callable that takes a pytree and returns a flattened representation of the pytree and additional context to represent the flattened pytree. unflatten_fn: A callable that takes a flattened version of the pytree, additional context, and returns an unflattened pytree. serialized_type_name: A keyword argument used to specify the fully qualified name used when serializing the tree spec. to_dumpable_context: An optional keyword argument to custom specify how to convert the context of the pytree to a custom json dumpable representation. This is used for json serialization, which is being used in torch.export right now. from_dumpable_context: An optional keyword argument to custom specify how to convert the custom json dumpable representation of the context back to the original context. This is used for json deserialization, which is being used in torch.export right now. zutorch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.) stacklevelNzpto_str_fn and maybe_from_str_fn is deprecated. Please use to_dumpable_context and from_dumpable_context instead.rG)warningswarnrM)rHr6r7rPrQrArBrCr<r<r=_register_pytree_nodes  rVcCstK|tvrt|dt|||}|t|<|du|duAr)td|d|dur6|jd|j}t||||}|t |<|t |<WddS1sPwYdS)zThis is an internal function that is used to register a pytree node type for the Python pytree only. End-users should use :func:`register_pytree_node` instead. zM is already registered as pytree node. Overwriting the previous registration.Nz7Both to_dumpable_context and from_dumpable_context for z must be None or registered..) rKr>rTrUr4rLr9r:r?rErF)rHr6r7rArBrCZnode_defserialize_node_defr<r<r=rMs4   "rMdcCt|t|fSNlistvalueskeysrYr<r<r= _dict_flattenrar^contextcCstt||Sr[)dictzipr^rcr<r<r=_dict_unflatten!rgcCs|dfSr[r<r`r<r<r= _list_flatten%ricCt|Sr[r]rfr<r<r=_list_unflatten)rjrm.cCs t|dfSr[rlr`r<r<r=_tuple_flatten- rncCrkr[)tuplerfr<r<r=_tuple_unflatten1rjrqcCst|t|fSr[)r]r5r`r<r<r=_namedtuple_flatten5srrcCstt||Sr[)rrrfr<r<r=_namedtuple_unflatten9rhrscCs|j|jd}|S)N) class_namefields)r8_fields)rcZjson_namedtupler<r<r=_namedtuple_serialize=srwdumpable_contextcCs(|d}t|ts Jt||d}|S)Nrtru) isinstancerDr)rxrtrcr<r<r=_namedtuple_deserializeEsrzcCrZr[r\r`r<r<r=_ordereddict_flattenLrbr{cCstddt||DS)Ncss|] \}}||fVqdSr[r<).0keyvaluer<r<r= Tz)_ordereddict_unflatten..)rrerfr<r<r=_ordereddict_unflattenPsrcCst|\}}||j|gfSr[)radefault_factory)rYr^ dict_contextr<r<r=_defaultdict_flatten[s rcCs|\}}t|t||Sr[)rrg)r^rcrrr<r<r=_defaultdict_unflatten`srcCs|\}}|j|j|d}|S)N)default_factory_moduledefault_factory_namer)r9r:)rcrrZjson_defaultdictr<r<r=_defaultdict_serializehs rcCsrt|tsJt|hdksJ|d}|d}t|ts Jt|ts'Jt|}t||}|d}||gS)N>rrrrrr)ryrdsetrD importlib import_modulegetattr)rxrrmodulerrr<r<r=_defaultdict_deserializers  rdeqcCst||jfSr[)r]maxlen)rr<r<r=_deque_flattenrhrcCs t||dS)Nr)rrfr<r<r=_deque_unflattenrorzbuiltins.tuple)rAz builtins.listz builtins.dictzcollections.namedtuple)rBrCrAzcollections.OrderedDictzcollections.defaultdictzcollections.dequetreecCsVt|}|j}t|dks|dtkrdSt|dd}t|ts"dStdd|DS)Nr2rFrvcss|] }t|tkVqdSr[)r5rD)r|entryr<r<r=rrz*_is_namedtuple_instance..)r5 __bases__lenrprryall)rr@basesrur<r<r=_is_namedtuple_instances  rcCst|rtSt|Sr[)rrr5rr<r<r=_get_node_typesrcCs t|tvSr[)rr>rr<r<r=_is_leafrorc@sVeZdZUeed<eed<eded<dddZdd ede fd d Z de fd d Z dS)rr5rcchildren_specsrINcCstdd|jD|_dS)NcSsg|]}|jqSr<) num_leaves)r|specr<r<r= z*TreeSpec.__post_init__..)sumrrselfr<r<r= __post_init__szTreeSpec.__post_init__rindentcsd|jjd|jd}d}t|jrAd7||jd7}|t|jdkr+dnd7}|dfd d |jddD7}|d }||S) Nz TreeSpec(z, z, [rNrRrr2,cs"g|] }dd|qS)  )__repr__r|childrr<r=rsz%TreeSpec.__repr__..z]))r5r8rcrrrjoin)rrZ repr_prefixZchildren_specs_strZ repr_suffixr<rr=rs    zTreeSpec.__repr__cCs t|tSr[)ryrrr<r<r=is_leafs zTreeSpec.is_leafrINr) r8r9r:rr;rr rintrDrboolrr<r<r<r=rs   cs2eZdZd fdd Zd dedefddZZS) rrINcstddgd|_dS)Nr2)super__init__rr __class__r<r=rs zLeafSpec.__init__rrcCdSN*r<)rrr<r<r=rzLeafSpec.__repr__rr)r8r9r:rrrDr __classcell__r<r<rr=rsrleavescsRt|r |tSt|}t|j}||\}}fdd|D}t|||S)Ncsg|]}t|qSr<_tree_flatten_helperrrr<r=rsz(_tree_flatten_helper..)rappend _LEAF_SPECrr>r6r)rr node_typer6 child_pytreesrcrr<rr=rs    rcCsg}t||}||fS)zkFlattens a pytree into a list of values and a TreeSpec that can be used to reconstruct the pytree. r)rrrr<r<r=rs rtreespeccCst|tstdt|dt|ttfst|}t||jkr2tdt|d|jd|dt|t r;|dSt |jj }d}d}g}|j D]}||j7}| t|||||}qJ|||jS)zqGiven a list of values and a TreeSpec, builds a pytree. This is the inverse operation of `tree_flatten`. zftree_unflatten(leaves, treespec): Expected `treespec` to be instance of TreeSpec but got item of type rWz6tree_unflatten(leaves, treespec): `leaves` has length z, but the spec refers to a pytree that holds z items (z).r)ryr TypeErrorr5r]rprrrLrr>r7rrr rc)rrr7startendr child_specr<r<r=r s6       r cCsLt|r ||dSt|}t|j}||\}}|D]}t||qdSr[)rrrr>r6_tree_leaves_helper)rrrr6r_rr<r<r=r>s    rcCsg}t|||S)z!Get a list of leaves of a pytree.)r)rrr<r<r=r!Ls r!cCs t|dS)zGet the TreeSpec for a pytree.r2)rrr<r<r=r"Ss r"funccs$t|\}}tfdd|D|S)Ncsg|]}|qSr<r<)r|irr<r=rZztree_map..)rr )rr flat_argsrr<rr=r#Xs r#cCst|}tt||dd|S)Nrr)r!rmap)rrrr<r<r=r$]sr$__type_or_typescCdSr[r<rr<r<r=map_onlyqrcCrr[r<rr<r<r=rvrcCrr[r<rr<r<r=r{rcCrr[r<rr<r<r=rrcs.dttgtfdttgtfffdd }|S)a Suppose you are writing a tree_map over tensors, leaving everything else unchanged. Ordinarily you would have to write: def go(t): if isinstance(t, Tensor): return ... else: return t With this function, you only need to write: @map_only(Tensor) def go(t): return ... You can also directly use 'tree_map_only' rrIcsdtdtffdd }|S)NxrIcst|r |S|Sr[ry)r)rrr<r=wrappeds z*map_only..wrapper..wrapped)r.r)rrrrr=wrapperszmap_only..wrapper)rr.r)rrr<rr=rs* cCrr[r<rrrr<r<r=r%r%cCrr[r<rr<r<r=r%rcCrr[r<rr<r<r=r%rcCtt|||Sr[)r#rrr<r<r=r%cCrr[r<rr<r<r=r&rr&cCrr[r<rr<r<r=r&rcCrr[r<rr<r<r=r&rcCrr[)r$rrr<r<r=r&rpredcCt|}tt||Sr[)r!rrrrrr<r<r=r'r'cCrr[)r!anyrrr<r<r=r(rr(cCrr[r<rrrr<r<r=r)rr)cCrr[r<rr<r<r=r)rcCrr[r<rr<r<r=r)rc t|}tfdd|DS)Nc3"|] }t|r|VqdSr[rr|rrrr<r=r z tree_all_only..)r!rrrrrr<rr=r)cCrr[r<rr<r<r=r*rr*cCrr[r<rr<r<r=r*#rcCrr[r<rr<r<r=r*,rcr)Nc3rr[rrrr<r=r;rz tree_any_only..)r!rrr<rr=r*5rc Cst|tsJt|r|g|jSt|trdSt|}||jkr#dSt|j}||\}}t |t |j ks<||j kr>dSg}t ||j D]\}}t ||} | durX|| 7}qFdS|Sr[)ryrrrrrr5r>r6rrrcre_broadcast_to_and_flatten) rrrr6rctxresultrrZflatr<r<r=rFs&       rc@s2eZdZUdZeeed<eed<eded<dS)_TreeSpecSchemaa _TreeSpecSchema is the schema used to serialize the TreeSpec It contains the following fields: - type: A string name of the type. null for the case of a LeafSpec. - context: Any format which is json dumpable - children_spec: A list of children serialized specs. r5rc children_specN) r8r9r:__doc__rrDr;rr r<r<r<r=rds  rc@s2eZdZUeegefed<eegefed<dS) _ProtocolFntreespec_to_jsonjson_to_treespecN)r8r9r:rrrr;r<r<r<r=rss r_SUPPORTED_PROTOCOLSc Cst|tr tddgS|jtvrtd|jdt|j}|j}|tkr.td|jd|jdurMzt |j }Wnt yL}zt d|d}~ww||j }dd|j D}t|||S)Nz Serializing  in pytree is not registered.z%No registered serialization name for zZ found. Please update your _register_pytree_node call with a `serialized_type_name` kwarg.zUnable to serialize context. Please make the context json dump-able, or register a custom serializer using _register_pytree_node.cSsg|]}t|qSr<)_treespec_to_jsonrr<r<r=rrz%_treespec_to_json..)ryrrr5rENotImplementedErrorrAr3rBjsondumpsrcrr)rrXrAZserialized_contexteZ child_schemasr<r<r=r{s4         r json_schemac Cs|ddur|ddurt|ddkrtS|dtvr'td|ddt|d}t|}|jdurQz t|d}WntyP}ztd|d}~ww||d}g}|dD] }| t |q^t |||S)Nr5rcrrzDeserializing rzUnable to deserialize context. Please make the context json load-able, or register a custom serializer using _register_pytree_node.) rrrFrrErCrloadsrr_json_to_treespecr)rr@rXrcexrZ child_stringr<r<r=rs4       rprotocolcCsvt|tstdt|d|durt}|tvr!t||}ntd|dtt t |t |f}|S)Nzhtreespec_dumps(treespec, protocol): Expected `treespec` to be instance of TreeSpec but got item of type rWUnknown protocol . Available protocols: )ryrrr5'DEFAULT_TREESPEC_SERIALIZATION_PROTOCOLrrrLr]r_rr dataclassesasdict)rrZ json_specZstr_specr<r<r=r+s"  r+ serializedcCs@t|\}}|tvrt||Std|dtt)Nrr)rrrrrLr]r_)rrrr<r<r=r,s r,c@seZdZdefddZdS) _DummyLeafrIcCrrr<rr<r<r=rrz_DummyLeaf.__repr__N)r8r9r:rDrr<r<r<r=rsrcCs"tddt|jD|}t|S)NcSsg|]}tqSr<)r)r|rr<r<r=rrz#treespec_pprint..)r rangerrepr)rZ dummy_treer<r<r=r-s r-cCtdt|S)Nz6pytree_to_str is deprecated. Please use treespec_dumps)rTrUr+)rr<r<r= pytree_to_str r rcCr )Nz6str_to_pytree is deprecated. Please use treespec_loads)rTrUr,)rr<r<r= str_to_pytreer rargskwargscOs4g}|D]}t||q|D]}t||q|S)zpGet a flat list of arguments to this function A slightly faster version of tree_leaves((args, kwargs)) )rr^)rrrar<r<r=arg_tree_leavess    r)NNr[)|rrrr threadingrT collectionsrrrrtypingrrrr r r r r rrZGenericOrderedDictrrrrr__all__r.r/r0r1rr3rrrrrrrrDZ ToStrFuncZMaybeFromStrFuncr4LockrKr>r;r?rErFrrVrMrargrirmrnrqrrrsrwrzr{rZ_odict_flattenZ_odict_unflattenrrrrrrrpr]rdrrrr dataclassrrrrrr rr!r"r#r$ZType2ZType3ZTypeAnyZFn2ZFn3ZFnZFnAnyZ MapOnlyFnrr%r&r'r(r)r*rrrrrrrr+r,rr-r rrr<r<r<r=sH   =  : &/""&"&  &   "     "*.$                                   $#