o H&i@sUddlZddlZddlZddlmZmZmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZddlm Z m!Z!ddl"m#Z#m$Z$m%Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-m.Z.dd l/m0Z0dd l1m2Z3d Z4d Z5e5dZ6dZ7e7dZ8dZ9ee:Z;e<Z=ee e>d<eeeej?e@eAe:fZBeeBeeBeeBe e:dffZCe e:eCfZDeeDZEe e:eeDeEffZFejGddZHeGdddZIeGdddeIZJdSdejKde:deLde;fddZMddd dejKd!eejNjOd"fd#eLd$eeejKd%eeIdeJf d&d'ZPd(e e:eCfd)eFd*eJddfd+d,ZQd-eejKejNjOfd.e:de fd/d0ZRdejKd*eJde e:eCffd1d2ZSdejKd3e e:eCfd*eJde0fd4d5ZTd6ejNjOddfd7d8ZUdejKd9eejNjOd"fd*eJdeFfd:d;ZVdejKd6ejNjOd)eFd*eJdeFf dd?ZXddd dejKd$eeejKd%eeIde e:eCffd@dAZYddd dejKd9eejNjOeejNjOfd$eeejKd%eeIdeFf dBdCZZddd dejKd9eejNjOeejNjOfd$eeejKd%eeIdee e:eCfeFff dDdEZ[dejKd3ee ejKe e:eCffe e:eCffde e:eCffdFdGZ\ddHdejKd(ee ejKe e:eCffe e:eCffd%eeIde0fdIdJZ]ddHdejKd9eejNjOeejNjOfd)eFd%eeIddf dKdLZ^ddHdejKd9eejNjOeejNjOfd(ee ejKe e:eCffe e:eCffd)eFd%eeIde0f dMdNZ_eddHdejKd%eeIddfdOdPZ`eddHdejKd9eejNjOd"fd%eeIddfdQdRZadS)TN)asdict dataclassfield)chain) AnyCallablecastDictIterableList no_type_checkOptionalSetTupleUnion) ShardedTensor)DTensor)_gather_state_dict_offload_state_dict_to_cpu)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)_IncompatibleKeys)DistributedDataParallelZ _flat_param param_groups.stateparams_patched_state_dict ValueTypeccsPt}tzdVWt|rtdSdSt|r'twwN)gc isenableddisableZcollectenable) is_enabledr-RC:\wamp64\www\opt\env\Lib\site-packages\torch/distributed/checkpoint/state_dict.py gc_contextCs  r/c@sNeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<d S) StateDictOptionsa This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes``: when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). The default value is False. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictN) __name__ __module__ __qualname____doc__r1bool__annotations__r2r3r4r5r-r-r-r.r0Ps     r0c@seZdZUeedZeeee j fee e j ffe d<ee dZeee d<ee dZeee d<dZee d<dZee d<ejZee d<eedZeeje d <d S) _StateDictInfo)default_factoryfqn_param_mappingall_fqnssubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)r6r7r8rdictr>r rstrtorchTensorFQNS_Tr;setr?rr@rAr:rB contextlib nullcontextrCrlistrDr nnModuler-r-r-r.r<ws   r<Tmodelnameskip_ddp_prefixreturnc sd|vr|hS|d}g}|}t|D]b\}}t|tr.|dks#J|j}|s-||qt|trl||dtkrXd|t |t}rLdfdd|j DSt |t }|t krk||t ||}q||t ||}qd|hS)a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `Set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. r"modulecsh|]}|qSr-r-).0fqnprefixr-r. sz_get_fqns..) split enumerate isinstanceDDPrTappendFSDP FLAT_PARAMjoingetattrZ_fqnsr) rPrQrRZ obj_namesZ fqn_obj_namesZcurr_objiZ curr_obj_nameZ flat_paramr-rXr. _get_fqnss6              re) submodulesoptionsoptims. optim_onlyrfrgc Csx|r|std|p t}i}t}|D]\}}t||} | ||<| D] } ||| <|| q%qt} |ret|}|D]%\}} | |vrHq?t||} t| dksWJd| D] } | | dqYq?t |} | r|j rt |j |j d}t |j |j d}tj}n t}t|j d}tj}tjtj||||d}ntj}td it|||| |tttj| | t|dkd S) zW Verify the model and options passed by the user and generates _StateDictInfo. z;Optimizers are not passed in but optim_only is set to True.rUz)Submodule FQN should only have 1 instancer")offload_to_cpuZ rank0_only)rj)rTstate_dict_typestate_dict_configoptim_state_dict_configr)r>r?r@rCrDrArBNr-) RuntimeErrorr0rJnamed_parametersreadd named_moduleslenr`rDr1rr2rrZFULL_STATE_DICTrrZSHARDED_STATE_DICT functoolspartialrkrKrLr<rrr rNrO)rPrhrirfrgr>r?rQparamfqnsrWr@rTrDrlrmrkrCr-r-r._verify_optionssx        rwmodel_state_dictoptim_state_dictinfocCsd}|jD]}t|}|dusJd|jrd}nq|jr$|s$td|jrB|sB|jsB|jsB|jr5|jsB|j rBtdt d|j rX|rK|t sX|jrQ|jsXtd||D]}t|vrlt|dtd q\dS) NFz)Expected a fsdp_state with a fsdp module.Tz:The model has FSDP modules but no FSDP root module exists.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=r"zgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rDrZ_is_rootrnrAr@r3r2r1r5distZget_rankrBSTATEkeysra)rxryrzZ has_fsdp_rootrTZ fsdp_statekeyr-r-r._verify_state_dicts^    robjapicCs,t||}|tvrtjt|j||d}|S)N)self)rcr%rsrt __class__)rrcallr-r-r._state_dict_fn5s rc Cs|jsiS|t|d}Wdn1swYt|D]6}t||}t|dks4Jtt|}||kr[dt fdd}|||sTt d|d|| |||<q%|j ri}|D]&}|j D] }| |srqj|jr|||||<qj|t|d} |||| <qjqe|}|jr|D]\}} | jrqt||}|D]}| |qqt|D] \}} | jr| |q|jr|jstnd} t||j| d S|jrt|S|S) N state_dictrUrScSst|t|kr dS|d}|d}d}t|D]&\}}|||kr9|d7}|t|kr8|t|dkSq|dkr>qdSdS)NFr"rrUrTT)rrr[r\)r~rWZ fqn_splitZ key_splitZfqn_idxZkey_idxZkey_namer-r-r.verifyLs    z%_get_model_state_dict..verifyzAn unexpected key, z, exists. FQN is rr2 ranks_only)rArCrrMr}rerrnextiterr:rnpopr@ startswithr4r3ro requires_graditemsis_metar1r2tuplerr) rPrzrr~rvrWrnew_state_dictrYZnew_fqnruprr-r-r._get_model_state_dict<s^          rrc Cs|jr|s tiiS|D]%\}}t||}t||dd}t||D]\}}||kr2||||<q#q|ttt|d||j dWdS1sPwYdS)NF)rRload_state_dict)rr5) rArroreziprrCrrr5) rPrrzr~_rvZfqns_with_ddp_prefixrWZfqn_with_ddp_prefixr-r-r._load_model_state_dicts$    $roptimcCsd|jrdS|jD]}|tD]}|jdurtd|jr"t||_qq|jdd|j dddS)zH Initialize optim states by calling the step() with zero grads. Na state_dict can only be used if the optimizer states are initialized (usually after one step() with gradients) or gradients are None. For the later case, state_dict will fake the gradients as zero to initialize the optimizer states. However, the gradients are not None.)closureT)Z set_to_none) r#r!PARAMSZgradrnrrGZ zeros_likestepZ zero_grad)r param_grouprur-r-r._init_optim_states     r optimizersc s|jsiStitgi}|D]}t|t|d}|jr8|t|||}Wdn1s2wYnut t dd|j D}t t|tt|}i|D](\}} t||} t| dksgJtt| } | |vrrqV|| } | | <| | <qVt |tD]}|} |t||t| <q|tD]} fdd| tD| t<q|sq tt|t|ttt|t|tq |jr|jstnd}t ||j|dS|jrt!|S|S) Nrcss|]}|tVqdSr')r)rVgr-r-r. sz(_get_optim_state_dict..rUcsg|]}|qSr-r-)rVpidZfqn_pid_mappingr-r. sz)_get_optim_state_dict..rr)"rBr|PGrrrDrCr`ryrMr from_iterabler!rErrangerrrorerrr}rrr DictValueTypeupdateListDictValueTypeextendr1r2rrr)rPrrzryrZosdr$Zparam_pid_mappingr~rurvrWrgrouprr-rr._get_optim_state_dictsP       rcCs4i}g}t|t|i}i}|jD]]}|tgi|tD]O} |j| D]G} |dt} t| ts2J| | | jrEt t |t| || <t t |tD]} | t} t| tsYJ| | vrit |td|t | <qLq#qqt t |tD]#}|t |d} | dkrqt|D]\}}|tkrq||| |<qqt|S)a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. rU)r|rr!r_rr>r]rMrrrrrridgetr)rPrryrzr#Zpg_stateZ return_osdZ pg_mappingrrurWr$Zloaded_param_groupidxr~valuer-r-r._split_optim_state_dicts@      rc Csz|jsdS|D]3}t||||}|jr.|t|||}Wdn1s)wYt|t|d|dqdS)Nrr)rBrrDrCr`Zoptim_state_dict_to_loadrr)rPrrrzrryr-r-r._load_optim_state_dict"s rcCsXtt|td||d}t||}t|i||WdS1s%wYdS)a Return the model state_dict of ``model``. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. submodules: Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``model``. FrirfrgN)r/rwrrr)rPrfrgrzrxr-r-r.get_model_state_dict:s  $rcCstt-t|tjjr|fnt|}t||d||d}t|||}ti|||WdS1s3wYdS)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules: Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. TrN) r/r]rGr Optimizerrrwrr)rPrrfrgrzryr-r-r.get_optimizer_state_dict]s    $rcCst4t|tjjr|fnt|}t||d||d}t||}t|||}t |||||fWdS1s:wYdS)a) Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: import torch from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.nn.parallel import DistributedDataParallel as DDP from torch.distributed.checkpoint.state_dict import get_state_dict fsdp_model = FSDP(copy.deepcopy(model)) fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) ddp_model = DDP(copy.deepcopy(model)) ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) fsdp_state_dict, fsdp_optim_state_dict = get_state_dict(fsdp_model, fsdp_optim) # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), # the asserts will fail. assert ddp_state_dict == fsdp_state_dict assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules: Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. FrN) r/r]rGrrrrwrrr)rPrrfrgrzrxryr-r-r.get_state_dicts"D    $rc s|siSttt|tjr_tttjttt ff|}i}| D]8\}}| D]/\}}||kr5q,t ||}t |dksDJdtt|d|fdd| Dq,q$|Stttt f|S)NrUz/FQNs for a submodule should only have 1 elementr"csi|] \}}||qSr-r-)rVZsubfqnrrXr-r. sz/_unflatten_model_state_dict..)r]rrr}rNrOrr rFr&rrqrerrr) rPrZcast_state_dictr submoduleZsub_state_dictrQmrvr-rXr._unflatten_model_state_dicts$  r)rgcCs^t||}tt|td|d}t|i|t|||WdS1s(wYdS)a2Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys FrirgN)rr/rwrrr)rPrxrgrzr-r-r.set_model_state_dicts  $rcCsrt,t|tjjr|fnt|}t||d|d}ti||t||||WddS1s2wYdS)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None TrN) r/r]rGrrrrwrr)rPrryrgrzr-r-r.set_optimizer_state_dict s  "rcCst||}t2t|tjjr|fnt|}t||| |d}t|||t ||||t |||WdS1s=wYdS)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. rN) rr/r]rGrrrrwrrr)rPrrxryrgrzr-r-r.set_state_dictEs)    $rcsjtjt||dfdd}||_tjt||ddtttfffdd }||_t |t |dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rPrgcSr'r-r-_state_dict_callr-r.state_dict_callz0_patch_model_state_dict..state_dict_callrc|ddS)N)rxr-r_load_state_dict_callr-r.load_state_dict_callz5_patch_model_state_dict..load_state_dict_callN) rsrtrrrr rFrrr%rp)rPrgrrr-rrr._patch_model_state_dicts   rcstjt|||dfdd}tjt|||ddtttfffdd }t|t|t |t j j r9|fnt |}|D]}||_||_q?dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rPrrgcrr'r-r-rr-r.rrz4_patch_optimizer_state_dict..state_dict_callrcr)N)ryr-rrr-r.rrz9_patch_optimizer_state_dict..load_state_dict_callN)rsrtrrr rFrr%rpr]rGrrrrr)rPrrgrrrr-rr._patch_optimizer_state_dicts0    r)T)brKrsr( dataclassesrrr itertoolsrtypingrrrr r r r r rrrrGZtorch.distributed distributedr{Ztorch.nnrNZ'torch.distributed._shard.sharded_tensorrZtorch.distributed._tensorrZ.torch.distributed.checkpoint._state_dict_utilsrrZtorch.distributed.fsdprrrr`rrrrrZ$torch.distributed.fsdp._common_utilsrrZtorch.nn.modules.modulerZtorch.nn.parallelr r^rarZ PG_PREFIXr|Z STATE_PREFIXrrFrIrJr%r;rHintfloatZ PrimitiveTyper&rrZOptimizerStateTypecontextmanagerr/r0r<rOr:rerrrwrrrrrrrrrrrrrrrrrr-r-r-r.s  4    (       & 2  S  $1  J   5 6    '  /  W$     , -  = 6