Ph ddlmZmZmZddlZddlZddlmZddl m Z ddl m Z ddl mZddlmZddlmZmZd d gZ dd eeefd e deej,dededeeddfdZddd ddd eeefd e deej,dededeeddfdZ dd eeefd e deej,dededeeddfdZy))AnyDictOptionalN)Stateful) StorageReader) LoadPlanner)DefaultLoadPlanner) _DistWrapper_all_gather_keysload_state_dictloadF state_dictstorage_reader process_groupcoordinator_rankno_distplannerreturncLtjdt||||||S)z3This method is deprecated. Please switch to 'load'.zb'load_state_dict' is deprecated and will be removed in future versions. Please use 'load' instead.)warningswarn_load_state_dict)rrrrrrs yC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/checkpoint/state_dict_loader.pyr r s- MMl J GWY`bi jj)rrrrct|}|t|jk7rtjdi}|D]1}||vr||} t | t r| jn| ||<3t|||||||D]5}||vr||} t | t r| j||| ||<7y)a Load a distributed ``state_dict`` in SPMD style. Each rank will try to read the least amount of data necessary to fullfill the requested `state_dict`. When loading :class:`ShardedTensor` or :class:`DTensor` instances, each rank only reads data for their local shards. For each ``Stateful`` object (having both a ``state_dict`` and a ``load_state_dict``), load will first call ``state_dict`` before attempting deserialization, followed by ``load_state_dict`` once the deserialization is complete. .. warning:: All tensors in ``state_dict`` must be allocated on their destination device *prior to* calling this function. All non-tensor data is loaded using `torch.load()` and modified in place on state_dict. .. warning:: Users must call `load_state_dict` on the root module to ensure load pos-processing and non-tensor data properly propagates. .. note: This function can be used for local inference and load a checkpoint produced by ``save_state_dict`` without having a process group initialized by passing ``no_dist=True`` and by using Tensors instead of ShardedTensors. Args: state_dict (Dict[str, Any]) : The state_dict to load. Note that this state dict will updated in place. storage_reader (StorageReader): StorageReader used to load data from. process_group (ProcessGroup): ProcessGroup to be used for cross-rank synchronization. coordinator_rank (int): Rank to use to coordinate the checkpoint. rank0 is used by default. no_dist (bool): If ``True``, distributed checkpoint will not load in SPMD style. (Default: ``False``) Returns: None. Examples >>> # xdoctest: +SKIP >>> my_model = MyModule() >>> optimizer = Adagrad(my_model.parameters()) >>> model_state_dict = my_model.state_dict() >>> fs_storage_reader = torch.distributed.checkpoint.FileSystemReader("/checkpoint/1") >>> torch.distributed.checkpoint.load_state_dict( >>> state_dict=model_state_dict, >>> storage_reader=fs_storage_reader, >>> ) >>> # module.load_state_dict() function might have customized steps >>> # to flush the state_dict, must call it to >>> # ensure correct behavior. >>> my_model.load_state_dict(model_state_dict) .. note:: load_state_dict uses collectives to coordinate reads across ranks. For NCCL-based process groups, internal tensor representations of objects must be moved to the GPU device before communication takes place. In this case, the device used is given by ``torch.cuda.current_device()`` and it is the user's responsibility to ensure that this is set so that each rank has an individual GPU, via ``torch.cuda.set_device()``. z|Detected mismatched keys in state dict after all gather! This behavior is unsupported and may cause errors may cause errors.N) r sortedkeysrr isinstancerrrr ) rrrrrrr statetful_sdkeyelems rrr!sZ J 'D vjoo'((  S L j #1;D(1KDOO-QU S  \>=BRT[]de j # dH %  c!2 3 3 rc tjjdt|| |  t  fd}fd} j d|| fd} j d|} y)Nz,torch.distributed.checkpoint.load_state_dictcJj}j|jj|jj }j |}|SN) read_metadataset_up_planneris_coordinatorset_up_storage_readercreate_local_planprepare_local_plan)metadata local_plandistWrrrs r local_stepz$_load_state_dict..local_stepso"""!//1z8U5I5IJ,,Xu7K7KL..0 #66zB rcTJj|}j|}|Sr%)create_global_planprepare_global_plan)all_local_plansrrs r global_stepz%_load_state_dict..global_steps5"""!44_E(<<_MrplanctJj}j|}|jyr%) finish_plan read_datawait)final_local_plan all_reads central_planrrs rr8z#_load_state_dict..read_datas@""""..|<",,-=wG rread)torch_C_log_api_usage_oncer r reduce_scatter all_gather) rrrrrrr/r4r8_r<r.s `` ` @@rrrso HH  !OP G 5E FE$& '' KHL +Ar)NrFN)typingrrrrr>torch.distributed distributeddist%torch.distributed.checkpoint.statefulrstoragerrr default_plannerr utilsr r __all__str ProcessGroupintboolr rrrrrRs&& :!/1 f % 26%) kS#X k! kD--. k k  k k " k  k&26%)bS#Xb!bD--. b  b  bk "b bN26%) ),S#X),!),D--.), ),  ), k " ), ),r