PhiFddlZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZmZmZddlZddlmZddlmcmcmZddlmcmcmZddlmcmcmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7dd l8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@dd lAmBZBdd lCmDZDdd lEmFZFdd lGmHZHdZI ddlJmKZKmLZLeNdZOdZPeejejfZReeejeRfZSe>je5je>je5je>je5je>je5je>je5jiZYe>je>jgZZe>je>jfZ[e d\de+deSde>deeBdee&de+f dZ\e de+deSde&de+fdZ]e dede^fdZ_e de&de^fdZ`e deNdejfdZae dejdeNdejfd ZbdejdeNdeejejffd!Zce d\de+d"e!jd#ee ejBjd$eee ejBjee ejBjfde+f d%Zfd$e ed&e^ddfd'Zge de+d"e!jd(ee!jd)eeeNejfde+f d*Zie de+d"e!jde+fd+Zje de+dee>d,ee=d-ee:d.e^d/e^d0eNd1eNde+fd2Zke de+de+fd3Zle de+d4e9d5e^de+fd6Zme d\de+de&de+fd7Zne de+de+fd8Zoe de+d9e!jd)eeeNejfd:eee!jgdfd;e^de+f d<Zpe de+d=e e!jd9e!jfd>Zqd?e!jd@ee ejBjdee!jfdAZr d\d?ejBjd#eejBjdBee ejBjdeejBjfdCZsd?ejBjd#eejBjdeetfdDZud?e!jdeetfdEZvd"e!jd(ee!jd)eeeNejfddfdFZwd)eeeNejfdGeNdeejfdHZxd"e!jd(ee!jd#ee!jdee^e^ffdIZyd?e!jd:ee!jgdfd#ee!jddfdJZzd?e!jdKeejd#ee!jfdLZ{d?e!jd#ee!jde e!jfdMZ|d"e!jd(ee!jdNeejdKeejddf dOZ~d=e e!jdPe ejdKeejddfdQZdRZd"e!jd(ee!jdKeejdGeNdejf dSZd"e!jd=e e!jdejddfdTZd=e e!jdPe ejdejddfdUZdVe ejddfdWZd"e!jd(ee!jde e!jfdXZd(ee!jddfdYZde>fdZZdejde#jfd[Zy#eM$rdZIYwxYw)]N) AnyCallableDequeDict GeneratorIterableIteratorList no_type_checkOptionalSetTupleUnion) default_hooks)_mesh_resources DeviceMesh)_get_default_group)_FSDPDeviceHandle _FSDPState_get_module_fsdp_state_is_fsdp_flattened!_named_parameters_with_duplicatesclean_tensor_name TrainingState)_FSDP_USE_FULL_PREC_IN_EVAL FlatParameterFlatParamHandleHandleShardingStrategy)_FreeEventQueue)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigMixedPrecisionShardingStrategyStateDictConfig StateDictType)_Policy)DTensorExtensions)_sync_params_and_buffers)RemovableHandleT) deferred_initfakeFi _fsdp_syncedstate process_groupsharding_strategypolicy device_meshreturnc(| | td|tv}|r#|||td|dt|||}n4|r||_|j d|_n||n t |_|j j|_|j j|_ |j}|r||jjz}tjj||_||jz |_|S)NzcCannot pass both process_group and device_mesh at the same time. Please just pass only one of them.zManual wrapping with z@requires explicit specification of process group or device_mesh.rmesh_dim) ValueErrorHYBRID_SHARDING_STRATEGIES*_init_process_group_state_for_hybrid_shard _device_mesh get_groupr0rranksize world_size_inter_node_pgr DefaultState_get_gradient_predivide_factor_gradient_predivide_factor_gradient_postdivide_factor)r/r0r1r2r3is_hybrid_strategydata_parallel_world_sizes mC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/fsdp/_init_utils.py_init_process_group_staterHasC [%< <  +.HH  V^ 8K'(9':;R  ?}kE !,E "-"7"7"7"CE "/!: @R@T  $$))+EJ**//1E$// E$8$8$=$=$?? ""AA $  $ !5#C#CC % Lc|rft|r6||_|jd|_|jd|_nt dt |j|@t}t||jj\}}||_||_n2t|r|\|_|_nt dt|t|j|_|S)Nrr6z,Expected device_mesh to have ndim=2 but got zmExpected process_group to be passed in as either None or Tuple[dist.ProcessGroup, dist.ProcessGroup] but got r0)"_is_valid_hybrid_shard_device_meshr;r<r@r0r8lenr!_init_intra_and_inter_node_groups_device_handle device_count_is_valid_hybrid_shard_pg_typetype_get_default_comm_hook_state_inter_node_state)r/r0r3 default_groupintra_node_groupinter_node_groups rGr:r:s  -k :!,E $/#8#8!#8#DE "-"7"7"7"CE {44678:   *, -N 5//<<>. **// *- 89F 5E !5GGKMGZF[]  ;**E LrIcjt|txr"t|dk(xrtd|DS)Nc3PK|]}t|tj ywN) isinstancedist ProcessGroup).0pgs rG z1_is_valid_hybrid_shard_pg_type..sJMb 2t001M$&)r]tuplerNallrLs rGrRrRs: =%( K  ! # K JMJ JrIctj|}|td|d|ddt|txr|j dk(S)NzFound device_mesh z$ passed in has a parent device_mesh .z*Hybrid sharding + TP is not supported yet.rZ)rget_parent_mesh RuntimeErrorr]rndim)r3 parent_meshs rGrMrMs]!11+>K  -QR]Q^^_ ` 8   k: . H;3C3Cq3HHrInum_devices_per_nodec6tj|\}}|S)a5 Return a process group across the current node. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return an intra-node subgroup across [0, 7] or [8, 15] depending on the process's rank. For example, rank 3 would get [0, 7]. )r^ new_subgroups)rlintra_node_subgroup_s rG_init_intra_node_process_grouprqs!"//0DE rIglobal_process_groupcTd}tj|}tj|}||z}tj||z}t |D]?}t |Dcgc] }|||zz } }tj | |} ||k(s>| }A| J|d|Scc}w)a Return an inter-node process group where each contained rank has the same local rank. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return inter-node process group {0, 8}, {1, 9}, {2, 10}, and so forth depending on the process's rank. For example, rank 1 would get {1, 9}, rank 5 would get {5, 13}. N)ranksbackendz. expected to assign inter-node pg, but did not)r^ get_backendget_world_sizeget_rankrange new_group) rrrl inter_node_pgsharding_backendr? num_nodes my_local_rank local_rankiranks_for_inter_groupgrps rG_init_inter_node_process_grouprs M''(<=$$%9:J22IMM"67:NNM01 =B9=M! =MJ!22 3=M ! nn#8BRS  &M2 !H FGH! ! s%B%c0t|t||fS)a Initialize intra and inter-node process groups and return the ones corresponding to this process's rank. This function can be used to initialize process groups for ``HYBRID_SHARD`` or ``_HYBRID_SHARD_ZERO2`` in FSDP. This function assumes each node has an equal number of CUDA-enabled devices. Returns: Tuple[dist.ProcessGroup, dist.ProcessGroup]: Intra and inter-node process group. )rqr)rrrls rGrOrO s# '';<&';=QR rImoduleignored_modulesignored_statesc| | tdd}|du}|rt|}t|dng}t| t|ngdt|dkDr"t |dt j r|}n|}t|||_t||j||_ t||j|_ |S)NzfCannot pass both ignored_modules and ignored_states at the same time. Please just pass ignored_states.TFr) r8list_check_ignored_statesrNr]nn Parameter_get_ignored_modules_ignored_modules_get_ignored_params_ignored_params_get_ignored_buffer_names_ignored_buffer_names)r/rrrignored_parameterspassed_as_ignored_statesignored_states_lists rG_init_ignored_module_statesrs"~'A :  -T9">2148 %4%@D !b%  !# )!,bll ;!4 1O1&/JE/ E #< #E LrIrct|dk(ry|r`td|D}td|D}|s9|s6t|Dchc] }t|c}t}t d|yytd|Ds6t|Dchc] }t|c}t}t d|ycc}wcc}w) z Check that the ignored states are uniformly parameters or uniformly modules. We may remove this check in the future if we permit mixing. rNc3PK|]}t|tj ywr\)r]rrr`r/s rGrbz(_check_ignored_states..XsUnUE2<<8nrcc3PK|]}t|tj ywr\r]rModulers rGrbz(_check_ignored_states..YsSN5*UBII6Nrc)keyzUignored_states expects all nn.Parameter or all nn.Module list elements but got types c3PK|]}t|tj ywr\rrs rGrbz(_check_ignored_states..bsL^E:eRYY/^rcz>ignored_modules expects nn.Module list elements but got types )rNresortedrSreprr8)rr all_params all_modulesr/ sorted_typess rGrrMs >aUnUU SNSS +!N"KN54;N"KQUVL**69 #.zL^LL!N"KN54;N"KQUVL%( M #L#Ls B; Cignored_params device_idc*d}|1t|tjr|ntj|}|t||D]|}|jjdvr| |j}+|jj|jk7sOt d|jd|jj|xs2tjdtj j}tj||_ |S)a Determine device handle used for initializing FSDP. If a device is specified by ``device_id``, then returns device handle corresponds to that device type. Otherwise, If the module is already on a non-CPU device, then the device type is that non-CPU device type. If the module is on CPU or meta, then the device type is the current cuda device. This method will be called once ignored paramters was determined, as the device handle maybe needed for other initialization. N>cpumetazLFSDP does not support modules with different device types but got params on z and cuda) r]torchdevice_get_orig_paramsrSrircurrent_devicer from_devicerP)r/rrrdetermined_deviceparams rG_init_device_handlerjs$)U\\2 i(   %fn=E||  O3 ($)LL!<<$$(9(>(>>&-->-C-C,DE%,,J[J[I\^>.  EJJ--/2 -889JKE LrIct||_i}|jD]\}}t|}|j||<!||_|Sr\)_get_buffer_names _buffer_names named_buffersrdtype_buffer_name_to_orig_dtype)r/rr buffer_namebuffers rG_init_buffer_statersZ ,F3E :<%335 V' 4 28,,";/ 6(BE$ LrImixed_precision cpu_offloadlimit_all_gathersuse_orig_paramsbackward_prefetch_limitforward_prefetch_limitc:|jdk(rN|tjk7r+tjd|xstj dtj}|xstj |_|xs t|_|5tjjdt|jtjjt ddk(|_|xs t%|_||_||_t,j.|_d|_t5|_t9j:|_t?j@|j<|||_!tE}||_#d} | |_$g} | |_%|S)NrKz/FSDP is switching to use `NO_SHARD` instead of z since the world size is 1.z'torch.distributed.fsdp.mixed_precision.1)&r?r%NO_SHARDwarningswarn FULL_SHARDr1r$rr_C_log_api_usage_oncestrosenvirongetr_use_full_prec_in_evalr!rr_use_orig_paramsrIDLEtraining_state_is_rootr_free_event_queuer^get_debug_level _debug_levelexec_order_utils_ExecOrderData_exec_order_datadict_fully_sharded_module_to_handle_handleparams) r/r1rrrrrrrrrs rG_init_core_statersv 1  0 9 9 9 MMA$C(8(C(CDE''  -55/N3C3N3NE+?~/?E" $$5c%:O:O6P5Q R  2B73>  $3z|E/E,E(--EEN-/E--/E-<< EIM#,KE) $GEM"$FEL LrIcfg}||_g}||_g}||_d|_d|_d|_|S)NT)_root_pre_forward_handles_pre_forward_handles_post_forward_handles_sync_gradients _comm_hook_comm_hook_state)r/rrrs rG_init_runtime_statersK8:&?E#24!5E35"7E EE!E LrIbackward_prefetchforward_prefetchc"||_||_|Sr\)rr)r/rrs rG_init_prefetching_staters 0E-E LrIcx|r0tj|jt|_|Sd|_|Sr\)rrhr;r)_fsdp_extension)r/r3s rG_init_extensionrs>66u7I7IJV 1 3 L!% LrIctj|_t}t |_||_i}||_|Sr\)r'FULL_STATE_DICT_state_dict_typer#r"_optim_state_dict_config_state_dict_config_unshard_params_ctx)r/state_dict_configunshard_params_ctxs rG_init_state_dict_stater s?*::E)<)>%=%?E"0E57 2E LrIfully_sharded_module param_init_fnsync_module_statesct|j|t|j}t |jj \}}|s|r|t ||j n6|rt||j n|rtj|fdj D chc]}|jD]} | } }} t|j| |t|j|j_ tt|j} |r@t!|| j"j$t&vrt!|| j(t+| |Scc} }w)zHInitialize a ``FlatParamHandle`` from a module ``fully_sharded_module``.c>t|duxr|jvSr\)rr) submoduler/s rGz0_init_param_handle_from_module..2s('=i'HD'P(8!7!77(8rI)check_fn)_check_single_device_moduler_get_device_from_device_idr=_need_to_materialize_moduler_materialize_with_param_init_fn_materialize_meta_moduler,materialize_modulebuffers_move_module_to_device_get_compute_devicecompute_devicerr_sync_module_params_and_buffersr0r1r9r@_init_param_handle_from_params) r/rrrrdevice_from_device_idis_meta_moduleis_torchdistX_deferred_initignored_modulerignored_buffersmanaged_paramss ` rG_init_param_handle_from_modulers 4e6K6KYW6y%**M2Me33U5K5K3/N/ 5=;T' -1G1G   )U-C-C  %(( 8 $444N$,,.F . 4    /  E*+?AVAVWXN' .%2E2E   " "&@ @ +$ne6J6J #5.:NO L;s6Frc t|dk(ryt|||jt|j|j j |jj|jj|jj|j|j|j }|j|jrJ|j j#|j$||_||j&|j(<t+j,d}|j j r,|j$j,|k7r|j/|yyy)Nr)fsdp_extensionr)rNrrSHARDING_STRATEGY_MAPr1roffload_paramsr param_dtype reduce_dtypekeep_low_precision_gradsr0rrshardrrappend flat_paramr_fully_sharded_modulerr flat_param_to)r/rrhandle cpu_devices rGr r Vs%  6{a  e556 (( )) ** 66  ,, F LLN}}  LL))*EMJPE))&*F*FGe$J ''F,=,=,D,D ,RZ(-S'rI root_modulerc d} | t|n t}|D]V}t|tj j st|dt|zt|sMtd|jD])}tj|r|j|+|Dchc]3}|jD]}t|tjs| 5}}}||vrtj d|jD]9}t|} | t#| dsJ|j%| j&;|S#t$r }t|dt|z|d}~wwxYwcc}}w)ah Check that ``_ignored_modules`` is an iterable of ``nn.Module`` s without any FSDP instances. Return the modules contained in their module subtrees as a :class:`set`. Nested FSDP instances are excluded, but their already-computed ignored modules are included. ``_ignored_modules`` represents the argument passed by the user to FSDP. z>`ignored_modules` should be an iterable of `torch.nn.Module`s Nzbut got zbut got an iterable with z1`ignored_modules` should not include FSDP moduleszTrying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: r)set TypeErrorrSr]rrrrr8modulestraversal_utils _composableadd fsdp_fileFullyShardedDataParallelrrhasattrupdater) rr msg_prefixignored_root_moduleserchildrroptional_fsdp_states rGrrusRJQ%5%AC !su  '&%((//2J+DT&\N)SST T !& )PQ Q '%%'**62 $ $V ,(+*F^^%E%!C!CD % * o%  228 ; !((* 4Y?  *.0BC CC  " "#6#G#G H + I Q x5E0F/G%HHIqPQ$sE78F F%FFrct}|Dchc]%}|jD]}t|r|'}}}|j||,|Dchc]}t|r|}}|j||j D]9}t |} | t | dsJ|j| j;|Scc}}wcc}w)z Return the parameters of the modules in ``ignored_modules`` and the parameters in ``ignored_parameters``. :class:`FlatParameter` s are excluded from the result. r)r! parametersrr*r#rr)r) rrrall_ignored_paramsmpparams_in_ignored_modulesparams_in_ignored_parametersrr/s rGrrs36%#!"aALLNqBTUVBWN?!78%)( )!1CA1FA) %(  !!">?!((* 4Y?  *.0AB BB  % %&9&I&I J + '!( s#C C C%Cc t}|Dchc]}|jD]}|}}}|j|jDchc]\}}||vr t |c}}|j D]9}t |}|t|dsJ|j|j;|Scc}}wcc}}w)z6Return the cleaned buffer FQNs in ``ignored_modules``.r) r!rr*rrr#rr)r) rrall_ignored_buffer_namesr3rbuffers_in_ignored_modulesrrr/s rGrrs *-("'1aiikFk"##(3'@'@'B 'B# V33 k *'B !((* 4Y?  *.0GH HH $ + +,?,U,U V + $#'"  s B>C cf|jDchc]\}}t|c}}Scc}}w)zrReturn the fully prefixed names of all buffers in the module hierarchy rooted at ``root_module`` as a class:`set`.)rr)rrrps rGrrs:>I=V=V=X =X>;+&=X  s-ct||Dchc]}|j}}t|dk(r%tjd|vr| t dyt|dkDrt d|ycc}w)z Raise an error if ``module`` has original parameters on multiple devices, ignoring the parameters in ``ignored_params``. Thus, after this method, the module must be either fully on the CPU or fully on a non-CPU device. rZrNzTTo support a module with both CPU and GPU params, please pass in device_id argument.rKz;FSDP only supports single device modules but got params on )rrrNrri)rrrrdevicess rGrrs*:&.)QR)Qu||)QGR 7|qU\\%0G;  5   W I' S   SsA4r=c x|yt|tjr|ntj|}|tjdk(rmtjd|d|dtj j dtjdtj j }|S)z Return a ``torch.device`` for the specified ``device_id``. Processes ``device_id`` and returns either the corresponding device or ``None`` if ``device_id`` is ``None``. Nrz"FSDP got the argument `device_id` z on rank zJ, which does not have an explicit index. FSDP will use the current device z. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.)r]rrrrrr)rr=rs rGrrs 5<<8 ell9>U f%% 0 9f005 0I0I0K/LM1 1 fejj&?&?&AB MrIctt||}td|D}|jD]-}||vr|j dD]}||j z}/| xrt xrtd|D}||fS)z Return if ``module`` has parameters on meta device and if ``module`` is using torchdistX deferred initialization. At most of the returned bools can be ``True``. If either is ``True``, then ``module`` needs to be materialized. c34K|]}|jywr\)is_metar`rs rGrbz._need_to_materialize_module..;sCN5NsFrecursec3FK|]}tj|ywr\)r-is_fakerAs rGrbz._need_to_materialize_module..Gs@ U#s!)rranyr#rr@_TORCHDISTX_AVAIL)rrrrr rbufr s rGrr.s*6>BCNCNCCN^^%  ' $$U$3C ckk )N4&  A  A @@ @ 6 66rIct|std|dt|t||}|D] }|| y)Nz Expected z to be callable but got )callabler8rS_get_modules_to_materialize)rrrmodules_to_materializers rGrrLsQ M " &>tM?R>S T  9oV(f)rIr c R|xs1tjtjj}t ||} tj 5|D]u}t j|jd|jd}tt|dkD}|sS|j|d|jw dddy#1swYyxYw#t$r5}tj dt#|dt%d|d}~wwxYw)NFrBr)rrCzIUnable to call `reset_parameters()` for module on meta device with error z(. Please ensure that your module oftype z* implements a `reset_parameters()` method.)rrrrrKno_grad itertoolschainr1rrNrto_emptyreset_parameters BaseExceptionrrrrS) rr rmaterialization_devicerLrmodule_state_iterhas_module_statesr-s rGrrZs  3ell !!#79oV]]_0%.OO%%e%4fnnUn6S%!%(->(?$@1$D!$OO+A5OQ++-1__   !!$Q)L>!K M  s=C(AC-%CC(C%!C(%C(( D&10D!!D&c"g}tj|g}|h}|rq|j}|j||j D]:}||vst |||vs|j ||j|<|rq|Sr\) collectionsdequepopleftrchildrenrr&)rrrLqueuevisited_modulesr child_modules rGrKrK{s /1   {m ,E'2mO %%f-"OO-LO3*<8@ 7##L1 \*.  "!rIrc tjd | tj}|j |g}g}|r|j }|j fd|jdD|j fd|jdD|jD].}t|tjr|j |0|r|D cgc] } | |vs|  } } |D cgc] } | |vs|  } } t| | |ytt||d} | | j k(r t!yyycc} wcc} w)a Move ``module`` depending on ``device_from_device_id`` and its current device. This includes moving ignored modules' parameters. - If ``device_from_device_id`` is not ``None``, then this moves ``module`` to the device. - If ``device_from_device_id`` is ``None``, then this does not move ``module`` but warns the user if it is on CPU. Precondition: ``_check_single_device_module()``. rNc3@K|]}|jk(r|ywr\r)r`rrs rGrbz)_move_module_to_device..s%BE<<:-BFrBc3@K|]}|jk(r|ywr\ra)r`rrs rGrbz)_move_module_to_device..s%@F==J.@rb)rrrXrYrrZextendr1rr[r]r'r(_move_states_to_devicenextr_warn_cpu_init)rrrr r\rr curr_modulerr4params_to_move bufs_to_moverrs @rGrrsY$e$J(#."3"3"5 V%'&(--/K MM(33E3B  NN)11%1@  )113 !)Y-O-OPLL+4!&&,GVq/F!VG#*G7aa.F7 G~|=RS !&.94 @E U\\Z78 HGs1 E;E EErc6t|dk(rt|dk(ryt|dkDr|dj}nt|dkDr|dj}tjd}||D]k}tj5|j ||_|j *|j j ||j _dddm|D]}|j ||_y|k(r tyy#1swYxYw)z Move states to the specified device. Precondition: ``_check_single_device_module()`` and module's parameters and buffers have been materialized if needed. rNr)rNrrrNtodatagradrg)rrr rrrrs rGreres 6{aCLA- 6{Q)) W  **e$J(E"XX&;< ::)&+jjmm4I&JEJJO! F ))$9:FK : % & !s A DD c.tjdy)NamThe passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.)rrrIrGrgrgs MM 1rIcDtt||d}|&|jjdk7r |j}nF||jdk7r|}n2t jdtj j }|||k7rtd|d|d||S)ax Determine and return this FSDP instance's compute device. If a device is specified by ``device_id``, then returns that device. Otherwise, If the module is already on a non-CPU device, then the compute device is that non-CPU device. If the module is on CPU, then the compute device is the current device. Since this method should be called after materializing the module, any non-CPU device should not be meta device. For now, the compute device is always a CUDA GPU device with its explicit index. Precondition: ``_check_single_device_module()`` and ``_move_module_to_device()``. Nrrz4Inconsistent compute device and `device_id` on rank z: z vs )rfrrrSrrrr8)rrr r=rrs rGrrs, !&.94 @E U\\..%7 ,1F1K1Kv1U2N"\\&%**2K2K2MNN(^?T-TB4&d#8"9 ;   rIc"g}|jD]D}t|tdrt|td|j |j F|j d|Dt|t||tdy)z Synchronize module states (i.e. parameters ``params`` and all not-yet-synced buffers) by broadcasting from rank 0 to all ranks. Precondition: ``sync_module_states == True`` and ``self.process_group`` has been set. FTc3<K|]}|jywr\)detachrAs rGrbz2_sync_module_params_and_buffers..*s# rIc|Dcgc]}|jc}|Dcgc]}|jc}z}t|t||tdycc}wcc}w)Nrru)rtrzr*r{)rrr0rrparams_and_bufferss rG_sync_module_statesr4sm7==fU%,,.f=&-A&-F gA00BC#  >As AA"r|cD|rtd|Dr tdyy)Nc3`K|]&}|jtjdk((yw)rN)rr)r`tensors rGrbz>_check_module_states_for_sync_module_states..Js&;H e,,=s,.zThe module has CPU parameters or buffers when `sync_module_states=True`, which requires them to be on GPU. Please specify the `device_id` argument or move the module to GPU before passing it to FSDP.)rFr8)r|s rGrzrzGs7;H C  }rIc#K|j} t|}||vrt|s|#t$rYywxYww)aD Return an iterator over the original parameters in ``module``. The iterator does not return the parameters in ``ignored_params``, any ``FlatParameter`` s (which may be present due to nested FSDP wrapping), or any original parameters already flattened (only relevant when ``use_orig_params=True``). N)r1rfr StopIteration)rr param_genrs rGrrTsT!!#I OEN*3Ee3L     sA 4 AAAAc t|D]A\}}||vs t|rtd|d|jd|jy)a5 Check that original parameters in ``fsdp_module`` have been flattened. The flattened parameters are made invisible to ``named_parameters()`` for the module hierarchy rooted at ``fsdp_module``. This should be called as a sanity check after flattening the wrapped module's parameters. z Found an unflattened parameter: z;  N)rrrir> __class__) fsdp_moduler param_namers rG_check_orig_params_flattenedrjsZ?{K E  &/A%/H2:,b::<.%//!24 LrIch|tjk(rtjStjSr\)r%rrallreduce_hookreduce_scatter_hook)r1s rG_get_default_comm_hookr~s3  0 9 9 9 $$ . .rIc.tj|S)NrL)rrArLs rGrTrTs  % %M BBrIr\)rXrOrrtypingrrrrrrr r r r r rrrtorch.distributed distributedr^(torch.distributed.fsdp._exec_order_utilsfsdp_exec_order_utilsr'torch.distributed.fsdp._traversal_utils_traversal_utilsr$2torch.distributed.fsdp.fully_sharded_data_parallelfully_sharded_data_parallelr'torch.nnr(torch.distributed.algorithms._comm_hooksrtorch.distributed.device_meshrr"torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr"torch.distributed.fsdp._flat_paramrrrr%torch.distributed.fsdp._limiter_utilsrtorch.distributed.fsdp.apir r!r"r#r$r%r&r'torch.distributed.fsdp.wrapr(&torch.distributed.tensor.parallel.fsdpr)torch.distributed.utilsr*torch.utils.hooksr+rG torchdistxr,r- ImportErrorintr{rxr_HybridShardProcessGroupTypeProcessGroupTyperr SHARD_GRAD_OP HYBRID_SHARD_HYBRID_SHARD_ZERO2rr9#NO_RESHARD_AFTER_FORWARD_STRATEGIESrHr:boolrRrMrqrrOrrrrrrrrrrrrrr rrrrrrrrrrrKTensorrrergrrrrzrrrrArTrprIrGrs    CCAAFFBEA B   0D<-.""34 #D$5$5t7H7H$HIE$"3"35P"PQR 5>>!7!B!B""$:$H$H!!#9#F#F((*@*T*T !!(( ""(('# )- 0 0#0(0 W  0 *% 0  00f) )#)) ))X#$IJI4II  ARAR  !++!! !!H++ 4  d// /0& + + II+huxx78+%((,,-.%((//9R0SS +++\I9= :) ) II) %)c5<</01 )  ))X  II"4 4 014n-4*% 4  4  4!4 444n         '     : J *  *: :)):c5<</01:Hbii[$%678 :  :  ::z) )  )))))<66x896 ^6xBF)!%((*<*    D$$)$ X$:299S II  % c5<</01   <c5<</01 ell87 II7 %7^7 4: 7<  RYYK-. ^  #ELL1^B""-0^" "))_",3 II3 %3&3$ELL1 3  3l   %,, $ELL1 @# II# %#$ELL1#  #  \\ #N II  $$ 6   %,, $$ &   %      II  % bll , % (.>C$$CCM"s ^^ ^