Ph0 UddlZddlZddlZddlmZmZmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZddlm Z m!Z!ddl"m#Z#m$Z$m%Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-m.Z.dd l/m0Z0dd l1m2Z3d Z4d Z5e5dZ6dZ7e7dZ8dZ9ee:Z;e<Z=ee e>d<eeeej~e@eAe:fZBeeBeeBeeBe e:dffZCe e:eCfZDeeDZEe e:eeDeEffZFejdZHeGddZIeGddeIZJd>dejde:deLde;fdZMddddejdeejjdfd eLd!eeejd"eeIdeJf d#ZPd$e e:eCfd%eFd&eJddfd'ZQd(eejejjfd)e:de fd*ZRdejd&eJde e:eCffd+ZSdejd,e e:eCfd&eJde0fd-ZTd.ejjddfd/ZUdejd0eejjdfd&eJdeFfd1ZVdejd.ejjd%eFd&eJdeFf d2ZWdejd0eejjdfd,eFd&eJddf d3ZXddddejd!eeejd"eeIde e:eCffd4ZYddddejd0eejjeejjfd!eeejd"eeIdeFf d5ZZddddejd0eejjeejjfd!eeejd"eeIdee e:eCfeFff d6Z[dejd,ee eje e:eCffe e:eCffde e:eCffd7Z\dd8dejd$ee eje e:eCffe e:eCffd"eeIde0fd9Z]dd8dejd0eejjeejjfd%eFd"eeIddf d:Z^dd8dejd0eejjeejjfd$ee eje e:eCffe e:eCffd%eFd"eeIde0f d;Z_edd8dejd"eeIddfd<Z`edd8dejd0eejjdfd"eeIddfd=Zay)?N)asdict dataclassfield)chain) AnyCallablecastDictIterableList no_type_checkOptionalSetTupleUnion) ShardedTensor)DTensor)_gather_state_dict_offload_state_dict_to_cpu)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)_IncompatibleKeys)DistributedDataParallel _flat_param param_groups.stateparams_patched_state_dict ValueTypec#Ktj}tj dtj|rtjyy#tj|rtjwwxYwwN)gc isenableddisablecollectenable) is_enableds rC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/checkpoint/state_dict.py gc_contextr2Cs[JJJL   IIK   IIK s)B A,B -B  B cXeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<y ) StateDictOptionsa This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes``: when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). The default value is False. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictN) __name__ __module__ __qualname____doc__r5bool__annotations__r6r7r8r9r1r4r4Ps=<"OT!K!&$&$(T(FDrAr4c(eZdZUeeZeeee jfee e jffe d<ee Zeee d<ee Zeee d<dZee d<dZee d<ej*Zee d<eeZeej8e d <y ) _StateDictInfo)default_factoryfqn_param_mappingall_fqnssubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)r:r;r<rdictrEr rstrtorchTensorFQNS_Tr?setrFrrGrHr>rI contextlib nullcontextrJrlistrKr nnModuler@rAr1rCrCws d#t c5<< % (<"==$s3Hc#h3#(#=C=L$L$'33L(3$)$$?L$ryy/?rArCmodelnameskip_ddp_prefixreturncvd|vr|hS|jd}g}|}t|D]\}}t|tr(|dk(sJ|j}|r,|j |>t|t r||dztk(rHdj|}t|t} |r|d}| jD chc]} ||  c} cSt|t}|tk7s|j |t||}|j |t||}dj|hScc} w)a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `Set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. r$module) split enumerate isinstanceDDPr\appendFSDP FLAT_PARAMjoingetattr_fqnsr) rWrXrY obj_names fqn_obj_namescurr_obji curr_obj_nameprefix flat_paramfqns r1 _get_fqnsrps< $v  3IMH%i0= h $ H, ,,H"$$]3 $ 'Q:--0$Xz:  &xq\F4>4D4DE4DS6(3%(4DEEx)<=H 33$$]3"8];   /x7H'1* HH] # $$Fs; D6) submodulesoptionsoptims. optim_onlyrqrrc |r |s td|xs t}i}t}|jD]3\}}t ||} | ||<| D]} ||| <|j | 5t} |rdt|}|j D]F\}} | |vr t ||} t| dk(sJd| D]} | j | dHtj|} | r|jrSt|j|j}t|j|j}tj}n0t!}t#|j}tj$}t'j(tj*||||}nt,j.}t1d it3|||| |t5t6t8j:| | t|dkDd S) zW Verify the model and options passed by the user and generates _StateDictInfo. z;Optimizers are not passed in but optim_only is set to True.r]z)Submodule FQN should only have 1 instancer$)offload_to_cpu rank0_only)rv)r\state_dict_typestate_dict_configoptim_state_dict_configr)rErFrGrJrKrHrIr@) RuntimeErrorr4rQnamed_parametersrpadd named_moduleslenrcrKr5rr6rrFULL_STATE_DICTrrSHARDED_STATE_DICT functoolspartialrxrRrSrCrr r rUrV)rWrsrtrqrrrErFrXparamfqnsrorGr\rKryrzrxrJs r1_verify_optionsrs& I  +)+G uH--/ e%#'% C%* c " LL 0_ !//1LD&Z'UD)Dt9> N#N N>"&&#ay1 2$$U+L  " " 3&22w?R?R! '?&22w?R?R' #,;;O 6 8 &A&22' #,>>O ((  +/$;  "--   / +-!$ryy/<8#^&kAo  rAmodel_state_dictoptim_state_dictinfocXd}|jD]&}t|}|Jd|js$d}n|jr |s td|jr_|s]|j sQ|j sE|jr |js-|jr!tdtjd|jr1|r |ts&|jr |jstd||jD]}t|vs t|dtd y) NFz)Expected a fsdp_state with a fsdp module.Tz:The model has FSDP modules but no FSDP root module exists.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=r$zgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rKr_is_rootr{rHrGr7r6r5r9distget_rankrISTATEkeysrd)rrr has_fsdp_rootr\ fsdp_statekeys r1_verify_state_dictrs@ M##CFK %R'RR%    M  $  WXX  ''))!!d&:&: KK 'mmo'q *    %5e%<   !5!5::J9KM   $$&  %z*.** 'rAobjapict||}|tvr+tjt|j||}|S)N)self)rfr'rr __class__)rrcalls r1_state_dict_fnr5s9 3 D ""  !<3G KrAc\|jsiS|j5t|d}dddtj D]p}t ||}t |dk(sJtt|}||k7s9dtfd}|||std|d||j|||<r|jrgi}|j D]P}|jD]?}|j|s|jr ||||<*|t |d} |||| <AR|}|jrI|j!D]6\}} | j"rt ||}|D]}|j|8t|j%D]#\}} | j&s|j|%|j(r0|j*s t-nd} t/||j*| S|j*r t1|S|S#1swYxYw) N state_dictr]rZct|t|k\ry|jd}|jd}d}t|D];\}}|||k(r'|dz }|t|k(s"|t|dz k(cS|dk(r;yy)NFr$rr]r\T)rr^r_)rro fqn_split key_splitfqn_idxkey_idxkey_names r1verifyz%_get_model_state_dict..verifyLss8s3x' IIcN IIcN )29)=%GX9W#551 "c)n4#*c)nq.@#@@!X- $*>rAzAn unexpected key, z, exists. FQN is rr6 ranks_only)rHrJrrTrrprnextiterr>r{poprG startswithr8r7r| requires_graditemsis_metar5r6tuplerr) rWrrrrrornew_state_dictrmnew_fqnrprs r1_get_model_state_dictr<s      8^E<8:  JOO%&$4yA~~4: #: D "#s#"%8=Nse#TUU(nnS1JsO5'8 /1??$C11~~f-//*4S/N3'!#f+-0G.8oN7+2%$    002JC""UC(Ds# 3z'')*Q 99 NN3 + $($4$4UW$ ! D$4$4     )*55C  s H!!H+rc |jr|s tiiS|jD]M\}}t||}t||d}t ||D]\}}||k7s |j |||<!O|j 5ttt|d||jcdddS#1swYyxYw)NF)rYload_state_dict)rr9) rHr r|rpziprrJr rr9) rWrrr_rfqns_with_ddp_prefixrofqn_with_ddp_prefixs r1_load_model_state_dictrs   J R((((*Q$(UK(+D2F(G $C$))2<..2E ./)H+      4N5"3 4%dkk     s ,CC optimc |jry|jD]N}|tD]@}|j t d|j s't j||_BP|jd|jdy)zH Initialize optim states by calling the step() with zero grads. Na state_dict can only be used if the optimizer states are initialized (usually after one step() with gradients) or gradients are None. For the later case, state_dict will fake the gradients as zero to initialize the optimizer states. However, the gradients are not None.)closureT) set_to_none) r%r#PARAMSgradr{rrN zeros_likestep zero_grad)r param_grouprs r1_init_optim_staters {{))  (Ezz%"."""--e4 )* JJtJ OOO%rA optimizersc |jsiStitgi}|D]}t|t |d}|j r2|j 5tj|||}dddn/ttjd|jD}tt|tt!|}i}|j#D]I\} } t%|| } t!| dk(sJt't)| } | |vr;|| } | || <| || <Kt|tj+D])} || } |tj-| |t| <+|tD]#}|t.D cgc]} ||  c} |t.<%|st1t2|tj5|tt1t6|tj9|t|j:r0|j<s t?nd}tA||j<|S|j<r tC|S|S#1swYxYwcc} w)Nrc3.K|] }|tywr*)r).0gs r1 z(_get_optim_state_dict..s-TASAaiASsr]rr)"rIrPGrrrKrJrcrrTr from_iterabler#rLrrangerr|rprrrrrr DictValueTypeupdateListDictValueTypeextendr5r6rrr)rWrrrrosdr&param_pid_mappingfqn_pid_mappingrrrropidgrouprs r1_get_optim_state_dictrs>    ,12r2+>% 1nUL13   ""$++E5#>%$%---TASAS-TTUF $Ss6{1C%D E  O#446 U ,4yA~%~4:& 11'.'*$'*$7CJOO-.%c*"%e*.."5E 3/RAFv O#!5 Of !  ],U34;;CJG  0 45<>B ct5t|td||}t||}t |i||cdddS#1swYyxYw)a Return the model state_dict of ``model``. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. submodules: Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``model``. FrtrqrrN)r2rrrr)rWrqrrrrs r1get_model_state_dictr:sM,   G!  1=+R6 s 3AAct5t|tjjr|fn t |}t ||d||}t|||}ti|||cdddS#1swYyxYw)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules: Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. TrN) r2r`rNr Optimizerrrrr)rWrrqrrrrs r1get_optimizer_state_dictr]sy2 *ekk&;&;<Mz"    !  1 DI2/6 s AA33A<ct5t|tjjr|fn t |}t ||d||}t||}t|||}t|||||fcdddS#1swYyxYw)a) Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: import torch from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.nn.parallel import DistributedDataParallel as DDP from torch.distributed.checkpoint.state_dict import get_state_dict fsdp_model = FSDP(copy.deepcopy(model)) fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) ddp_model = DDP(copy.deepcopy(model)) ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) fsdp_state_dict, fsdp_optim_state_dict = get_state_dict(fsdp_model, fsdp_optim) # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), # the asserts will fail. assert ddp_state_dict == fsdp_state_dict assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules: Optional[Set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. FrN) r2r`rNrrrrrrr)rWrrqrrrrrs r1get_state_dictrsH *ekk&;&;<Mz"    !  1=0 DI+-=tD!11! s A,BB c ||siSttt|jtj rt ttj tttff|}i}|jD]\}}|jD]y\}}||k7r t||}t|dk(sJdtt|d} |j|jD cic] \} } | | z|  c} } {|St tttf|Scc} } w)Nr]z/FQNs for a submodule should only have 1 elementr$)r`rrrrUrVr r rMr(rr~rprr) rWrcast_state_dictr submodulesub_state_dictrXmrrmsubfqnrs r1_unflatten_model_state_dictrs!  $tJOO-./;tBIItCN/C$CDjQ/1)8)>)>)@ %I~ ..0a > -4yA~X'XX~ d,-Q/%%AOAUAUAWXAW Vf_e+AWX1*ADi(*55 Ys=D8)rrct||}t5t|td|}t |i|t |||cdddS#1swYyxYw)a2Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys FrtrrN)rr2rrrr)rWrrrrs r1set_model_state_dictrsS:.I . ueg%Q+R6%e-=tD s 1AAct5t|tjjr|fn t |}t ||d|}ti||t||||dddy#1swYyxYw)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None TrN) r2r`rNrrrrrr)rWrrrrrs r1set_optimizer_state_dictr sk2 *ekk&;&;<Mz"  ujT7S2/6uj2BDI s AA11A:c.t||}t5t|tjj r|fn t |}t||| |}t|||t||||t|||cdddS#1swYyxYw)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. rN) rr2r`rNrrrrrrr)rWrrrrrrs r1set_state_dictrEsR.I . *ekk&;&;<Mz"   :.>*>  +-=tDuj2BDI%e-=tD s A*B  Bc$tjt||fd}||_tjt||dt t tfffd }||_tj|tj|y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rWrrcSr*r@_state_dict_callsr1state_dict_callz0_patch_model_state_dict..state_dict_call !!rArc|y)N)rr@r_load_state_dict_calls r1load_state_dict_callz5_patch_model_state_dict..load_state_dict_call z:rAN) rrrrrr rMrrr'r})rWrrrrrrs @@r1_patch_model_state_dictrs6!(( "'E%-- ;c3h;1EO,01rActjt|||fd}tjt|||dtt t fffd }tj|tj|t|tjjr|fn t|}|D]}||_||_y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rWrrrcSr*r@rsr1rz4_patch_optimizer_state_dict..state_dict_callrrArc|y)N)rr@rs r1rz9_patch_optimizer_state_dict..load_state_dict_callrrAN)rrrrr rMrr'r}r`rNrrrrr)rWrrrrrrrrs @@r1_patch_optimizer_state_dictr s>!((  "&--  ;c3h;O,01 j%++"7"7 8  :  * 4rA)T)brRrr+ dataclassesrrr itertoolsrtypingrrr r r r r rrrrrNtorch.distributed distributedrtorch.nnrU'torch.distributed._shard.sharded_tensorrtorch.distributed._tensorr.torch.distributed.checkpoint._state_dict_utilsrrtorch.distributed.fsdprrrrcrrrrr$torch.distributed.fsdp._common_utilsrrtorch.nn.modules.moduler torch.nn.parallelr!rardr PG_PREFIXr STATE_PREFIXrrMrPrQr'r?rOintfloat PrimitiveTyper(rrOptimizerStateTypecontextmanagerr2r4rCrVr>rprrrrrrrrrrrrrrrrrrrr r@rAr1r s 00     A-   6< d!H {  S%(US]*g}ellCKL 4 &m(TT  S)^$ '#u]4E%EFFG    ## #L  @% @  @*%RYY*%c*%D*%F*%d,0*. P 99P %++'', -PP RYY( P & ' PPf.3 >*.(. . .bbii)>)>>?chG 99G*G #y.GT 99 S)^$     0&U[[22&t&22 992 ekk++S012  2  2 j3 993 ;; 3)3  3  3lN 99Nekk++S01N#N  N  N6,0*.  99  RYY(  & '   #y.  N,0*. ( 99( ekk++Xekk6K6K-LLM( RYY( ( & ' (  ( ^,0*. T2 99T2ekk++Xekk6K6K-LLMT2RYY( T2 & ' T2  4Y !3 34 T2n6 996d299d3 >&::;T#y.=QQR6 #y.6@+/ $E 99$E RYYS)^, ,-tCN/CC$E & ' $E$EX+/ "J 99"Jekk++Xekk6K6K-LLM"J) "J & ' "J  "JZ+/8E 998Eekk++Xekk6K6K-LLM8E RYYS)^, ,-tCN/CC 8E)8E& '8E8Ez+/12 9912& '12 1212l +/ ;5 99;5ekk++S01;5& ' ;5  ;5;5rA