Ph3ddlZddlmZddlmZmZmZmZmZmZm Z m Z m Z ddl m Z mZddgZGddeZd d e d e d ed ze_ dd e ede ede ede ede edeededededededededefdZd e ede ede ede ede ededededededededefdZd e ede ede ede ede ededededededededefdZy)N)Tensor) Optimizer_use_grad_for_differentiable _get_value_stack_if_compiling_default_to_fused_or_foreach_differentiable_doc _maximize_doc _foreach_doc _view_as_real)ListOptionalAdamaxadamaxcheZdZ d ddddeededeffdZfdZdZed d Z xZ S) rF)maximizedifferentiableforeachrrc @d|kstd|d|kstd|d|dcxkrdksntd|dd|dcxkrdksntd|dd|kstd |t||||||| } t | || y) NzInvalid learning rate: zInvalid epsilon value: r?z#Invalid beta parameter at index 0: rz#Invalid beta parameter at index 1: zInvalid weight_decay value: )lrbetaseps weight_decayrrr) ValueErrordictsuper__init__) selfparamsrrrrrrrdefaults __class__s ]C:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/optim/adamax.pyr zAdamax.__init__ sby6rd;< <cz6se<= =eAh$$B58*MN NeAh$$B58*MN Nl";L>JK K%)  *ct|||jD]8}|jdd|jdd|jdd:t |j j }t|dk7xrtj|dd}|s<|D]6}tjt|dtj|d<8yy)NrrFrrstepdtype) r __setstate__ param_groups setdefaultliststatevalueslentorch is_tensortensorfloatfloat32)r!r/group state_valuesstep_is_tensorsr$s r%r+zAdamax.__setstate__/s U#&&E   Y -   Z /   -u 5'DJJ--/0 l+q0 eoo OF #7 !!LLqy)9O& "r&cd}|dD]F}|j|tj|z}|j||jjr t d|j|j|j |} t| dk(rxtjdtj| d<tj|tj| d <tj|tj| d <|j| d |j| d |j| dI|S) NFr"z(Adamax does not support sparse gradientsrrr)r() memory_formatexp_avgexp_inf) gradr2 is_complexappend is_sparse RuntimeErrorr/r1r4r6 zeros_likepreserve_format) r!r7params_with_gradgradsexp_avgsexp_infs state_steps has_complexpr/s r% _init_groupzAdamax._init_group=s xAvv~ 5++A. .K  # #A &vv"#MNN LL JJqME5zQ % S Ff #(#3#3U%:%:$i $)#3#3U%:%:$i  OOE), - OOE), -   uV} -/!0r&cNd}|$tj5|}ddd|jD]a}g}g}g}g}g}|d\} } |d} |d} |d} |d}|d}|d}|j||||||}t |||||| | | | | |||| c|S#1swY{xYw) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrr) rbeta1beta2rrrrrrK)r2 enable_gradr,rMr)r!closurelossr7rFrGrHrIrJrOrPrrrrrrrKs r%r(z Adamax.stepYs  ""$y%&&E! EHHK >LE5,CtB 0LI&GZ(H"#34N**52BE8U]_jkK  )!-' #'D K%$s BB$)gMb`?)g?g+?g:0yE>rNN) __name__ __module__ __qualname__rboolr r+rMrr( __classcell__)r$s@r%rr sh  "& +$ +$ + + +D P8".".r&aImplements Adamax algorithm (a variant of Adam based on infinity norm). .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{13mm} \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, u_0 \leftarrow 0 \text{ ( infinity norm)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}u_t \leftarrow \mathrm{max}(\beta_2 u_{t-1}, |g_{t}|+\epsilon) \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \frac{\gamma m_t}{(1-\beta^t_1) u_t} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. a  Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 2e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) z zd .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 r"rGrHrIrJrrrrKrrOrPrrc >td|Ds td|t||d\}}|r)tjj r td|r%tjj st }nt}||||||| | | | | ||| y)zrFunctional API that performs adamax algorithm computation. See :class:`~torch.optim.Adamax` for details. c3PK|]}t|tj ywrT) isinstancer2r).0ts r% zadamax..s@Kqz!U\\*Ks$&zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers)rrOrPrrrrrK)allrCr r2jit is_scripting_multi_tensor_adamax_single_tensor_adamax)r"rGrHrIrJrrrrKrrOrPrr_funcs r%rrs. @K@ @ ^  1&.TYZ 7599))+STTuyy--/#$   !%r&c:t|D]\} }|| }| s|n| }|| }|| }|| }|dz }| dk7r|j|| }tj|rTtj|}tj|}tj|}tj|}|j |d|z tj |j|jd|jj|jdgd}| stj|dd|n'|jtj|ddd|t|zz }||z }|j||| y)NrralphaFkeepdimout)rl)value) enumerateaddr2r@ view_as_reallerp_catmul_ unsqueezeabsadd_ unsqueeze_amaxcopy_raddcdiv_)r"rGrHrIrJrrOrPrrrrrKiparamr?r=r>step_tnorm_bufbias_correctionclrs r%reresu"f%5Qx#t$1+1+Q!  1 88E86D   E "&&u-E%%d+D((1G((1G  dAI&99 \\% * *1 -txxzs/C/N/Nq/Q RTU  JJxEw ? MM%**Xq%@ Aez&'999?" wt4C&r&c f| rJdt|dk(rytj|||||g} | jD]\\}}}}}}| rt j |}| rt |||||djr.t j|t jdddnt j|d|dk7r3| rt j|||nt j|||}t j||d|z t j||t||D]\}}t j|jd|j!j#| j%dgd}t j&|dd||j)j+f |Dcgc]}d|t-|zz }}t/|Dcgc] }d ||z z c}}t j0||||ycc}wcc}w) Nz#_foreach ops don't support autogradrrcpu)devicerirFrk)r1r"_group_tensors_by_device_and_dtyper0r2 _foreach_negr is_cpu _foreach_add_r4 _foreach_add_foreach_lerp_ _foreach_mul_ziprsrurvrwrxmaxnewlongrr_foreach_addcdiv_)r"rGrHrIrJrOrPrrrrrrKgrouped_tensorsgrouped_params grouped_gradsgrouped_exp_avgsgrouped_exp_infsgrouped_state_stepsrfr>r?rr(bias_correctionsrrs r%rdrd%s"DDD  6{aBBFES[]egrCstOixiijBe a.-)9;KM`cd !..}=M  .-9IK[ \ q ! ( (    3U\\#e5T\_ `    3Q 7 1 ##M>V % 2 2=.Xd e  -}a%iH ,e4 !1=AMGTyy""1%txxzs';'F'Fq'IJAH IIh5w @R@R@T6U V B GZZFYdAD)9 99FYZ!Vf"gVf?2o)=#>Vf"gh 0@BRTWXMjBF["gs H)7H. )NFFF)r2r optimizerrrrrr r r r r typingrr__all__r__doc__rXr5rrerdr&r%rsc '''" X |Y|~2      3*j# 5 L5 <56l56l 5 f 5d^5555 5 5 5 !5"#5p25 L25 <256l256l 25 f 25 25 25 25 2525252525j=Y L=Y <=Y6l=Y6l =Y f =Y =Y =Y =Y=Y =Y=Y=Y=Yr&