PhddlZddlmZmZddlmZmZmZddlZddl m Z ddl m cm cmcmZddgZGddeZGddeZy)N)ABCabstractmethod)UnionIterableDict ModelAveragerPeriodicModelAveragerc(eZdZdZddZedZy)raHBase class for all model averagers. Args: process_group: The process group to be used for all-reduce. If ``None``, the default process group, which is created by :func:`torch.distributed.init_process_group`, will be used. (default: ``None``) NcX||ntjj|_d|_y)Nr)distgroupWORLD process_groupstep)selfrs C:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/algorithms/model_averaging/averagers.py__init__zModelAverager.__init__s'*6MDJJ`_, by running :class:`~torch.nn.DistributedDataParallel` (DDP) using the subgroups created by :meth:`~torch.distributed.new_subgroups`. Args: period (int): The number of steps per model averaging. Usually the period should be greater than ``1`` to reduce the communication cost. Otherwise, only DDP needs to be used. warmup_steps (int): The number of warm-up steps. During this stage, model averaging is skipped. process_group: The process group to be used for all-reduce. If ``None``, the default process group, which is created by :func:`torch.distributed.init_process_group`, will be used. (default: ``None``) Example:: >>> # xdoctest: +SKIP("undefined variables") >>> import torch >>> import torch.distributed as dist >>> import torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook as post_localSGD >>> import torch.distributed.algorithms.model_averaging.averagers as averagers >>> import torch.nn as nn >>> >>> dist.init_process_group("nccl", rank=rank, world_size=16) >>> torch.cuda.set_device(rank) >>> module = nn.Linear(1, 1, bias=False).cuda() >>> model = nn.parallel.DistributedDataParallel( >>> module, device_ids=[rank], output_device=rank >>> ) >>> # Register a post-localSGD communication hook. >>> state = PostLocalSGDState(process_group=None, subgroup=None, start_localSGD_iter=100) >>> model.register_comm_hook(state, post_localSGD_hook) >>> >>> # In the first 100 steps, run global gradient averaging like normal DDP at every step. >>> # After 100 steps, run model averaging every 4 steps. >>> # Note that ``warmup_steps`` must be the same as ``start_localSGD_iter`` used in ``PostLocalSGDState``. >>> averager = averagers.PeriodicModelAverager(period=4, warmup_steps=100) >>> for step in range(0, 200): >>> optimizer.zero_grad() >>> loss = loss_fn(output, labels) >>> loss.backward() >>> optimizer.step() >>> # Will average model parameters globally every 4 steps. Thus, >>> # inter-node communication only occurs every 4 iterations after >>> # the initial ``warmup_steps`` period. >>> averager.average_parameters(model.parameters()) ct|||dkr td||_|dkr td|dk(rt j d||_y)Nrz3Arg ``warmup_steps`` must be a non-negative number.z(Arg ``period`` must be a positive value.aWhen period is 1, no need to use model averaging because the communication cost of all-reducing parameters will be no less than the cost of all-reducing gradients by DistributedDataParallel in the backward pass. Therefore, only DistributedDataParallel should be used for this case.)superr ValueError warmup_stepswarningswarnperiod)rr(r%r __class__s rrzPeriodicModelAverager.__init__Tsc ' ! RS S( A:GH H q[ MMH   rrc|j|jk\rI|j|jz |jzdk(r tj||j |xjdz c_y)aV Averages parameters or parameter groups of an optimizer if ``step`` is no less than ``warmup_steps`` and it can be divided by ``period``, where ``step`` is increased by 1 at each iteration in the training loop. Args: params: The parameters of a model or parameter groups of an optimizer. rr"N)rr%r(utils&average_parameters_or_parameter_groupsrrs rrz(PeriodicModelAverager.average_parametersisY II** *T...$++=B  8 8ASAS T Q r)rN)rrrrrrrtorchnn Parameterrstrr __classcell__)r)s@rr r sY2n *x8J8J/KXVZ[^`e`h`h`r`r[rVsMt/t)ur)r&abcrrtypingrrrr-torch.distributed distributedr 2torch.distributed.algorithms.model_averaging.utils algorithmsmodel_averagingr+__all__rr rrrr:sB#(( BB 3 4"C"*XMXr