PhddlmZmZddlmZmZmZmZmZm Z ddl Z ddl m Z ddlmcmZddlmZddlmZmZddlmZmZddlmZmZmZmZddlm Z Gd d eZ!Gd d e!Z"Gd de!Z#y))ABCabstractmethod)AnyCallableDictListOptionalTupleN)FakeTensorMode)DataParallelStylepartition_data_parallel)_convert_to_distributedSchema) DeviceMesh Placement ReplicateShard) GraphModuleceZdZdZededejjde ejjde e efde e efdeedfd e e efd efd Zeded efd Zy ) ParallelModez Basic Parallel Mode interface. Each parallelism pattern should implement this interface to describe how to partition and compile the graph in the spmd compiler. gmmodel optimizerparams_and_buffers named_statesargs.kwargsreturnct)z Partition a single device graph to a distributed graph. TODO(@wanchaol): some of these arguments are not necessary for partitioning, remove the unnecessary ones later. NotImplementedError)selfrrrrrrrs pC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/_spmd/parallel_mode.py partitionzParallelMode.partitions ""##ct)a Transform and compile a distributed graph with a set of graph transformation and optimization passes for each parallel mode. The returned result should be a compiled executable graph in the distributed environment. r r"rs r#transform_and_compilez"ParallelMode.transform_and_compile,s "##r%N)__name__ __module__ __qualname____doc__rrtorchnnModuler optim Optimizerrstrrr r$r(r%r#rrs $ $xx$EKK112 $ !cN $ 38n $CHo$S#X$ $$$ $ $ $ $r%rceZdZdZ dddddededeeegeffdZ d ed e jjd ee jjd eeefd eeefdeedfdeeefdefdZd edefdZy) DataParallelzData Parallelism mode.rN)input_batch_dim custom_passesparallel_styler6r7c|dk(rtj|_nD|dk(rtj|_n)|dk(rtj|_nt d|||_|||_yd|_y)a DataParallel Mode that partition the model and graph to data parallel style parallelism (i.e. DDP/FSDP/ZERO-3). It currently supports three different parallel styles: "replicate", "fully_shard", and "default". See :class:`DataParallelStyle` for more details. Args: parallel_style (str): parallel style to use. Currently supports "replicate", "fully_shard", and "default". Keyword args: input_batch_dim (int): the batch dimension of the input tensor. default: 0 custom_passes (Callable[[GraphModule], GraphModule], optional): A custom callable that overrides the default graph transformation and optimization passes. replicate fully_sharddefaultzUnknown parallel style: Nc|SNr3rs r#z'DataParallel.__init__..er%)r REPLICATEr8 FULLY_SHARDDEFAULT RuntimeErrorr6 _gm_passes)r"r8r6r7s r#__init__zDataParallel.__init__<sz0 [ ("3"="=D  } ,"3"?"?D  y ("3";";D !9.9IJK K /  $DQDO,DOr%rrrrrr.rrc tdtjtj}t |||||||||j |j }|S)Ncuda)rr-arangedistget_world_sizer r8r6) r"rrrrrrrmeshs r#r$zDataParallel.partitiongs]&%,,t/B/B/D"EF $                r%c$|j|S)z>optimize a distributed graph with a set of optimization passesrFr's r#r(z"DataParallel.transform_and_compilesr""r%)r:)r)r*r+r,r2intr rrrGr-r.r/r0r1rrr r$r(r3r%r#r5r59s *), !HL ),), ),  + )C DE ),V xxEKK112  !cN  38n CHoS#X 6# # #r%r5ceZdZdZ ddeeegeffdZdedejjdeejjde eefd e eefd eed fd e eefd efdZded efdZy)DTensorExpandModez The DTensor Expand mode. It's replicating the parameters and shard the inputs to represent DDP like behavior, it's currently a transitent mode before we move to the new data parallel expansion. Nr7c6i|_|||_yd|_y)Nc|Sr>r3r?s r#r@z,DTensorExpandMode.__init__..rAr%)_placements_overriderF)r"r7s r#rGzDTensorExpandMode.__init__s$AC!  $DQDO,DOr%rrrrrr.rrc 2tj|i|}tdtjt j j} t| tdg} t| tg} gg} } tj|D]Q}t|tjsJdt|| j|| j| Stj|D]t}t|tjr#| j|| j| @| jtj d| j| v|D]}t|tjrm| j|t#||j$vr3| jt| |j$t#|x| j| | jtj d| j| t'd5| Dcgc]}tj(|}}dddt+|| | ddScc}w#1swY!xYw) NrIr)rM placementszexpecting Tensor but got T)allow_non_fake_inputsF) default_mesh_allow_partial)pytreearg_tree_leavesrr-rJrKrLrIrrr tree_leaves isinstanceTensortypeappendemptyidrUr empty_liker)r"rrrrrrr flat_argsrM shard_schemareplicate_schemainpsschemaspoainp fake_inpss r#r$zDTensorExpandMode.partitions **D;F; &%,,t/B/B/D"E"J"J"LM%4U1XJG #)t #NBg##$67Aa. U2KDQRG90U U. KKN NN+ ,8 ##L1A!U\\* A/0 EKKN+/0 2A!U\\* Aa5D555NNDT5N5NrRSu5UVNN<0 EKKN+|,!$$ 7:>?$3))#.$I?8'  7e   @8 7s J J+J J  Jc$|j|S)z Transform and compile a distributed graph with a set of graph transformation and optimization passes for the dtensor fallback parallel mode. rOr's r#r(z'DTensorExpandMode.transform_and_compiles r""r%r>)r)r*r+r,r rrrGr-r.r/r0r1rr2rr r$r(r3r%r#rRrRsOS,%h }k/I&JK,7 7 xx7 EKK112 7 !cN 7 38n 7 CHo7 S#X7  7 r# # #r%rR)$abcrrtypingrrrrr r r-torch.distributed distributedrKtorch.utils._pytreeutils_pytreer[torch._subclassesr %torch.distributed._spmd.data_parallelr r "torch.distributed._spmd.distributerrtorch.distributed._tensorrrrrtorch.fxrrr5rRr3r%r#r|s[#== $$,OMM $$3$$NL#<L#^P# P#r%