Ph, ddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z ddl mcmcmcmZddlmZmZddlmZddlmZddlmZmZddlmZdd lm Z dd l!m"Z"dd l#m$Z$gd Z%e$e&Z'eGd dZ(GddZ)de ee*dfde ede*fdZ+de de e e*e e,ffdZ-de(de ee*dfde edee,effdZ.y)N) dataclassfield)AnyCallableDictListOptionalTupleUnion)eventsmetrics) WorkerSpec)LocalElasticAgent)SignalExceptionStd)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) LaunchConfigelastic_launch launch_agentceZdZUdZeed<eed<eed<dZeed<dZeed<dZ eed <d Z eed <e e Z eeefed <dZeed<dZeed<dZeed<dZeed<dZeeed<dZeeed<ej2Zeeeeeffed<ej2Zeeeeeffed<e e Zeeefed<dZeeed<dZy)ra Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). log_dir: base log directory where log files are written. If not set, one is created in a tmp dir but NOT removed on exit. redirects: configuration to redirect stdout/stderr to log files. Pass a single ``Std`` enum to redirect all workers, or a mapping keyed by local_rank to selectively redirect. tee: configuration to "tee" stdout/stderr to console + log file. metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. ..note: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_noderun_id default_rolerole rdzv_endpointetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsmonitor_intervalspawn start_methodNlog_dirlog_line_prefix_template redirectstee metrics_cfg local_addrcd}|jdk7r|j|jd<yd|jvr||jd<yy)Nir&timeout)r'r%)selfdefault_timeouts iC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/launcher/api.py __post_init__zLaunchConfig.__post_init__[sN    "+/+<+L$sCx.>L#L# e L#!GXc]!.2hsm2,/HHIuS$sCx.()4&)hhCsDcN" #."'"=Kc3h= $J $;r:rc2eZdZdZdedeeedffdZdZ y)ra Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") config entrypointNc ||_||_yN)_config _entrypoint)r6rGrHs r8__init__zelastic_launch.__init__~s  %r:cVt|j|jt|SrJ)rrKrLlist)r6argss r8__call__zelastic_launch.__call__sDLL$*:*:DJGGr:) r;r<r=r>rr rrArMrQrEr:r8rrcs04&&(C-.&Hr:rrHrPreturnct|tr |jSt|tr(|tj k(rt d|DdS|Sy)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. c32K|]}|ddk7s |yw)r-NrE).0args r8 z'_get_entrypoint_name..s>A# s r) isinstancerr;rAsys executablenext)rHrPs r8_get_entrypoint_namer]sL*h'""" J $  '>>C C r:rdzv_parametersc|jdk7ry|j}|j}|s tdt |d\}}|dk(rtd|d||fS)Nstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr&) default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstrip ValueErrorr)r^rc master_addr master_ports r8_get_addr_and_portrhs(*''H~~H  Y   9PRSKb+H:5S T    %%r:rGcx|jsDttjj}t j d|||_t||}t jd||j|j|j|j|j|j|j|j|j |j"|j$d t'd |j|j|j|j|j|j(d|j}t+|\}}t-|j.|j|t1|t3j4||j|j |j6|j8|||j( }t;||j<|j"|j>} d} tAjBtAjD|j$| jG} tIjJ| jM| jOrtQ|| jR| jT| r|jVjYSS#tP$rtZ$r'd } tIjJ| j]t^$r%tIjJ| j]wxYw#| r|jVjYwwxYw) Nz3config has no run_id, generated a random run_id: %saStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s ) rHrrrrr#r!r%r)r+r.r2)rbrcrrrr3) r local_world_sizerHrP rdzv_handlerr)r+r0r1rfrgr3)specr-r.r/T)namefailuresFrE)0rrAuuiduuid4r?loggerwarningr]inforrrr#r!r%r)r+r.r2rr3rhrr tuple rdzv_registryget_rendezvous_handlerr0r1rr-r/r initialize_metrics MetricsConfigrunr recordget_event_succeeded is_failedrrn return_valuesrkshutdownrget_event_failed Exception) rGrHrPrentrypoint_namer^rfrgrlagent shutdown_rdzvresults r8rrs ==TZZ\%%&LfU *:trs  (DDDEE5AXJMEP> < H  C;C; C;L$H$HNhT)*26s),&)& 8C=(3- '(&&m) m)hT)*m) s)m) #s(^ m)r: