Ph? ddlmZmZddlZddlZddlZddlmZddlZddlZddl Z ddl Z ddl m Z m Z mZmZddlZddlmZddlmZddlmZdd lmZmZdd lmZmZmZdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'dd l(m)Z)ddl*m+Z+ddgZ,eGddZ-eGddZ.dZ/dej"dej"fdZ0de$defdZ1GddeZ2Gdde2Z3Gdd e2Z4de$de5fd!Z6d"e e$de e e$fd#Z7d$Z8d%ejrd&ejrd'e"d(e5d)e:f d*Z;Gd+deZ<Gd,deZ=y)-)ABCabstractmethodN) dataclass)ListUnionDictcast)Tensor)Future)Path)Metadata MetadataIndex) StorageReader StorageWriter WriteResult) LoadItemType LoadPlannerLoadPlanSavePlan SavePlannerReadItem WriteItem WriteItemType)_create_file_view)narrow_tensor_by_index)_get_device_moduleFileSystemWriterFileSystemReaderc0eZdZUdZeed<eed<eed<y) _StorageInfoz#This is the per entry storage info. relative_pathoffsetlengthN)__name__ __module__ __qualname____doc__str__annotations__intrC:\Users\daisl\Desktop\realtime-object-detection\venv\Lib\site-packages\torch/distributed/checkpoint/filesystem.pyr!r!2s- K Kr-r!ceZdZUeed<y)_StoragePrefixprefixN)r%r&r'r)r*r,r-r.r0r0;s Kr-r0z.distcptensorreturnc|jj}|jj|j k7r|j }|SN)detachcpu_typed_storage_sizenumelclone)r2s r._trimr<CsF ]]_ "F $$&&,,.8 Mr-itemc2t|j||S)N)index size_in_bytes storage_data)rr?)r=r@rAs r._result_from_write_itemrBJs jj L r-c<eZdZedZedZedZy) _TensorLoadercyr5r,selfsizeobjs r.addz_TensorLoader.addS r-cyr5r,rGs r. start_loadingz_TensorLoader.start_loadingWrKr-cyr5r,rMs r.valuesz_TensorLoader.values[rKr-N)r%r&r'rrJrNrPr,r-r.rDrDRs<      r-rDc$eZdZdZdZdZdZy)_SerialCpuLoaderc ||_g|_yr5) resolve_funitems)rGrTs r.__init__z_SerialCpuLoader.__init__as& r-c>|jj||fyr5)rUappendrFs r.rJz_SerialCpuLoader.addes 4+&r-cyr5r,rMs r.rNz_SerialCpuLoader.start_loadingh r-c#K|jD]y\}}|j|j}|j}|j j |j k7r|j}||f{ywr5)rUrTr6r7storagerHr:r;rG_rIr2s r.rPz_SerialCpuLoader.valuesksujjFAs%%c*113FZZ\F~~$$&&,,.8  !sB B N)r%r&r'rVrJrNrPr,r-r.rRrR`s'  r-rRcHeZdZd dZedZdZdZdZdZ dZ d Z y) _OverlappingCpuLoaderNc||_g|_||_d|_t j |_d|_d|_|r |jntjdj|_ t|j|_|xs|jj|_|j |jjk7r4|j j#|jjyy)NrFcuda)rTrUinflight_threshholdin_flight_data collectionsdeque current_itemsidxstarted device_typetorchdevicetyper device_modulecurrent_streamstream wait_stream)rGrTrprcs r.rVz_OverlappingCpuLoader.__init__xs& #6 0;0A0A0C 176--U\\&=Q=V=V/0@0@AC 2 2 A A C ;;$,,;;= = KK # #D$6$6$E$E$G H >r-cF|jt|jk\Sr5)rhlenrUrMs r._donez_OverlappingCpuLoader._donesxx3tzz?**r-cg}|j|jk\r|jj|j|jk\r|jj }|xj|dj |djzzc_|j||j|jk\r|SNr) rdrcrp synchronizergpopleftr: element_sizerX)rGdrainedvals r._drainz_OverlappingCpuLoader._drains   $":": : KK # # %!!T%=%==$$,,.C   3q6<<>CF4G4G4I#I I  NN3 !!T%=%==r-c|jj|j5|jsz|j|jkr`|j |j \}}|xj dz c_|j|j}|jj|jk(r|jdd}na|jtjdk(r?|jj|j!k7r|j#}|j$j'||f|xj|j!|j)zz c_|js|j|jkr`dddy#1swYyxYw)Nr r7T)rl non_blocking)rnrprtrdrcrUrhrTr6rlrmrjtorkr\rHr:r;rgrXryr]s r._refillz_OverlappingCpuLoader._refillsH    & &t{{ 3JJ''$*B*BBDHH-3A ))#.557==%%)9)99#YYe$YGF]]ell5&99~~',,.&,,.@!'"")) ##v||~8K8K8M'MM#'JJ''$*B*BB4 3 3s FF77Gc|jsJt|jdkDr|jj |jSrv)rtrsrgrprwrMs r._finishz_OverlappingCpuLoader._finishs=zzz t!! "Q & KK # # %!!!r-cl|jr td|jj||fy)Nz&cannot add items after loading started)ri RuntimeErrorrUrXrFs r.rJz_OverlappingCpuLoader.adds+ <<GH H 4+&r-c|jryd|_|jjd|jy)NTc |dSrvr,xs r.z5_OverlappingCpuLoader.start_loading..sadr-key)rirUsortrrMs r.rNz#_OverlappingCpuLoader.start_loadings. <<   N+ r-c#K|j|js7|j}|j|Ed{|js7|j Ed{y7*7wr5)rNrtr|rr)rGrzs r.rPz_OverlappingCpuLoader.valuessY **kkmG LLN  ** <<>!! !s*AA3A/A3A3)A1*A31A3)Ni@B) r%r&r'rVpropertyrtr|rrrJrNrPr,r-r.r`r`ws; I++N0" ' "r-r`cd}|jJ|jjD]}||z} |jjj}|tj j |zSNr ) tensor_datarH propertiesdtyperk_utils _element_size)r=rHsrs r. _item_sizersk D    '' '    " "  #    ' ' - -E %,,,,U3 33r-rUcj|dk(r|gS|Dcgc]"}|jtjk(s!|$}}|Dcgc]"}|jtjk7s!|$}}t|Dcgc]}g}}t|Dcgc]}d}}|j t dt |D]\}}|||zj||D]F}tt |dd} || j||| xxt |z cc<H|Scc}wcc}wcc}wcc}w)Nr rT)rreversec |dSrr,rs r.rz)_split_by_size_and_type..s1r-r) rmrBYTE_IOrangerr enumeraterXmin) binsrUwibytes_wtensor_wr^buckets bucket_sizesirhs r._split_by_size_and_typers$ qyw!FEbRWW 0E0E%ErEGF"GUrbgg1F1F&FUHG27+%>+Qb+G%>$T{+{!A{L+ MMj$M/7#2D  $$)L)~>qA BSZ^+  N#GG%>+s!"D!D!"D&D&0 D+ D0c |j}|jtjk(r'  ")(rBGG}7L7L,L[ M%")J"// ;D!((#FD*kJ#* +1--/&FJ!==(=!((#FFJ L+: HHV]]_-&   ] +S &% ;;   sOA)G+"F?F?8G "G-G1GBG $%G GGG+*G+c eZdZdZ ddeeejfdedede de ddf fd Z d eddfd Z d e de fd Z dee dee fdZd e dedeeefdZdedeeeddfdZxZS)raa Basic implementation of StorageWriter using file IO. This implementation makes the following assumptions and simplifications: * The checkpoint path is an empty or non-existing directory. * File creation is atomic The checkpoint consist of one file per write request plus a `.metadata` file with the serialized metadata. pathsingle_file_per_rank sync_files thread_countper_thread_copy_aheadr3Nct|t|ts t|}||_||_||_||_||_y)a Initialize the writer pointing to `path`. Args: path: directory where the checkpoint will be written to. single_file_per_rank: Produce one file per rank instead of one file per tensor/blob. Default to True. sync_files : force files to be synced to permanent storage. Default to True. thread_count: Number of IO threads to use to write. Default to 1. per_thread_copy_ahead: How many bytes to copy from the GPU ahead of saving then. Default 10Mb. N. B. If sync_files is disabled, there's no guarantee that the checkpoint will be consistent in the case of a failure. N) superrVrr rrrrr)rGrrrrr __class__s r.rVzFileSystemWriter.__init__BsI( $%:D $8!$(%:"r-is_coordinatorcyr5r,)rGrs r.set_up_storage_writerz&FileSystemWriter.set_up_storage_writer_rZr-planc@|jjdd|S)NT)parentsexist_ok)rmkdirrGrs r.prepare_local_planz#FileSystemWriter.prepare_local_planbs t4 r- global_planc t|Dcgc])\}}tj|td|d+}}}|Scc}}w)N__r^rA)r dataclassesreplacer0)rGrrr new_planss r.prepare_global_planz$FileSystemWriter.prepare_global_planfsT %[1 14   >Bqc)3L M1    s.Arc |jdfd}tj}|jrNt |j |j D]*}|}|j|j|z ||f,n:|j D]+}|}|j|j|z ||gf-tj}g} td|j D]W} tjt||||j|jf} | j| j!| Yt||||j|j| D]} | j#g} | |j%z } #tj&$r! t)} | j+| | cYSwxYw)Nrc>jt}dz |Sr)r1DEFAULT_SUFFIX)r file_count storage_plans r.gen_filez-FileSystemWriter.write_data..gen_filews,'../ |N;KLI !OJ r-r )targetargs)rrrrcr)rArQueuerrrrUrrr threadingThreadrrrstartrXjoinrrr set_result)rGrrrrbucketrr=rthreadsr^tresfutrrs @@r. write_datazFileSystemWriter.write_dataos (,'8'8    #(++-  $ $1!!4::%J  I 5y&IJ   $J  I 5y4&IJ#%*KKM q$++,A  . ..OO  A GGI NN1 - !% $ : :oo  A FFH |..00{{  -3XC NN3 J  sF1G  G metadataresultsct}|D]6}|j|Dcic]}|j|jc}8||_|jdz j d5}t j|||jr#tj|jddd|jdz j|jdz ycc}w#1swY:xYw)Nz .metadata.tmpr .metadata) dictupdater?rArrpickledumprrrrrename)rGrr storage_mdwr_listwr metadata_files r.finishzFileSystemWriter.finishsV G   7K7Rrxx87K L *ii/) / / 5 KK- 0--/06 _ $,,TYY-DEL 5 5sC% +AC**C3)TTr i)r%r&r'r(rr)rPathLikeboolr+rVrrrrrrr rrrr __classcell__rs@r.rr4s  &*%/ ;C$%;#; ;  ; # ; ;: D T xH> h??? [! " ?B F  F+/[0A+B F  Fr-ceZdZdeeej fddffd ZdefdZ de de de dfd Z defd Zd ed eddfd Zde de fdZdee dee fdZxZS)rrr3Nct|t|ts t|}||_t |_yr5)rrVrr rrrA)rGrrs r.rVzFileSystemReader.__init__s2 $%:D ?Cvr-sinfocDt||j|jSr5)rr#r$)rGfiler s r. _slice_filezFileSystemReader._slice_files u||U\\BBr-rrc t}|jD]H}|j|j}|j}|j |gj |J|jD]\}}|j|z jd5} |D]|} |j| j}|j| |} | jtjk(rRtj| j|j } | j#d|j%| | t't(t+j,| d} t/| | j0| j2} |j5| j7}|j9| j9k(s6Jd| jd|j9d| j9|j;| |j=| | dddt?}|jAd|S#1swYxYw)Nrbrr7) map_locationzreq z mismatch sizes z vs )!rrUrA storage_indexr" setdefaultrXrrrrmrrrrreadr$seek load_bytesr r rkloadrstorage_offsetslengthsresolve_tensorr6rHcopy_ commit_tensorr r)rGrrper_file read_itemitem_mdrr"reqsrreq file_slicebytesr2 target_tensorrs r. read_datazFileSystemReader.read_datas.2fI'' (?(?@G((D   b ) 0 0 ;$ $,>>#3 M4))m+11$74C"//0A0ABG!%!1!1$!@Jxx<#7#77 " :??7>>+J K 1 **36!%"EJJz$N""8"C$7$7")0(>(>s(C(J(J(L *..0FKKMAo!#"3"3!44D]EWEWEYDZZ^_e_j_j_l^mnoA%++F3--c=A) 87$42h t 587s FII c|jdz jd5}tj|cdddS#1swYyxYw)Nrr)rrrr)rGrs r. read_metadatazFileSystemReader.read_metadatas3ii+% + +D 1];;}-2 1 1s >ArrcB|j|_|jJyr5r)rGrrs r.set_up_storage_readerz&FileSystemReader.set_up_storage_readers"$11  ,,,r-c|Sr5r,rs r.rz#FileSystemReader.prepare_local_plans r-rc|Sr5r,)rGrs r.rz$FileSystemReader.prepare_global_plans r-)r%r&r'rr)rrrVr!rrrr r&rr(rr*rrrrr s@r.rrsFU3 #34FFC|C#h###L.x.-h--QU-xH> hr-)>abcrrrrrerrrrrtypingrrrr rkr torch.futuresr pathlibr rrrr\rrrrrrrrrrrrutilsrtorch.distributed._shard._utilsr torch._utilsr__all__r!r0rr<rBrDrRr`r+rrrrrrrrr,r-r.r5s# !  **     %B+      %,,5<<   C  }.N"MN"b4Y434i $y/4"3  3 ++3 3  3  3 lHF}HFV?}?r-