Phޢ dZddlZddlZddlmZddlmcmcmZddl m Z ddl m Z ddl mZddlmZmZddlmZdd lmZmZdd lmZmZmZdd lmZmZmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBmCZCe=eje=ejejffZEGddejZGGddeGZHGddejZJ dWdeKde de?e=e eKfd ed!e@ee>ffd"df d#ZL dXd$eKd%e d&eKd'e d(e@ee>ffd"efd*ZM dXd$eKd+ejd&eKd,ejd(e@ee>ffd"efd-ZNdeKde d.e?e=e eKeKfd/e?e=e eKeKfd0e>d"ejf d1ZO dYd2eKd%e d3eKd'e d0e>d4ePd(e@ed4ePd(e@ed"dfd7ZSd+ejd,ejd0e>d8eKd"ef d9ZT dXd2eKd%e d3eKd'e d0e>d4ePd(e@ed4ePd(e@ed8eKd"efd>ZWd ed?eKd@eKdAe>ejejgejfdBeKd"df dCZX dXdejjdDeBdEe:dFe"dGe@e>dHe@edSe@er? isinstancetorchTensorr.appenddetachtuplelenr/)r@xnew_ress rBforwardzOutputLogger.forwards ||H$$H a & JJ  ahhj )5 !c!fkc!A$i1nt{{}qtAw~~'71a9I&JKG NN ! !' *rCc|jjDcic]\}}|dk7r|jds|| }}}d|dScc}}w)Ntraining_z OutputLogger()__dict__items startswithr@kv clean_dicts rB__repr__zOutputLogger.__repr__sa ++- -1Zc): qD-  zl!,,  #A ))__name__ __module__ __qualname____doc__r'rHrI__annotations__ RNNReturnType _is_impurestrintr(r=rPr] __classcell__rAs@rBr-r-s  M""J&(?%?%?% ?%  ?% # ?%"?%?%?%?%c]?%c]?%F$-rCr-c.eZdZdZfdZdZdZxZS)OutputComparisonLoggerz Same as OutputLogger, but also requires the original activation in order to calculate the comparison at calibration time ct||i|tjjj j j|_d|_ g|_ y)Nsqnr) r<r=rHaonsfxutils compute_sqnr comparison_fncomparison_fn_name comparisons)r@argskwargsrAs rBr=zOutputComparisonLogger.__init__sE $)&)"XX[[^^11>>"(rCc$|js|St|tjsJd|jr)|j j |j|jj |j|||S)rEz#non-tensor inputs not yet supported) r>rGrHrIr?r.rJrKrvrt)r@rNx_refs rBrPzOutputComparisonLogger.forward sq||H!U\\*Q,QQ*  JJ  ahhj )  2 21e <=rCc|jjDcic]\}}|dk7r|jds|| }}}d|dScc}}w)NrRrSzOutputComparisonLogger(rTrUrYs rBr]zOutputComparisonLogger.__repr__sa ++- -1Zc): qD-  ) A66  r^)r`rarbrcr=rPr]rirjs@rBrlrls  7rCrlcXeZdZdZdej j dedeffd Z xZ S)NSTracerzy Just like a regular FX quantization tracer, but treats observers and fake_quantize modules as leaf modules. mmodule_qualified_namereturnct|tjjjryt|tjjj ryt |||S)rET)rGrHro quantization ObserverBaseFakeQuantizeBaser<is_leaf_module)r@r~rrAs rBrzNSTracer.is_leaf_module'sR a..;; < 58800AA Bw%a)>??rC) r`rarbrcrHnnModulergboolrrirjs@rBr}r}"s5@@@QU@@rCr}r2modelnodes_and_names_to_instrumentresults"op_to_type_to_weight_extraction_fnrctjjd|D]F\}}tjj }t |||}|s0||vr|ii||<|g||||<Hy)Nz=quantization_api._numeric_suite_fx._extract_weights_one_model)rH_C_log_api_usage_oncerWEIGHTvaluer ) r2rrrrnoder3res_typeextracted_weights rB_extract_weights_one_modelr1sy HH  !`a7h+22883 %;= w&%-rN!7G6HGH h ' 38rC model_name_agm_a model_name_bgm_b base_name_to_sets_of_related_opsunmatchable_types_mapctjjdt||||}g}g} |j D]D\} } | \} } |j | j | f| j | j | fFi}t|||||t||| ||t|t||}|S)Nz8quantization_api._numeric_suite_fx._extract_weights_impl) rHrrrrWrJ base_op_noderr r )rrrrrrrmatched_subgraph_pairsnodes_and_names_to_instrument_anodes_and_names_to_instrument_b match_namematch subgraph_a subgraph_brs rB_extract_weights_implrCs HH  ![\8 d4 ?A#>@#399; E!& J'.. 0G0G/TU'.. 0G0G/TU<  Gd;W*,d;W*, 7#6g|LG NrCmodel_amodel_bc tjjd| t}t |}g}g} t || } t || } t || j|} t|d} | | | _ t || j|}t|d}|||_ t|| |||||S)a Extract weights from model A and model B, and return a comparison. Args: model_name_a: string name of model A to use in results model_a: model A model_name_b: string name of model B to use in results model_b: model B base_name_to_sets_of_related_ops: optional override of subgraph base nodes, subject to change unmatchable_types_map: optional override of unmatchable types, subject to change op_to_type_to_weight_extraction_fn: optional override of function which extracts weight from a type, subject to change Return: NSResultsType, containing the weight comparisons z2quantization_api._numeric_suite_fx.extract_weightsnode_name_to_scope) rHrrrrr}rtracer_node_name_to_scoper)rrrrrrrtype_a_related_to_bskipped_module_namesskipped_module_classestracer_atracer_br maybe_model_a_node_name_to_scoper maybe_model_b_node_name_to_scopes rBextract_weightsrks4 HH  !UV'/ 0 2 ) @A')-/,.DEH,.DEH ww 7 8D'FwPd'e$'3#C ww 7 8D'FwPd'e$'3#C dL$0PA CCrC$nodes_and_names_to_instrument_inputs%nodes_and_names_to_instrument_outputs logger_clsctjjdi}i}|D] \}}} || f||<|D] \}}} || f||<t|||||}|S)Nz9quantization_api._numeric_suite_fx._add_loggers_one_model)rHrrr ) r2rrrr%node_to_instrument_inputs_to_ref_name&node_to_instrument_outputs_to_ref_namerr3 ref_node_types rB_add_loggers_one_modelrs HH  !\]JL)JL*)M%h 7?6O-d3*N)N%h 8@-7P.t4*O ! 4. J HE LrCname_aname_bshould_log_inputsc2tjjdt||||}g} g} g} g} |j D]\} \}}t |j |}t |j |}|r<| j|j| |f| j|j| |f| j|j| |f| j|j| |ft||| | |}t||| | |}||fS)Nz4quantization_api._numeric_suite_fx._add_loggers_impl) rHrrrrWrrrJ start_nodeend_noder)rrrrrrrrr&nodes_and_names_to_instrument_inputs_a&nodes_and_names_to_instrument_inputs_b'nodes_and_names_to_instrument_outputs_a'nodes_and_names_to_instrument_outputs_brrrref_node_type_aref_node_type_b new_model_a new_model_bs rB_add_loggers_implrsL HH  !WX8 d(*?A.0*-/*.0+.0+0F0L0L0N, ,Z-j.E.EtL-j.E.EtL  2 9 9&& OD F 2 9 9&& OD F 066  *o > @/66  *o > @1O")</=K)</=K  %%rCc ^tjjdg}g} t|| } t|| } t || j |} t |d} | | | _t || j |}t |d}|||_t|| ||||||S)aD Instrument model A and model B with loggers. Args: name_a: string name of model A to use in results model_a: model A name_b: string name of model B to use in results model_b: model B logger_cls: class of Logger to use base_name_to_sets_of_related_ops: optional override of subgraph base nodes, subject to change unmatchable_types_map: optional override of unmatchable types, subject to change Return: Returns a tuple of (model_a_with_loggers, model_b_with_loggers). Modifies both models inplace. z.quantization_api._numeric_suite_fx.add_loggersr)rrr) rHrrr}rrrrr)rrrrrrrrrrrrrrrrs rB add_loggersrs4 HH  !QR&(-/,.DEH,.DEH ww 7 8D'FwPd'e$'3#C ww 7 8D'FwPd'e$'3#C fdJ+)I3  55rCc tjjd|jD]\}}t ||xs5t |tj j xr|jdk(}|sM|j}||vri||<|j||vsJ|jd|j||vri|||j<|j|||jvrg|||j|j<|j}t|jdkDr |j}|j||j|j|j |j"|j$|j&|j(|j*d }t-|dr|j.|d<|j0|d<n g|d<d|d<|||j|jj3||||j|jj5d y) NzAquantization_api._numeric_suite_fx._extract_logger_info_one_modelr-z is already present in resultsr) typevaluesr0r5r1r4r7r8r9r:rvrur_c|dd|dS)Nr8:r7)ress rBz0_extract_logger_info_one_model..Bs~&'q-?)@(ABrC)key)rHrr named_modulesrGjitRecursiveScriptModule original_namer3r2r6r.rMr/r0r5r1r4r7r8r9r:hasattrrvrurJsort) rrrgm_namemod is_loggerr stats_to_usedatas rB_extract_logger_info_one_modelrs*  HH  !de++-  sJ ' 3 ? ?@8%%7  ,,C'!! >>5 B>>""@A B5ws|313 S--.~~WS\#2B2B%CCAC S--.s~~>99L3==!A%"}} ((&!$!2!2(+(@(@"%"4"4),)B)B$'$8$8 # 0 0ww" DsM*&)oo]#-0-C-C)*&(]#-/)* CL)) *3>> : A A$ G CL)) *3>> : ? ?C @ U.rC!model_name_to_use_for_layer_namesctjjdi}||fD]}t|||t |t ||}|S)a Traverse all loggers in `model_a` and `model_b`, and extract the logged information. Args: model_a: model A model_b: model B logger_cls: class of Logger to use model_name_to_use_for_layer_names: string name of model to use for layer names in the output Return: NSResultsType, containing the logged comparisons z6quantization_api._numeric_suite_fx.extract_logger_info)rHrrrr r )rrrrrrs rBextract_logger_inforIsV( HH  !YZG7#&ugzB$7#524G NrCnode_type_to_io_type_mapc tjjdt||||} t ||||| |||} | S)Nz;quantization_api._numeric_suite_fx._add_shadow_loggers_impl)rr)rHrrrr ) rrrrrrrrrrgm_a_shadows_bs rB_add_shadow_loggers_implrisU HH  !^_8 d4(fd$:J+!9;N rCc `tjjdg} g} t| | } t| | } t || j |} t |d}||| _t || j |}t |d}|||_t|| ||||||| S)a Instrument model A and model B with shadow loggers. Args: name_a: string name of model A to use in results model_a: model A name_b: string name of model B to use in results model_b: model B logger_cls: class of Logger to use should_log_inputs: whether to log inputs base_name_to_sets_of_related_ops: optional override of subgraph base nodes, subject to change unmatchable_types_map: optional override of unmatchable types, subject to change z5quantization_api._numeric_suite_fx.add_shadow_loggersr)rrrr) rHrrr}rrrrr)rrrrrrrrrrrrrrrrrs rBadd_shadow_loggersrs0 HH  !XY&(-/,.DEH,.DEH ww 7 8D'FwPd'e$'3#C ww 7 8D'FwPd'e$'3#C #fdJ+)I!93  55rCmodel_a_shadows_bctjjdtjt }t |||t|t||}t |S)al Traverse all loggers in a shadow model, and extract the logged information. Args: model_a_shadows_b: shadow model logger_cls: class of Logger to use model_name_to_use_for_layer_names: string name of model to use for layer names in the output Return: NSResultsType, containing the logged comparisons z=quantization_api._numeric_suite_fx.extract_shadow_logger_info) rHrr collections defaultdictdictrr r )rrrrs rBextract_shadow_logger_inforsV$ HH  !`a(44T:G"#4gzJ7#524G =rC model_name_1 model_name_2rtcomparison_namec|jD]}|jD]}||vs J|d||vs J|d||}||}|D]r} | d} | d} d} |D]} | d}| d}|| k(s|| k(s| } n| J| d}| d}g| |<t||D]"\}}|||}| |j|$ty)aY Compares the logged values from `model_name_2` against the corresponding values in `model_name_1`, using `comparison_fn`. Records the result in `model_name_2`'s results under `comparison_name`. Modifies `results` inplace. Args: results: the result data structure from `extract_logger_info` or `extract_shadow_logger_info`. model_name_1: string name of model 1 model_name_2: string name of model 2 comparison_fn: function to compare two Tensors comparison_name: string name of model to use for layer names in the output z not found in resultsr7r8Nr)rziprJ)rrrrtrresults_type_to_resultsmodel_name_to_results results_1 results_2result_2index_within_arg_2index_of_arg_2result_1 cur_result_1index_within_arg_1index_of_arg_1values_1values_2value_1value_2comparison_results rB%extend_logger_results_with_comparisonrsE*$+>>#3%<%C%C%E !#88 7. 56 78#88 7. 56 78.l;I-l;I%%-.@%A"!).!9$-L)56H)I&%1.%AN+/AA'>9#/%. +++#H-#H-,.)(+Hh(?$GW(5gw(G%_-445FG)@'&&F$4rCexample_inputsqconfig_multi_mappingbackend_configcustom_prepare_fncustom_prepare_kwargs custom_tracerc |tjgg}n|}tjj ||j |}|j |_t|} | j|t|jd} t|} t|} g} g}g}t|j| | | | ||}t!|}g}|j"D]6}t%|| |j||j }|j'|8t)|j+D]$\}\}}t-|||||j"|||&|S)a Given a model with a graph with M ops such as args_kwargs_m -> op_m -> output_m And a set of N qconfigs for each op, creates a new model, with each of the subgraph of `op_m` transformed into .. code:: |---------> op_m_n -> log_m_n | / args_kwargs_m ---------> op_m -> log_m_0 Where op_m_n is op_m wrapped in a submodule and transformed with qconfig_n, and its inner graph looks like .. code:: args_m -------- op_m_prepared_with_qconfig_n -> out_m_n / kwargs_m --- This is useful for testing different quantization of multiple layers in a single pass through the model. High level TODOs for future PRs: * figure out a better way to name the output structure * return a results data structure instead of printing it out * add examples to docblocks Fremove_duplicate) quantize_fxQuantizationTracerrHrqrrrrr propagaterrrrrgraphrqconfig_mappings_listrrJ enumeraterWr )rrrr r r r tracermt output_propmodulespatternsroot_node_getter_mappingstandalone_module_namesstandalone_module_classescustom_module_classesmatchessubgraphs_deduplist_of_node_name_to_qconfigqconfig_mappingnode_name_to_qconfig subgraph_idxrnodes_in_this_subgraphs rBprepare_n_shadows_modelr&s{V//B7   eV\\%%8 9B#66BR.KK>*2##U#;)+,.(* '8%=!: op0 -> x1 -> op1 -> x2 Example config: quantize op0 to int8, do nothing to op1. The following graph will be created: .. code:: x0_0 -> op0_0 -> x1_0 -> log -----> op1_0 -> x2_0 -> log \ \ \ # noqa: W605 ---> op0_1 -> x1_1 ----> clog -> op1_0 -> x2_1 ----> clog Where op0_0 is op0, op0_1 is op0 wrapped in a submodule and quantized to int8, op1_0 is op1 (appearing in the graph twice), log is a logger, and clog is a comparison logger. Fr)rrrHrqrrrrrrrrrrrrrrr!)rrr"r rrrrrrrrrrr r#s rB$_prepare_n_shadows_add_loggers_modelr(asN + +B 3F   eV\\%%8 9B#66BR.KK>*2##U#;)+,.(* '8%=!:cd|jD]\}}t|ts||_y)z; Sets the `enabled` setting on a `model`'s loggers N)rrGr-r>)rr>namechilds rBloggers_set_enabledr3s,**, e e\ *#EM-rCr?cd|jD]\}}t|ts||_y)zD Sets the `save_activations` setting on a `model`'s loggers N)rrGr-r?)rr?r1r2s rBloggers_set_save_activationsr5s-**, e e\ *%5E "-rCcustom_convert_fncustom_convert_kwargsc\|jjD]}|jjts#t ||j}|4t jjjj|}n |i}||fi|}t||j||S)zg Given a model from `prepare_n_shadows_model`, runs `convert_fx` on each shadow submodule. ) rnodesr1rXrgetattrrHrorr convert_fxsetattr)rr6r7rorig_mod converted_mods rBr+r+s !! 99   ? @udii0H ( % 5 5 A A L L! )0,.) 1( T>S T E499m 4" LrCc,i}t||t|S)z/ Extracts logger results from `model`. )rr-)rrs rBextract_results_n_shadows_modelr@s G"5'<@ NrCcFt|}t|}t|y)z2 Prints a summary of extracted `results`. N)rrr)rresults_groupedresults_comparisons rB!print_comparisons_n_shadows_modelrDs"08O2?C./rC)N)NNN)NN)FNN)FNNN)arcrrHtorch.nnr!torch.ao.quantization.quantize_fxrorrtorch.fxrtorch.fx.graphrtorch.ao.ns.fx.mappingsrtorch.ao.ns.fx.graph_matcherrrfx.weight_utilsr fx.graph_passesr r fx.utilsr r r fx.ns_typesrrr*torch.ao.quantization.backend_config.utilsr$torch.ao.quantization.backend_configr$torch.ao.quantization.fx.match_utilsr%torch.ao.quantization.fx.graph_moduler.torch.ao.quantization.fx.qconfig_mapping_utilsr)torch.ao.quantization.fx.quantize_handlerrtorch.ao.quantization.qconfigrtorch.ao.quantizationrtorch.ao.ns.fx.n_shadows_utilsrrrrrrr r!r"$torch.ao.ns.fx.qconfig_multi_mappingr#typingr$r%r&r'r(r)r*r+rIrerr-rlrr}rgrrrrrrrrrrrrrr&r(r/r3r5r+r@rDrrCrBrZs|Tl 77     ^>>QYW40   EHHHellE%,, *D$EEF f-299f-R#7\#7L @{-- @(Y] II I$(dCi(8#9I I )1c4(@R;S6S1T(U I  I.TXHLX\%% %%  % '/tC=M9N4N/O&P % $Dc2B.C)C$DE %)1c4(@R;S6S1T(U%%ZTXHLX\0C0C YY0C0CYY 0C '/tC=M9N4N/O&P 0C $Dc2B.C)C$DE 0C)1c4(@R;S6S1T(U0C0Cf +/uT3^/D*E,0dCn0E+F    YY >TXHL)& )& )& )&  )&  )&  )&'/tC=M9N4N/O&P)&$Dc2B.C)C$DE)& 299bii  )&d %SWHL,5 ,5 YY,5 ,5YY ,5  ,5  ,5'/tC=M9N4N/O&P,5$Dc2B.C)C$DE,5 299bii  ,5^3 993 33 3p YY YY(+   NTXKOHL         '/tC=M9N4N/O&P'tC5E1F,F'GH$Dc2B.C)C$DEYY8$SWKOHL+5 +5 YY+5 +5YY +5  +5  +5'/tC=M9N4N/O&P+5'tC5E1F,F'GH+5$Dc2B.C)C$DE+5YY+5\yy(+ :4H 4H4H4HU\\5<<8%,,FG 4H  4H  4Hv-16:^ 88??^^/^" ^  ) ^ $DcN3 ^^^BI 88??II$I" I  XX__ IX 88??$"   0$uxx$$$$ 6 88?? 6 6  6-16: )$DcN3 2588??}0}00rC