P/j}*>ddlmZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZej&j)edZej&j)edZe e e eej2fZee egefZej8d Gd d Zej8d Gd dZej8d GddZ d&dZ d'dZ!d(dZ"d)dZ#d*dZ$ d+dZ% d,dZ&d-dZ' d.dZ( d/dZ)dZ*ej8GddZ+ d0dZ,d1dZ-Gd d!eZ. d2d"Z/ d3d#Z0ej8d Gd$d%Z1y)4) annotationsN)Enum)AnyCallableDictListOptionalSequenceTupleUnion)counters) InputType perf_hintscudagraph_static_inputsT)frozenceZdZUdZded<y) FunctionIDz9Unique counter of a function wrapped in cudagraphify_implintidN__name__ __module__ __qualname____doc____annotations__b/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/torch/_inductor/cudagraph_utils.pyrrs ? Grrc:eZdZUdZded<ded<ded<ded<y ) PlaceholderInfoz A serializable version of torch.fx.Node that contains information pertinent to placeholder stack traces. We use these in logging and error messages related to cudagraphs, and will cache these results. strname Optional[str] stack_traceList[PlaceholderInfo]usersmutating_use_stack_traceNrrrrr r s  I  ++rr cNeZdZUdZded<ded<ded<ded <d ed <ded <y )WrappedFunctionz Represents a function that you want to record for CUDA graph replay, with a little more metadata so we can identify if we have an applicable CUDA graph in our CUDA graph tree for it. zCallable[..., Any]model Sequence[int]static_input_idxsrrzTuple[torch.Tensor, ...] constantsSequence[PlaceholderInfo] placeholdersmutated_input_idxsNrrrrr)r),s, $$N''++%%rr)ct|jdk(r8tt|jjj ddS|jD]`}|j tjjjjk(s?|jj ddx}s^|cSy)Nr$) lenr&nextitermetagettargettorchopsatencopy_default)placeholder_nodeuser$s r&get_mutating_use_stack_trace_from_noder@<s  ! !"a'D)//0166::=$OO%%# ::--55 5!hhll=$??{?""# rc|jSN)r')placeholder_infos rget_mutating_use_stack_tracerDKs  4 44rc|j}|jjdd}g}d}|jdk(r-|jDcgc] }t |}}t |}t||||Scc}w)Nr$ placeholder)r"r6r7opr&to_placeholder_infor@r )r>r"r$r&r'is rrHrHOs  D"''++M4@K E#m+1A1G1GHA$Q'HH#I $   4e5M NN Is A7cr|jDcgc]}|jdk(st|c}Scc}w)NrF)nodesrGrH)graphnodes rget_placeholder_inforN^s4.3kk &*TWW =UD!  s4 4c d|S)Nzskipping cudagraphs due to r)reasons rformat_default_skip_messagerQds ( 11rcd}|D]}||}t|x}sntdt|d}|r|d|S|S)Nzmutated inputs (z instances). Found from : )rDrQr3)r/mutation_indicesr$idxrFmsgs rget_mutation_stack_tracerXhsp"$K"3' 6{C C; C  & 3/01= C( 66 Jrctjjjjr3|j Dcgc]}||j vs |||s|}}n |j }tjd|j tjd||rt|j|SdScc}w)Nz'check mutation static input indices: %sz#check mutation mutation indices: %s) r9 _inductorconfigtritoncudagraph_treesr0r,static_inputs_logdebugrXr/)funcinputsis_cuda_graph_recorded_tensorrVrUs rcheck_for_mutationrc{s  $$44..+ t---0= + +  22143I3IACST  !!2!24DE!+ s"B>cj|jD]$}|jjddx}s"|cSy)Nr$)r&r6r7)rMr?r$s r_get_use_stack_traceres:zz((,,}d; ;; ;  rc|jtjdx}r8d|jd}t |x}rt |d|St |St |dk(r0tt|jjdk(ryd|jD}t dd j|S) Ncpuz cpu device ()rTr2cudac32K|]}t|ywrB)repr).0keys r z:check_multiple_devices_or_any_cpu_nodes..sAscAszmultiple devices: z, ) r7r9devicer"rerQr3r4r5keystypejoin)device_node_mappingcpu_noderWr$ keys_reprs r'check_multiple_devices_or_any_cpu_nodesrvs'**5<<+>??x?X]]O1-.x8 8; 8.#6H /VW W*3//  A% )..01 2 7 76 AA&9&>&>&@AI &);DIIi?K $$44+, *<< ;@VC  +,1 +BHH5 ' 6FGG>889Q> 't8[8 s  B3B3c|jr |jS|jD]}|js|jcSy)zM Gets the first non-empty stack trace of a placeholder or its users. N)r$r&)rFusers rget_placeholder_stack_tracersH&&&!!$   ## #$ rc$eZdZdZdZdZdZddZy)CheckInvariantStatusr2c|jdk(ry|jdk(ry|jdk(ry|jd|jS)NCudagraphManagedIdxMismatchz-cudagraph managed tensor data pointer changedStaticInputIdxMismatchz!static input data pointer changed&ExpectedDeadIndicesBeforeGraphMismatchz+expected dead indices before graph are livez: )r"r)rs r__str__zCheckInvariantStatus.__str__sK 995 5B YY2 26 YYB B@ii[4::,/ /rN)returnr!)rrrSUCCESSrrrrrrrrrs$G#$./*0rrc t|t|k(rt|t|k(sJd|Dcgc]}|| }}|Dcgc]}|| }}|d}tt||D]t\}\} } t| tj sJ||} | j | k7s>|| } |d| jd| d| j dt| d }v|Scc}wcc}w)z} Logs the mismatch between input data pointers and recorded data pointers. This checks only idxs in target_idxs. zClength mismatch between inputs, recorded_data_ptr, and placeholdersz. z input name: z. data pointer changed from z to z. input stack trace:  ) r3 enumerateziprr9Tensordata_ptrr"r) r/rarecorded_data_ptr target_idxsmismatchrI t_tensors t_data_ptrs error_msgtensorrindexrFs rlog_data_ptr_mismatchrs1 v;#/0 0S[CE6MLM %00q0I01<=A$Q'=K=*C I!*3y++F!G  FH&%,,///A ??  (&u-K+\+*:*:);<--5Jd6??;L:MN&&A+&N%OrS   1=s C- C2c<t|jdzfd}tjjj j rLtjjj j kDrtj|yy)Nr2cddS)NzCUDAGraph supports dynamic shapes by recording a new graph for each distinct input size. Recording too many CUDAGraphs may lead to extra overhead. We have observed a0 distinct sizes. Please consider the following options for better performance: a) padding inputs to a few fixed number of shapes; or b) set torch._inductor.config.triton.cudagraph_skip_dynamic_graphs=True. Set torch._inductor.config.triton.cudagraph_dynamic_shape_warn_limit=None to silence this warning.r)num_cudagraphssrwarn_msgz4maybe_warning_due_to_dynamic_shape..warn_msg+s 00>/?@' ' rTF) r3rpr9rZr[r\"cudagraph_dynamic_shape_warn_limitr|r})fn_cache new_int_keyrrs @r"maybe_warning_due_to_dynamic_shaper%st)A-N   %%HH  // ' ' J J K hj) rc0eZdZUdZded<ded<ded<y) CudagraphCachedInfoz' Info needed to realign inputs r.r/zList[Optional[str]] stack_tracesz List[str]cudagraph_fail_reasonsNrrrrrrBs,+%%%%rr)r> torch.fx.Noderr#)rCr rr#)r>rrr )rLztorch.fx.Graphrr%)rPr!rr!)r/r.rUr+rr!)r`r)raList[InputType]rbzCallable[[torch.Tensor], bool]rr#)rr#)rs!Dict[torch.device, torch.fx.Node]rr#)rsr)rztorch.fx.GraphModuler,r+rr#)rFr rr#) r/r.rarrzSequence[Optional[int]]rr+rrrr!)rz)Dict[Tuple[int, ...], Callable[..., Any]]rrrbool)2 __future__r dataclassesenumrtypingrrrrr r r r r9torch._dynamo.utilsr torch._inductor.utilsr_logginggetArtifactLoggerrr|r^rr OutputType ModelType dataclassrr r)r@rDrHrNrQrXrcrervrxr~rrrrrrrrrrrsk"NNN (+00<H NN44 ' (5ell!234 5 d9o& 2 3 d#  $ d# , ,$ ,d# & &$ & #  5 O 2+?L&  $B >T:TT(H:H 1     99AN99. 0400+ /  #   >7 :d#&&$&r