P/jYddlZddlmZmZmZmZmZmZmZm Z m Z m Z ddl Z ddl mZddl mZddlmZdefdZde eeefee eeffde eeefee eeffdeeeeffd Zd ej*d ed ee eeffd eegefde e ed fe ed feeeefffdZd ej*de e ed fe ed feeeefffdZd ej*de e ed fe ed feeeefffdZ d2d ej*deedeededdf dZd ej*deeeefdeedeefdZ d2d ej*deedeededdf dZ d3dej*deedeedeedeedej*f dZdej*fd Zdej*fd!ZGd"d#ej*Z Gd$d%ej*Z! d2dej*d&ede e!e ed fffd'Z" d2dej*d&ede e e ed fe ed fffd(Z#d)e e ed fd fde ed ffd*Z$d+eej*de e e ed fe ed fffd,Z% d4d-e ej*d.e e de e&fd/e jNjPfd0Z) d4d-e ej*d.e e de e&fd/e jNjPfd1Z*y)5N) AnyCallableDictIterableListNoReturnSequenceTupleTypeUnion)Tensor)NamedMemberAccessorreturnctd)Na$make_functional(module): we don't yet support models that do parameter tying (also sometimes known as weight sharing). Please try to rewrite your model by replacing all instances of the tied parameter with another and/or comment your support in https://github.com/pytorch/functorch/issues/446) RuntimeErrorc/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/torch/_functorch/make_functional.pyraise_parameter_tying_errorr!s  : r named_paramstied_named_paramsct|}t|}t|j}t|j}|j|sJi}|j D] \}}|gf||<|j D]"\}}||vsJ||dj |$t|j S)a[ named_params is a dictionary of tensors: {'A': A, 'B': B} tied_named_params is another dictionary of tensors {'A': A, 'B': B, 'B_tied': B} with potentially tied (or 'duplicated') tensors This function creates a mapping from the names in named_params to the names in tied_named_params: {'A': ['A'], 'B': ['B', 'B_tied']}. )dictsetkeysissubsetitemsappendvalues)rrtensors_dict_keystied_tensors_dict_keystensor_to_mappingkeytensors rcreate_names_mapr&+s %L./L--/0 !2!7!7!9:  % %&< == ==?#))+. V%("I&!.(..01 V****&!!$++C01 !((* ++rmod named_members.subclasscNt|d}t|d}t||}i}t|}|D]@\}} | |vr |tj| d|| <|| } |j || Bt |dk(rd\} } n t|\} } | | |fS)NF)remove_duplicateTmeta)devicerrr)tupler&rtorch empty_like set_tensorlenzip) r'r(r)all_named_membersunique_named_members names_mapmemoaccessornamep replacementnamesparamss r_extract_membersr?Gs mUCD !EF !57HII D"3'H$/a D=u//&ABDG1g D+. /   A% v12 v 5) ##rcLt||jtjS)aZ This function removes all the Parameters from the model and return them as a tuple as well as their original attribute names. The weights must be re-loaded with `load_weights` before the model can be used again. Note that this function modifies the model in place and after this call, mod.parameters() will be empty. )r?named_parametersnn Parameterr's rextract_weightsrE`s C!5!5r|| DDrc2t||jdS)Nc|SNr)xs rz!extract_buffers..qsar)r? named_buffersrDs rextract_buffersrLns C!2!2K @@rr=r> as_paramsct|}|r"|Dcgc]}tj|}}|j||ycc}w)a  Reload a set of weights so that `mod` can be used again to perform a forward pass. Note that the `params` are regular Tensors (that can have history) and so are left as Tensors. This means that mod.parameters() will still be empty after this call. N)rrBrC set_tensors)r'r=r>rMr9r;s r load_weightsrPtsA#3'H+12a",,q/22 '3sAr7elemscg}t|}t|j|D]T\\}}}t|D]>\}} |dk(r"|j |j | |-|j | |@V|S)Nr)rr4r enumerater swap_tensorr2) r'r7rQresultr9_ attr_nameselemi attr_names r _swap_stater[sF"3'H!$Y__%6!>5J%j1 5LAyAv h229dCD##It4  55 Mrbuffersc>t|}|j||yrH)rrO)r'r=r\rMr9s r load_buffersr^s #3'H (rrmodelweights weight_names buffer_namesct|t|k(sJt|||t|dkDr&t|t|k(sJt||||S)zload_state(model, weights, weight_names, buffers=(), buffer_names=()) -> model load_state takes `weights` and `buffers` and assigns them to the model. This is the inverse operation of `make_functional_deprecated_v1`. r)r3rPr^)r_r`rar\rbs r load_staterds[ | G ,, , g. 7|a< CL000UL'2 Lrctj}t|dkDr tdt \}}fd}||fS)amake_functional_deprecated_v1(model) -> weights, func, weight_names Given an nn.Module, make_functional_deprecated_v1 extracts the state (weights) and returns a functional version of the model, `func`. This makes it so that it is possible use transforms over the parameters of `model`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, func, _ = make_functional_deprecated_v1(model) func(weights, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, _, func = make_functional_deprecated_v1(model) grad_weights = grad(func)(weights, (x,)) ``` To put the state back into a model, use `load_state`. rzmake_functional_deprecated_v1(model): `model` has buffers. Please use make_functional_with_buffers_deprecated_v1(model) instead.cRtj}t||||SrH)copydeepcopyrP)r`data mutable_model descriptorsr_s rfunz*make_functional_deprecated_v1..funs( e, ]K9d##r)listr\r3rrE)r_r\r`rVrlrks` @rmake_functional_deprecated_v1rns[45==?#G 7|a I  .e4G[!$ C $$rc`t\}}t\}}fd}|||fS)a`make_functional_with_buffers_deprecated_v1(model) -> weights, buffers, func, weight_names, buffer_names Given an nn.Module, make_functional_with_buffers_deprecated_v1 extracts the state (weights and buffers) and returns a functional version of the model, `func`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) grad_weights = grad(func)(weights, buffers, (x,)) ``` To put the state back into a model, use `load_state`. cltj}t||t||||SrH)rgrhrPr^)r`r\rirjbuf_descriptorsr_weight_descriptorss rrlz7make_functional_with_buffers_deprecated_v1..funs5 e, ]$6@]OW=d##r)rErL)r_r`rVr\rlrqrrs` @@r*make_functional_with_buffers_deprecated_v1rssA2&5U%;"G "1%"8G_a$ GS"4o EErc eZdZdZdej deedfdeedfdeee efdeee efdd f fd Z e dd ej d e dedee dfee dfffd Zdee dee defdZxZS)FunctionalModuleWithBufferszW This is the callable object returned by :func:`make_functional_with_buffers`. stateless_model param_names.rbparam_names_mapbuffer_names_maprNct|||_||_||_t ||_|j j|yrH)super__init__rvrwrbr all_names_mapupdate)selfrvrwrbrxry __class__s rr|z$FunctionalModuleWithBuffers.__init__sJ .&(!/2 !!"23rr_disable_autograd_trackingctj|}t|\}}}t|\}}}|r|D]} | j dt |||||||fSNF)rgrhrErLrequires_grad_ru) r_r model_copyr>rwrxr\rbryparams r _create_fromz(FunctionalModuleWithBuffers._create_froms} ]]5) />z/J, _2A*2M// $ ,$$U+ , (KHX      rr>r\c"t|j|jt|t|z} |j|i|t|j|j|S#t|j|j|wxYwrH)r[rvr}r/)rr>r\argskwargs old_states rforwardz#FunctionalModuleWithBuffers.forward(s~     &ME'N *  M'4''88 ,,d.@.@) LK,,d.@.@) Ls A++#BF__name__ __module__ __qualname____doc__rBModuler strrrr| staticmethodboolr rrrr __classcell__rs@rrurus4438_4CHo 4 c49n- 4 sDI~. 4 4 rwr7rs rrzFunctionalModule._create_fromHsW ]]5) )8)D& Y $ ,$$U+ , KCVKKrr>ct|j|j|} |j|i|t|j|j|S#t|j|j|wxYwrH)r[rvr7)rr>rrrs rrzFunctionalModule.forwardTsd 4 4dnnfM  I'4''88 ,,dnni HK,,dnni Hs A#A9rrrs@rrr8s # #38_ #T#Y' #  # func, params Given a ``torch.nn.Module``, :func:`make_functional` extracts the state (params) and returns a functional version of the model, ``func``. This makes it so that it is possible use transforms over the parameters of ``model``. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) func(params, x) And here is an example of applying the grad transform over the parameters of a model. .. code-block:: python import torch import torch.nn as nn from functorch import make_functional, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) def compute_loss(params, x, t): y = func(params, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, x, t) If the model has any buffers, please use :func:`make_functional_with_buffers` instead. Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. rzdmake_functional(model): `model` has buffers. Please use make_functional_with_buffers(model) instead.r)rmr\r3rrr)r_rr\s rmake_functionalr^sSv5==?#G 7|a ;    ( ( )B ) rc0tj||S)amake_functional_with_buffers(model, disable_autograd_tracking=False) -> func, params, buffers Given a ``torch.nn.Module``, make_functional_with_buffers extracts the state (params and buffers) and returns a functional version of the model ``func`` that can be invoked like a function. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) func(params, buffers, x) And here is an example of applying the grad transform over the parameters of a model: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) def compute_loss(params, buffers, x, t): y = func(params, buffers, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, buffers, x, t) Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. r)rur)r_rs rmake_functional_with_buffersrs#p ' 3 3 )B 4 rtuple_of_tuple_of_tensorscLtt|}td|D}|S)Nc3bK|]'}tj|j)ywrHr0stackdetach.0shardss r z"transpose_stack..s')/ F""$-/)r/r4)rresultss rtranspose_stackrs3!&c+D&E F3LG Nrmodelsczt|dk(r tdtd|Dstd|Ds tdt|dtfd|Ds tdt |Dcgc] }t |c}\}}}t |}t |}|d||fScc}w)a(combine_state_for_ensemble(models) -> func, params, buffers Prepares a list of torch.nn.Modules for ensembling with :func:`vmap`. Given a list of ``M`` ``nn.Modules`` of the same class, stacks all of their parameters and buffers together to make ``params`` and ``buffers``. Each parameter and buffer in the result will have an additional dimension of size ``M``. :func:`combine_state_for_ensemble` also returns ``func``, a functional version of one of the models in :attr:`models`. One cannot directly run ``func(params, buffers, *args, **kwargs)`` directly, you probably want to use ``vmap(func, ...)(params, buffers, *args, **kwargs)`` Here's an example of how to ensemble over a very simple model: .. code-block:: python num_models = 5 batch_size = 64 in_features, out_features = 3, 3 models = [torch.nn.Linear(in_features, out_features) for i in range(num_models)] data = torch.randn(batch_size, 3) fmodel, params, buffers = combine_state_for_ensemble(models) output = vmap(fmodel, (0, 0, None))(params, buffers, data) assert output.shape == (num_models, batch_size, out_features) .. warning:: All of the modules being stacked together must be the same (except for the values of their parameters/buffers). For example, they should be in the same mode (training vs eval). This API is subject to change -- we're investigating better ways to create ensembles and would love your feedback how to improve this. rz?combine_state_for_ensemble: Expected at least one model, got 0.c34K|]}|jywrHtrainingrms rrz-combine_state_for_ensemble..s+q +sc36K|]}|j ywrHrrs rrz-combine_state_for_ensemble..s2Raqzz>2RszTcombine_state_for_ensemble: Expected all models to have the same training/eval mode.c3:K|]}t|k(ywrH)type)rr model0_typs rrz-combine_state_for_ensemble..s5tAw*$5szHcombine_state_for_ensemble: Expected all models to be of the same class.)r3rallrr4rr)rr_funcsr>r\rs @rcombine_state_for_ensemblersP 6{a M   +F+ +s2R62R/R 0  fQiJ 5f5 5 V  !;A B% &u - BE67V $Fg&G 8VW $$ CsB8 model_classensemble_shaper-cfd}|S)Nct dk\r tdt dk(r" ij }t|S d}|dkrtd|dt  fdt |D}t i\}}}t d|D}t t |}t d|D}|||fS) N,NYI: ensemble_shape with more than 1 elementr num_models  should be > 0c3LK|]}ijywrHtorrVrr-rrs rrz3functional_init..wrapped..8+ 89K ( ( + +F 3 !$c38K|]}t|dyw)rN)rnrr_s rrz3functional_init..wrapped..<sTE5e.wrapped..>"K F+224Kr)r3 ValueErrorrrnr/ranger4) rrr_ num_modelsrrVfnr=r`r-rrs `` rwrappedz functional_init..wrapped.s ~ ! #KL L ~ ! #0033F;E07 7#A& ?{:,nEF F =B:=N  5[$5Q&5QR 2uTVTTW &K7KKE!!rrrrr-rs``` rfunctional_initr)s "& Nrcfd}|S)Nc t dk\r tdt dk(r" ij }t|S d}|dkrtd|dt  fdt |D}t  i\}}}}}tt d|D\} } t t| } t d| D} t t| } t d | D} | | |||fS) Nrrrrrc3LK|]}ijywrHrrs rrz@functional_init_with_buffers..wrapped..Srrc38K|]}t|ddyw)Nr)rsrs rrz@functional_init_with_buffers..wrapped..^s$;5A"1Erc3bK|]'}tj|j)ywrHrrs rrz@functional_init_with_buffers..wrapped..drrc3bK|]'}tj|j)ywrHrrs rrz@functional_init_with_buffers..wrapped..frr)r3rrrnr/rrsr4)rrr_rrrVrrarbr`r\r-rrs`` rrz-functional_init_with_buffers..wrappedIs* ~ ! #KL L ~ ! #0033F;E07 7#A& ?{:,nEF F =B:=N   7{D7SF7S T     #  W &K7KKW &K7KK\<??rrrs``` rfunctional_init_with_buffersrDs @@ Nrrr.)rcpu)+rgtypingrrrrrrr r r r r0torch.nnrBr %torch.nn.utils._named_member_accessorrrrr&rr?rErLrrPr[r^rdrnrsrurrrrrinttypesDevicerrrrrrs     EX,S&[)8E#v+4F+GGH,T#v+.sF{9K0LLM, #tCy.,8$ $C%V *$x'($ 5 uS#XS$s)^0D DE $2 E  E 5 uS#XS$s)^0D DE EA A 5 uS#XS$s)^0D DEA ( ( C=( V ( (  ("   #CcN3 C 99C15C U63;/ /0CN9>: 99:15: &fck(:E&#+