P/jVddlZddlmZmZddlZddlmZddlmZgZ ddej dej dej de d e d e d e d ej fd Z GddejjZGddejjZGddejjZGddejjZy)N)OptionalUnion)Tensor) functionalpsd_spsd_nreference_vectorsolutiondiagonal_loadingdiag_epsepsreturnc|dk(rtj||||||}|S|dk(rtj|}ntj|||||}tj||||||}|S)aCompute the MVDR beamforming weights with ``solution`` argument. Args: psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. reference_vector (torch.Tensor): one-hot reference channel matrix. solution (str, optional): Solution to compute the MVDR beamforming weights. Options: [``ref_channel``, ``stv_evd``, ``stv_power``]. (Default: ``ref_channel``) diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: the mvdr beamforming weight matrix ref_channelstv_evd)r r )Fmvdr_weights_soudenrtf_evd rtf_powermvdr_weights_rtf) rrr r r r r beamform_vectorstvs g/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/torchaudio/transforms/_multi_channel.py_get_mvdr_vectorrs:= //u>NP`bjlop  y ))E"C++eU, #(;;q>  We, We, \:6 \:6rrrmask_smask_nr r r rc D|jr$|jd}|jd}|jjdk(rM||_||_|j d|_|j d|_t||||||| S|j||}|j||}||_||_|j |j dz|_|j|j dz|_t||||||| S)aRecursively update the MVDR beamforming vector. Args: psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. mask_s (torch.Tensor): Time-Frequency mask of the target speech. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. mask_n (torch.Tensor or None, optional): Time-Frequency mask of the noise. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. reference_vector (torch.Tensor): One-hot reference channel matrix. solution (str, optional): Solution to compute the MVDR beamforming weights. Options: [``ref_channel``, ``stv_evd``, ``stv_power``]. (Default: ``ref_channel``) diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: The MVDR beamforming weight matrix. r)r*r>) rr,rndimrsumr?r@r_get_updated_psd_speech_get_updated_psd_noise) r$rrrErFr r r r r s r_get_updated_mvdr_vectorzMVDR._get_updated_mvdr_vectors L ??[[R[(F[[R[(F ::??a DJDJ$jjRj0DO$jjRj0DO#E52BHN^`hjmn n00?E//v>EDJDJ"oo r 0BBDO"oo r 0BBDO#E52BHN^`hjmn nrc|j|j|jdzz }d|j|jdzz }|j|dz||dzz}|S)aUpdate psd of speech recursively. Args: psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. mask_s (torch.Tensor): Time-Frequency mask of the target speech. Tensor with dimensions `(..., freq, time)`. Returns: torch.Tensor: The updated PSD matrix of target speech. rHr*r>.NN)r?rJr)r$rrE numerator denominators rrKzMVDR._get_updated_psd_speechkOOt9K'KL 4??VZZBZ-??@  Y77%+oB^:^^ rc|j|j|jdzz }d|j|jdzz }|j|dz||dzz}|S)aUpdate psd of noise recursively. Args: psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. mask_n (torch.Tensor or None, optional): Time-Frequency mask of the noise. Tensor with dimensions `(..., freq, time)`. Returns: torch.Tensor: The updated PSD matrix of noise. rHr*r>rO)r@rJr)r$rrFrPrQs rrLzMVDR._get_updated_psd_noiserRrr&c |j}|jdkrtd|j|j std|j|jt j k(r|j}|tjdd|z }|j||}|j||}t j|jdd|jt j}|d |jfjd|j r7|j#||||||j$|j&|j(}n.t+||||j$|j&|j(}t-j.||} | j1|S) a`Perform MVDR beamforming. Args: specgram (torch.Tensor): Multi-channel complex-valued spectrum. Tensor with dimensions `(..., channel, freq, time)` mask_s (torch.Tensor): Time-Frequency mask of target speech. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. mask_n (torch.Tensor or None, optional): Time-Frequency mask of noise. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. (Default: None) Returns: torch.Tensor: Single-channel complex-valued enhanced spectrum with dimensions `(..., freq, time)`. z?Expected at least 3D tensor (..., channel, freq, time). Found: ziThe type of ``specgram`` tensor must be ``torch.cfloat`` or ``torch.cdouble``. Found: Nz=``mask_n`` is not provided, use ``1 - mask_s`` as ``mask_n``.r>)devicedtype.)rXrIrAshape is_complexr5cfloatcdoublewarningswarnr-rCsizerWrfill_r;rMr r:r rrapply_beamformingto) r$r&rErFrXrruw_mvdrspecgram_enhanceds rr.z MVDR.forward#s& ==1 ^_g_m_m^nop p""$$NN+-  >>U\\ )'')H > MMY ZZF6*6* KK ,X__EMM Z #t  &&q) ;;22uffa@Q@QSWS`S`F&eUAt}}dFWFWY]YfYfgF//A ##E**r)rrFTHz>FrTrf:0yE>r!)r/r0r1r2intstrr3r4r#r5rrMrKrLrr.r6r7s@rr9r9asAJ% !!7!7!7 !7  !7  !7!7T&!%7o||7o||7o 7o  7o  ,, 7o7o7o7o7o 7orU\\5<<TYT`T`"ELL%,,SXS_S_$^b1+ 1+.3ll1+DLU\\DZ1+ 1+rr9cJeZdZdZ d dedededeeefdededed efd Z y ) RTFMVDRaMinimum Variance Distortionless Response (*MVDR* :cite:`capon1969high`) module based on the relative transfer function (RTF) and power spectral density (PSD) matrix of noise. .. devices:: CPU CUDA .. properties:: Autograd TorchScript Given the multi-channel complex-valued spectrum :math:`\textbf{Y}`, the relative transfer function (RTF) matrix or the steering vector of target speech :math:`\bm{v}`, the PSD matrix of noise :math:`\bf{\Phi}_{\textbf{NN}}`, and a one-hot vector that represents the reference channel :math:`\bf{u}`, the module computes the single-channel complex-valued spectrum of the enhanced speech :math:`\hat{\textbf{S}}`. The formula is defined as: .. math:: \hat{\textbf{S}}(f) = \textbf{w}_{\text{bf}}(f)^{\mathsf{H}} \textbf{Y}(f) where :math:`\textbf{w}_{\text{bf}}(f)` is the MVDR beamforming weight for the :math:`f`-th frequency bin, :math:`(.)^{\mathsf{H}}` denotes the Hermitian Conjugate operation. The beamforming weight is computed by: .. math:: \textbf{w}_{\text{MVDR}}(f) = \frac{{{\bf{\Phi}_{\textbf{NN}}^{-1}}(f){\bm{v}}(f)}} {{\bm{v}^{\mathsf{H}}}(f){\bf{\Phi}_{\textbf{NN}}^{-1}}(f){\bm{v}}(f)} r&rtfrreference_channelr r r rcftj||||||}tj||} | S)a Args: specgram (torch.Tensor): Multi-channel complex-valued spectrum. Tensor with dimensions `(..., channel, freq, time)` rtf (torch.Tensor): The complex-valued RTF vector of target speech. Tensor with dimensions `(..., freq, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. reference_channel (int or torch.Tensor): Specifies the reference channel. If the dtype is ``int``, it represents the reference channel index. If the dtype is ``torch.Tensor``, its shape is `(..., channel)`, where the ``channel`` dimension is one-hot. diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: Single-channel complex-valued enhanced spectrum with dimensions `(..., freq, time)`. )rrra) r$r&rmrrnr r r rdspectrum_enhanceds rr.zRTFMVDR.forwardrs<@##C0ACSU]_bc//A  rNTrfrh) r/r0r1r2rrrir3r4r.rrrlrlWso@"&"!"!"! "! !f- "!  "!"!"! "!rrlc^eZdZdZ d dedededeeefdededed e jfd Z y ) SoudenMVDRaMinimum Variance Distortionless Response (*MVDR* :cite:`capon1969high`) module based on the method proposed by *Souden et, al.* :cite:`souden2009optimal`. .. devices:: CPU CUDA .. properties:: Autograd TorchScript Given the multi-channel complex-valued spectrum :math:`\textbf{Y}`, the power spectral density (PSD) matrix of target speech :math:`\bf{\Phi}_{\textbf{SS}}`, the PSD matrix of noise :math:`\bf{\Phi}_{\textbf{NN}}`, and a one-hot vector that represents the reference channel :math:`\bf{u}`, the module computes the single-channel complex-valued spectrum of the enhanced speech :math:`\hat{\textbf{S}}`. The formula is defined as: .. math:: \hat{\textbf{S}}(f) = \textbf{w}_{\text{bf}}(f)^{\mathsf{H}} \textbf{Y}(f) where :math:`\textbf{w}_{\text{bf}}(f)` is the MVDR beamforming weight for the :math:`f`-th frequency bin. The beamforming weight is computed by: .. math:: \textbf{w}_{\text{MVDR}}(f) = \frac{{{\bf{\Phi}_{\textbf{NN}}^{-1}}(f){\bf{\Phi}_{\textbf{SS}}}}(f)} {\text{Trace}({{{\bf{\Phi}_{\textbf{NN}}^{-1}}(f) \bf{\Phi}_{\textbf{SS}}}(f))}}\bm{u} r&rrrnr r r rcftj||||||}tj||} | S)a Args: specgram (torch.Tensor): Multi-channel complex-valued spectrum. Tensor with dimensions `(..., channel, freq, time)`. psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. reference_channel (int or torch.Tensor): Specifies the reference channel. If the dtype is ``int``, it represents the reference channel index. If the dtype is ``torch.Tensor``, its shape is `(..., channel)`, where the ``channel`` dimension is one-hot. diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: Single-channel complex-valued enhanced spectrum with dimensions `(..., freq, time)`. )rrra) r$r&rrrnr r r rdrps rr.zSoudenMVDR.forwards<@&&ue5FHXZbdgh//A  rNrq) r/r0r1r2rrrir3r4r5r.rrrrrtrtsr>"&"!"!"! "! !f- "!  "!"!"! "!rrtrg)r]typingrrr5r torchaudiorr__all__rjr3r4rnnModulerr9rlrtrrrrr{s" & "!& <<& <<&ll& &  &  & & \\&R'%((//'Ts+588??s+l=!ehhoo=!@<!<!r