P/j ddlmZmZddlmZmZddlZddlmZ ddlm Z ddlm Z GddeZ Gdd e Z dd e d eed efd ZGddeZdZdZGddeZy))ABCabstractmethod)DictListN)Tensor) TokenSpanc8eZdZedeedeeefdZy) ITokenizer transcriptreturncy)a Tokenize the given transcript (list of word) .. note:: The toranscript must be normalized. Args: transcript (list of str): Transcript (list of word). Returns: (list of int): List of token sequences N)selfr s i/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/torchaudio/pipelines/_wav2vec2/aligner.py__call__zITokenizer.__call__ N)__name__ __module__ __qualname__rrstrrrrrr r s.  49  d3i    rr cDeZdZdeeeffdZdeedeeefdZy) Tokenizer dictionaryc||_yNr)rrs r__init__zTokenizer.__init__s $rr r c r|Dcgc]}|Dcgc]}|j|c}!c}}Scc}wcc}}wrr)rr wordcs rrzTokenizer.__call__s-?IJtT2#2JJ2Js 3. 33N) rrrrrintrrrrrrrrs8%4S>%K49Kd3iKrremissiontokensblankc|j}|jd}tj|gtj|}t j |||\}}|j}|d|d}}||fS)Nr)dtypedevicer%)r( unsqueezetorchtensorint32F forced_alignexp)r#r$r%r(targetsaligned_tokensscoress r_align_emission_and_tokensr4#sr __F!!!$HllF85;;vFG^^HgUKNF ZZ\F+A.q FN 6 !!rc BeZdZededeeedeeefdZy)IAlignerr#r$r cy)aGenerate list of time-stamped token sequences Args: emission (Tensor): Sequence of token probability distributions in log-domain. Shape: `(time, tokens)`. tokens (list of integer sequence): Tokenized transcript. Output from :py:class:`torchaudio.pipelines.Wav2Vec2FABundle.Tokenizer`. Returns: (list of TokenSpan sequence): Tokens with time stamps and scores. Nr)rr#r$s rrzIAligner.__call__0rrN) rrrrrrr"rrrrrr6r6/s:    d3i  T$y/EZ    rr6ct|t|k(sJd}g}|D]}|j||||z||z } |S)Nr)lensumappend)list_lengthsiretls r _unflattenrA?sX u:W %% % A C  5QU#$ Q Jrc@|Dcgc] }|D]}| c}}Scc}}wrr) nested_listr<items r_flattenrEIs") eZdZdZdedeeedeeefdZy)Alignerc||_yrr))rr%s rrzAligner.__init__Ns  rr#r$r c |jdk7rtd|jt|t ||j \}}t j||}t||Dcgc] }t|c}Scc}w)Nz&The input emission must be 2D. Found: ) ndim ValueErrorshaper4rEr%r. merge_tokensrAr9)rr#r$r2r3spanstss rrzAligner.__call__Qst ==A EhnnEUVW W!;HhvFVX\XbXb!c~v6%F!;b#b'!;<rUsq# !+  "K K " "c "3 "  s  = =h =r