0j(dZddlmZddlmZmZddlZddlm Z dZ eGdd Z eGd d Z dd Z dd Z ddZddZ ddZ ddZy)aDeterministic timing layer: align original utterances to dubbed speech and flag gaps/lag. Both tracks share the same video timeline, so no global alignment is needed: a correct dub puts speech where the original actor speaks. Every decision here is threshold-based and reproducible. ) annotations) dataclassfieldN)maskg{Gz?ceZdZUdZded<dZded<dZded<dZded <dZded <d Z ded <d Z ded<dZ ded<dZ ded<d Z ded<dZded<dZded<dZded<y) Thresholdsgffffff?floatline_merge_gap?min_line? match_window?missing_coverage late_start333333? early_startffffff? early_cutoff皙?overrunmid_gap unmatched_minint drift_window皙?driftg? fast_ratioN)__name__ __module__ __qualname__r __annotations__r rrrrrrrrrrr 2/root/repos/pocs/video-evals/src/dubeval/timing.pyr r s NE HeL%"e"JKL%GUGUM5L#E5Jr&r ceZdZUded<ded<ded<eeZded<d Zded <edd Z edd Z edd Z eddZ y)Linerindexr startend)default_factorylist[tuple[float, float]]dubcoveragec4|j|jz SN)r,r+selfs r'durzLine.dur*sxx$**$$r&cV|jrtd|jDSdS)Nc3&K|] \}}| ywr3r%).0s_s r' z!Line.dub_start..0*A1*)r/minr4s r' dub_startzLine.dub_start.".2hhs***@D@r&cV|jrtd|jDSdS)Nc3&K|] \}}| ywr3r%)r9r;es r'r<zLine.dub_end..4r=r>)r/maxr4s r'dub_endz Line.dub_end2rAr&cP|jsdS|j|jz Sr3)r/r@r+r4s r'onsetz Line.onset6s 88tD$**)DDr&N)returnr )rIz float | None) r!r"r#r$rlistr/r1propertyr6r@rFrHr%r&r'r)r)"s J L J%*4%@C "@He %%AAAAEEr&r)c `tdt|d|dt|d|dz S)Nr0rr)rEr?)abs r'_overlaprO;s0 sC!adOc!A$!o5 66r&c tjtjdg|jtjdgg}t t tj|dk(tj|dk(S)z([start, end) index runs where m is True.rr)npdiff concatenateastypeint8rJzip flatnonzero)mds r'_runsr[?s] QXXbgg%6<=>A BNN16*BNN17,CD EEr&c t|Dcgc]}|d|dz |jk\s|c}Dcgc]\}\}}t|||}}}}t|D] \}} || _g} t j |D cgc]} | j c} } |D] } d\} }t j| | ddz }||dD]} | j | d|jzkDrnt| | j | jf}|dkDrdn=tt| d| jz t| j | dz }|dkDr|n| }||jks| ||kDs| |}} | | j| | jj| t||t }|D]u} t#| j t z t#t j$| jt z }}||kDrt'|||j)nd| _w|| fScc}wcc}}}wcc} w)z]Assign each dub segment to the original line it overlaps most (or the nearest within window).rr)Nr0Nr0) enumerater r)r*rRarrayr+ searchsortedrrOr,r?absappendr/rHOPrceilr meanr1)origr/durationtxir:rDlinesln unmatchedstartssegbestbest_ovloovgapscoredmrMrNs r'alignrwEs#-6$6dQ!A$QRSTQU+YZYcYcJcq6d,e f fyq&1aT!Q] fE f5!2I XX%0Brxx0 1F!! g __VSVb[ 1* *Bxx#a&1>>11#"&&12Ba#SSVbff_)=s288cRSfCT?U%VCq&BsdEann$$,%'/ "Eg * <   S ! HHOOC ! c8S !B>288c>"C (=$>1/01ueBqGLLN+# > ) 17e f1sH>H>II c"||k\rdS||k\rdSdS)Ncriticalmajorminorr%)valuerzrys r'_sevr}cs(*:V5E>VwVr&c$g$t||t}t||t}dV$fd } |D]} t| jdt| jd| j dnt| j d| j dnt| j dt| jdd} | j|jkrt| jdk\re| dt| jddd| jz | j| jd | jd d | jd d d| fi| | js'| j} | |jkDrV| dt| ddtdd| dz z| j| j d| ddd| dd| fdt| di| ni| |j krY| dt| ddtdd| dz z| j | jd| ddd| dd| fdt| di| | j| j z } | |j kDrJ| d t| ddtdd| d!z z| j | jd"| dd#d$| dd%| fi| n| |j"kDrt%|t'| jtz t'| j tz j)}| d&t| ddtdd| d!z z| j| j d'| dd(|rd)nd*zd+| dd| fd,|i| t't+| j| j tz }t't| j| j tz }||kDst-||||||zD]_\}}||z tz}||j.k\s"||ztz}| d-t|ddtdd.|d!z z|||z|dd/d0| fi| a |D];\}}||z |j0k\s| d1t||z d2d3d4||||z dd5d6=|D cgc]7} | j| j|jk\s*| | jf9}} |j2}d7}||zt5|kr*t7t9j:||||zDcgc]\}}| c}}}t=||j>k\r||z}|t5|krt=t7t9j:|||z d8z|d8zDcgc]\}}| c}}|j>k\rg|d8z }|t5|krTt=t7t9j:|||z d8z|d8zDcgc]\}}| c}}|j>k\rg|||}t7t9j:|Dcgc]\}}| c}}}| d9tt=|d:dd;|d7d7j|d<d7jd=|d7kDrd>nd?d@t=|ddAt5|dBdC| dDdE|d7d7j@dF|d<d7j@dGt|d|D cgc]\} }| j@c}} H |}n|d8z }||zt5|kr*|rLg}|D]} | js|D cgc]3} | j | dI| dJzdz cxkr| j ks/nn| 5}!} | j | j z }"t5|!dk\sv|"dkDs||jC| tEdK|!D|"z f|rt7t9j:|D#cgc]\}}#|# c}#}}|D]l\} }#|#||jFzkDs| dL|#|dzkrdMndNdO| j | j dP|#d dQ|d dR|#|z d dSdT| t|#d8t|d8U n$Scc} wcc}}wcc}}wcc}}wcc}}wcc}} wcc} wcc}#}w)WNc  j|d|t|dt|dt|d|||dn |j|d y)Ntiming) categorysourceseverity confidencer+r, descriptionfix_hint line_indexevidence)rbroundr*) catsevconfr:rDdeschintlineevissuess r'addzdetect..addmsO 3([`aegh[i %a E!QKPTbf-1\TtzzWY[ \r&r) orig_startorig_endr@rFr1r MISSING_DUBg333333?g@g?zOriginal speech of z.1fz s has only z.0%z dubbed speech under it.zTGenerate/insert the dubbed line for this window; it was dropped or placed elsewhere. LATE_STARTrg@gffffff?r z Dub starts z.2fz+s after the original actor begins speaking.z#Shift this dubbed line earlier by ~zs (or trim leading silence). onset_delta EARLY_STARTrz#s before the original actor speaks.zDelay this dubbed line by ~zs. EARLY_CUTOFFz Dub ends zAs before the original line ends (actor still speaking, no voice).z3Slow this line down or extend translation to fill ~zs more.OVERRUNzDub keeps talking zs after the original line endedz and runs into the next line.z (over silence/closed mouth).z(Speed up or shorten the translation by ~spills_into_next MID_LINE_GAPg?z:s silence in the dub while the original actor is speaking.zJRe-time the segments around this pause or fill it with the missing phrase.UNMATCHED_SPEECHg?g@rz;s of dubbed speech where the original has no speech nearby.zYCheck if this line belongs to another timestamp or is a hallucinated/duplicated TTS line.rr SYNC_DRIFTrg?rQzDub is consistently lateearlyz by ~zs over z lines.zApply a z+.2fzs offset to dubbed lines -.)median_onset_deltarkr+r,c38K|]}t|dyw)wordN)len)r9ris r'r<zdetect..s%Ac!F)n%As FAST_SPEECHr{rzg?z Dub speaks z chars/s vs a median of z (xz); likely time-squeezed.z9Use a shorter translation instead of speeding up the TTS.) chars_per_secmedian_chars_per_secr3)$rrcrr+r,r@rFr1rr6r}r/rHrr?rrrboolranyrEr[rrrrr rRmedianrarr*rbsumr )%rkrmr/rfrg dub_wordsrhrvomrrlrrHtailspill_into_nextrMrNr0r1gr:rDonsetswrjr;omedjrorYratesriwsspanrrs% @r'detectrgsF c8S !B dHc "B\ ,p!"((A.E"&&!R3AQS(93sD1q5L;QSTVWZ[V[S'![\dfhplnp  pM,p\m1 q5AOO # "DQS$931q5+XYk mm(- k0DXYXjXjIjr288n kF k A A a%3v; BIIVAa!e_=TQq=>? s8qww AAc&k/c% PQTUPUXYPYZ[^_Z_I`:aA1:a0b*c&dhihoho&oQc&k/c% PQTUPUXYPYZ[^_Z_I`:aA1:a0b*c&dhihoho&o1+Cbiis 3tq! 345A  d3q634dCF1IOOSQSWUVZ^^&QvG&DE#a&QTU\]`ad]e\ffmnA2d)#;b:a!47Rd #8sB<aa(a9 a' a  a& a, a20a8a8 a=ct||tt||t}}|Dcgc]+}|j|jdk\s |j-}}t |j tzdt |j tzdt t ||zj td|j z dt t ||zj td|j z d|r(t t tj|dnd|r>t t tjtj|dddSddScc}w)NrrrZ)orig_speech_secondsdub_speech_secondsspeech_overlap_ratiounmatched_dub_ratiorp90_abs_onset_delta) rrcrHr1rrr rErRr percentilera)rkrfr/rgrrvrlrs r'statsrs. $# &S(C(@B!& W2"((*>2;;RVCVbhh WF W$RVVX^Q7#BFFHsNA6 %eR"WMMOc!RVVX>N,N&OQR S$UB"H>>+;c!RVVX>N+N%OQRSDJeE"))F*;$rs #(        EE E07F  H<We*e6;eHReWaefpeP  " r&