0j'\dZddlmZddlZddlmZd\ZZZ Z GddZ d d d Z d d Z y)aVisual layer: on flagged windows only, check whether a visible face is actually talking. MediaPipe FaceMesh (CPU) -> per-face mouth-opening ratio (inner-lip gap / face height). A talking face shows high variance in that ratio. Used to raise/lower severity of timing issues: - no face / tiny face -> off-screen speaker, timing error is less visible - large face, talking -> lip-sync error is very visible Also verifies the two videos are frame-identical (the dub must not alter the picture). ) annotationsN) sample_frames)  c eZdZdddZdddZy) MouthAnalyzercjddl}|jjjd|ddd|_y)NrFg?)static_image_mode max_num_facesrefine_landmarksmin_detection_confidencemin_tracking_confidence) mediapipe solutions face_meshFaceMeshmesh)self max_facesmps 2/root/repos/pocs/video-evals/src/dubeval/visual.py__init__zMouthAnalyzer.__init__s5LL**33e[dEJehLO4Q c vt|||||}i}i}d} |D]} |jj| } | js,| dz } t | jd} t | D]Z\} }|j }tj|Dcgc]}|jc}}tj|Dcgc]}|jc}}t|tj|tjz dz}|j| gjt|t j|t"jz |z t%|j'| dt)|j%|j+z |j%|j+z z|| <]|j-Dcic]3\}}t/|dk\s|t)tj0|5}}}|rt%||j&nd}||j'|dnd}|rt%|j3nd}|duxr|dk\}t5t/|| t7|d t7|d |r|d k\d Sdd Scc}wcc}wcc}}w) Nrrc4|jdjS)Nr)landmarkx)ms rz'MouthAnalyzer.analyze..%s1::a=??r)keygư>ggMbp?{Gz?)frames_analyzed face_frames max_face_areamouth_activityspeaking_likely)rrprocessmulti_face_landmarkssorted enumeraternparrayr yabsCHINFOREHEAD setdefaultappend LOWER_LIP UPPER_LIPmaxgetfloatminitemslenstdvaluesintround)rvideostartendfpswidthframestracksareasr(fresfacesslotlmpqxsysface_hsvactivitybestmouthmax_areausables ranalyzezMouthAnalyzer.analyzes^ueS#u=)+"$  nA))##A&C++ 1 K3339RSE%e, nbKKXXA.qqss./XXA.qqss./QtWYY867$>!!$+223q|~~) 7V3WZ`3`a!%))D#"6rvvx"&&(?RWYW]W]W_bdbhbhbjWj>k8lmd  n n5;LLNRDAqc!fPQkAuRVVAY''RR2:s8.+/+; T3'*/3u||~&ST!7h%&7"3v;/&"8Q/#E1o28   ?C   /.SsJ+ ?J0 J57!J5N)r%)rrB)g @i) rEr<rFr<rGr<rHrBreturndict)__name__ __module__ __qualname__rr]rrr r s Q rr cg}tjdtd|dz |D]}t|t |t |dzd|}t|t |t |dzd|}t |r t |sct tj |djt|djtz j} | dkDs|jdddd tt |d tt |dzd d | d d ddt| did |S)z[Compare sparse frames; any large pixel difference means the picture was changed or shifted.g?g{Gz?r VIDEO_MISMATCHrDmajorgffffff?z;Video frame differs from the original (mean abs pixel diff z.1fz).zYDub must reuse the original video stream untouched (stream copy); check re-encode/offset. mean_abs_diffr) categorysourceseverity confidencerErF descriptionfix_hintevidence) r0linspacer:rr<r?r3astyperBmeanr7rC) origdubdurationnrHissuestabdiffs rvideo_identityr~;s  F [[c#x#~6 :  $a%(T/2u E #uQxqD"e D1vSV RVVAaDKK,qt{{3/??@EEGH "9 MM,WdguQx+E%(T/14M!\]abe\ffhiw,eD!n=    Mrczgd}|j|d}|d}|d}|!|dvrtd|dz }|d xxd z cc<nn|rH|d vrD|d d k\rtd|dz}|d xxdz cc<ttd|ddzd|d<n$|dur |dvrtd|dz}|d xxdz cc<|||d<y)z9Re-weight a timing issue with visual evidence (in place).)minorrhcriticalrmrkr+N) EARLY_CUTOFF MID_LINE_GAP LATE_STARTOVERRUN EARLY_STARTrrrozF No clear face on screen (likely off-screen speaker), so less visible.)rrr MISSING_DUBr)r&rizH A visible face is moving its mouth during this window (lip-sync break).g\(\?rng?F)rUNMATCHED_SPEECHz8 On-screen face is not talking while dubbed voice plays.)indexr:r=rC)issuevisordersevcattalkings radjustrOs *E ++eJ' (C  C#$G Z Zaq/C - $l l SYY  4 'aq/C m jj#CeL.AC.G$H!Ll E c%DD!S1Wo m ZZc E*r)rf)rwr<rxrBrHrBr^z list[dict])rr_rr_r^None)__doc__ __future__rnumpyr0mediarr9r8r5r4r r~rrcrrrs8# '6$ 9h& & R(#r