/j] dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZd d lmZej$eZej*d Zej*d d j/eej2dZdZdZdZdZdZdZ ej*ejBedZ"de#e zde fdZ$de#e zde#de fdZ%de#e zde#de fdZ&de de'fdZ(de de'fdZ)de d e*dzde'fd!Z+de#e zde'fd"Z,d#e de'fd$Z-de#e zde'fd%Z.de#e zde#de dzfd&Z/de#e zde*fd'Z0de#e zde*fd(Z1de de#e zddfd)Z2d*e d+e de#e zddfd,Z3de#e zde fd-Z4d*e#e zde#e zde#dzfd.Z5d/d0d*e#e zde#e zd1e'de dzfd2Z6d*e#e zde#e zde dzfd3Z7de de fd4Z8de de fd5Z9ede de d6fd7Z:d8e d9e#ddfd:Z;d*e de de fd;Zde#de#e zd e*dzde'fd>Z?d/d?d*e#de#de#e zd e*dzd@e'de'f dAZ@de de#e zdeAe dzfdBZBde dCeAe de#e zde*fdDZCde#e zdeAe fdEZDde#e zdeEe e*ffdFZFd8e dGeAe de#e zddfdHZGde de#e zde*fdIZHy)JajCache-wide shared blob store, deduplicating Xet files across cached repos. Store entries at `/blobs/<2-hex-prefix>/` are regular files. Per-repo `blobs/` entries are relative symlinks to them. A companion `.refs` manifest lists those symlinks so cache deletion only needs to inspect the blobs it touches instead of scanning the entire store. The manifest is a hint, not trusted metadata: every entry is validated against the filesystem before use, and any missing or unreadable manifest makes garbage collection leak the shared blob rather than risk deleting referenced content. N)contextmanager)Path) Generator)FileLock SoftFileLock) constants)loggingz [0-9a-f]{64}z(?:|z)--.+blobsz.huggingface-shared-blobs1z.refsz.lock z\.[0-9a-f]{8}\.tmp cache_dirreturnc&t|tz S)zBReturn the path of the shared blob store inside a cache directory.)rSHARED_BLOBS_DIR_NAME)rs f/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_shared_blobs.pyshared_blobs_dirr7s  ?2 22xet_hashcrtj|td|dt||ddz |z S)zuReturn the store path for a given Xet file hash. Raises `ValueError` if `xet_hash` is not a valid Xet hash. NzInvalid Xet file hash: ''.r)_XET_HASH_REGEX fullmatch ValueErrorrrrs rshared_blob_pathr<sE   *23H:R@AA I &"1 5 @@rcHt||j|tS)z=Return the reference manifest path for a given Xet file hash.)r with_name_MANIFEST_SUFFIXrs rshared_blob_manifest_pathr"Fs% Ix 0 : :hZHXGY;Z [[rpathc| tj|jjS#t$rYywxYwNF)statS_ISREGlstatst_modeOSErrorr#s r_is_regular_filer,K4||DJJL0011  ,/ ;;c| tj|jjS#t$rYywxYwr%)r&S_ISDIRr(r)r*r+s r _is_directoryr1Sr-r. store_path expected_sizec x|y |j}tj|jsy|j |k7r*t jd|d|j d|dytj|tjst jd|dyy#t$rYywxYw)zPReturn whether `store_path` is a regular, readable payload of the expected size.Fz Shared blob 'z' has an unexpected size (z instead of z). Not using it.z ' is not readable. Not using it.T) r(r*r&r'r)st_sizeloggerwarningosaccessR_OK)r2r3 store_stats r_is_usable_store_entryr<[s%%'  << ** +]*J<'A*BTBTAUUaboapq   99Z )zl2RST  sB-- B98B9ct|}|tz }t|r t|sy |j t dk(S#t $rYywxYw)z?Return whether `path` is an owned, supported shared blob store.F )rSHARED_BLOBS_MARKER_NAMEr1r, read_textSHARED_BLOBS_LAYOUT_VERSIONr*)r# store_dir marker_paths ris_shared_blobs_dirrDrs]T I66K  #+;K+H$$&-H,I*LLL sA AArBctd}t|j}|D]P}tj |j  t |sy |j|jsyR|D]}|jdt|j S#ttf$rYywxYw)zRemove leftover marker temporaries. Returns whether the directory is empty afterwards, i.e. safe to mark. Any foreign content returns False. r>FT missing_ok) rAlistiterdir_MARKER_TMP_REGEXrnamer, startswithr@r* UnicodeErrorunlinkany)rBexpected_contententriesentrys r_cleanup_abandoned_marker_tempsrS~s 66b99$$&'G  & &uzz 2 :BRSXBY #..u/@AB &  %&9$$&' '' &  sB++B>=B>ct|}|tz } |jdt|ryt |r t |st jd|dyt|||j|jdtjdd} |jtd |jt!|t#j$|||j'd t|S#|j'd wxYw#t($r%}t jd |d |Yd }~yd }~wwxYw)zJCreate and mark the shared store, refusing to adopt an unmarked directory.Texist_okz0Refusing to use unmarked shared blob directory 'rF..tmpr>rFz,Could not initialize shared blob directory '': N)rr?mkdirrDr1rSr6debug_repair_shared_directory_moder rKsecrets token_hex write_textrAchmod_shared_blob_moder8replacerNr*)rrBrC tmp_markeres r_ensure_shared_blobs_dirrfs8 +I66K& y )Y'/Ny/Y LLKI;VXY Z%i; **k.>.>-?qARARSTAU@VVZ+[\  /  ! !%@$A"D E   .y9 : JJz; /     ."9--     .  CI;cRSQTUVs<D!/D!$AD!%AD -D! DD!! E*E  Ect|syt||j} |jdt|st j d|dy t|||S#t$r%}t j d|d|Yd}~yd}~wwxYw#t$r%}t j d|d|Yd}~yd}~wwxYw)NTrUz/Could not create shared blob prefix directory 'rZz2Refusing to use non-directory shared blob prefix 'rz1Could not set shared blob prefix permissions on ') rfrparentr[r*r6r\r1r])rr prefix_dirres r_ensure_prefix_dirrjs #I .!)X6==J$'  $ I*UWXY%j)<   FzlRUVWUXYZ  H TWXYWZ[\s/A* B* B3BB C $CC ctjdk(ry tjt |jj }d|tjzrdndz|tjzrdzSdzS#t $rYywxYw)zLReturn a read-only mode accessible to users who can traverse the cache root.nt rrX) r8rKr&S_IMODErr)r*S_IXGRPS_IXOTHr cache_modes rrbrbs ww$\\$y/"6"6"8"@"@A  Z$,,6EA >:X\XdXdKd% lljk ll s6B B  B c tjt|jj}|dtj ztj zzS#t$rYywxYw)zQMirror cache-root access and inheritance bits on newly created store directories.ii)r&rprr)r*S_ISGIDS_ISVTXrss r_shared_directory_moderxs[\\$y/"6"6"8"@"@A  - < == s6A A+*A+ct|}tj|jj|k7r|j |yyN)rxr&rpr(r)ra)r#r expected_modes rr]r]s:*95M ||DJJL(()]: =!;r blob_pathrict|}tj||tjdk(st tdsy|j j }|j j |k(ry tj|d|y#t$r,|t jzr|t jzsYyYywxYw)zKMake a payload immutable and readable according to the shared cache policy.rlchownN) rbr8rarKhasattrr&st_gidr~r*S_IRGRPS_IROTH)r|rir blob_mode target_gids r _prepare_shared_blob_permissionsrs!),IHHY " ww$gb'2"))J~~*, B +  t|| #I ,D -E #s?B0C  C ctj|}|ddjdk(r d|dd}n|jdr|dd}t tj j |S)zCReturn an absolute path without the Windows extended-length prefix.Nz\\?\unc\z\\z\\?\rX)r8fspathlowerrLrr#abspath)r#path_strs r_path_for_comparisonrsiyyH|~-(12,(   Y 'AB< ) **rc`t|}t|} |j|}t|jdk7sCt j |jd!|jddk7s|jdsy|j}d|vsd|vrdS|S#t$rYywxYw)Nrr r rr> )r relative_torlenparts_REPO_DIR_REGEXras_posix)r|r relative_path relative_strs r_relative_blob_pathrs$Y/I$Y/I!--i8  M   A%  $ $]%8%8%; < D   q !W ,""1% ))+L<'4<+?4Q\Q sB!! B-,B-Fraise_on_io_errorrct|}t||y tj|j j sytt j|}t|}|jr|n|j|z }t|}tt|} |j|}t!|j"dk7sDt$j'|j"d"|j"d|j"dddk7ryt||z S#t$rYyt$r|rYywxYw#t$rYywxYw)zMReturn the canonical store target if `blob_path` is a managed shared symlink.Nrr r)rrr&S_ISLNKr(r)r8readlinkFileNotFoundErrorr*r is_absoluterhrrrrrrr)r|rr link_targetcomparable_blob_pathtargetrBrelative_targets r_shared_blob_targetrsMYI9i08 ||IOO-5562;;y12 0 :'335[;O;V;VYd;dF !& )F$%5i%@AI ,,Y7 O ! !"a'  $ $_%:%:1%= > F   #'<' B CCr)NNNc#`Kt|}tjtjz}t tdr|tj z}tj ||d} tjtj|jstd|dt tdr tj|dtj|t|d} |j d  |j'y #t$rYUwxYw#tj|wxYw#t $r)t#|dd}|jt$YqwxYw#t$rYy wxYw# |j'w#t$rYwwxYwxYww) zBLock publication, reference creation, and GC for one content hash. O_NOFOLLOWrmz)Shared blob lock is not a regular file: 'rfchmod)modez.soft)timeoutN)rr8O_WRONLYO_CREATrropenr&r'fstatr)r*rcloseracquireNotImplementedErrorr_SOFT_LOCK_TIMEOUTrelease)r2 lock_pathflagsfdlocks r_shared_blob_lockr<se**5I KK"** $Er<   E5 )B ||BHHRL001Ei[PRST T 2x   "e$  IE *D1     LLN     1yk/e< / 01     LLN   sA'F.*AD+<D"F.5EF E:F. D(%D+'D((D++EF./E74F.6E77F.: FF.FF. F+ FF+ F(%F+'F((F++F. manifest_pathrelative_blob_pathctjtjztjz}t tdr|tj z}tj ||d} tjtj|jstd|dt tdr tj|d|dj}tj||t|k7rtd|dtj |tj"|y#t$rYzwxYw#tj"|wxYw) zBAppend and flush one reference before its symlink is made visible.rrmz-Shared blob manifest is not a regular file: 'rrr>z*Could not append a complete reference to 'N)r8rO_APPENDrrrrr&r'rr)r*rencodewriterfsyncr)rrrrlines r_append_manifest_referencer`s KK"++ % 2Er<   u -B ||BHHRL001I-XZ[\ \ 2x   "e$%%R(//1 88B T *F}oUWXY Y        s2.AEEAE EEEEE+c|jd|jdtjdd}tj j t|t|j}t j|||S)NrWrXz.shared)start) r rKr^r_r8r#relpathrrhsymlink)r|r2tmp_linkrs r_make_temporary_symlinkrvsn""Qy~~&6a8I8I!8L7MW#UVHggoo&::&FNbclNmNtNtouOJJ) OrcJtj xrtj S)z1Return whether the shared blob store may be used.)r HF_HUB_DISABLE_SHARED_BLOBSHF_HUB_DISABLE_XETrrshared_blobs_enabledr}s44 4 YY=Y=Y9YYrctj|tt|syt ||}t ||}|yd} t |5t||s ddd||jdyytt||}tt||tj||ddd||jd t j#d|dy#1swY8xYw#t$r:}t j#d|d|Yd}~||jdyyd}~wwxYw#||jdwwxYw) zMaterialize `blobs/` as a symlink to an existing store entry, if any. The reference manifest is flushed before the symlink becomes visible. Failures are best-effort misses and leave the regular download path untouched. NFTrFzCould not symlink 'z' from shared blob store: Blob 'z5' reused from shared blob store (no download needed).)rrrDrrrrr<rNrrrrr8rcr*r6r\)r|rrr3r2rrres rtry_link_from_shared_storersb  *2:MN^_hNi:j!)X6J,Y B! H - z * ,)*mD ,   OOtO , /tI KH &'DZ'PRd e JJx +  ,   OOtO , LL6)$YZ[  , ,  *9+5OPQsST   OOtO ,    OOtO , sO DC7"DAC7D7D<D E E'E EE E ctr)tj|tt |syt t |||S)zGReturn whether a usable store entry exists, without touching the cache.F)rrrrDrr<r)rrr3s rhas_shared_blobrsA ! "  $ $X . 6"#3I#>? !"29h"G WWr)replace_existingrc\|tj|yt||}t||}||yt |}t ||}t |} d} d} t|5t||} | xr t||} t| || s%t|||tj||d} tj| | ddd| | j'd t"j%d |d y#t$r| rtj ||d} wxYw#1swY`xYw#t$rL} t"j%d|d| | rtd|d| Yd} ~ | | j'dyyd} ~ wwxYw#| | j'dwwxYw) zMove a fresh Xet download into the store and replace its repo blob with a symlink. Best-effort: on failure the repo blob remains (or is restored as) a regular local file. Returns whether the repo blob was successfully shared. NFTzCould not publish 'z' to shared blob store: zCould not restore repo blob 'z' after shared-store failurerFrz!' published to shared blob store.)rrrjrrrrrrr<rrr8rcr*shutilcopyfiler6r\rN)r|rrr3rrir blob_path_objr2rr blob_movedstore_is_usableres rpublish_blob_to_shared_storers 9 9( C K#Ix8J,Y B/7OM!)X6J1*=M HJ- z * .}jIH"22h7MjZg7hO &}6H I"0 IV =*5!   8]3 *   OOtO , LL6)$EFG  OOJ >!&J      *9+5MaSQR 9)D`abhi i   OOtO ,    OOtO , sU D<(AD0:DD<%D--D00D95D<< F-F 2F FFF+ct|}t|sy |jj}t }ttjj|}|D]d}t|}|jsd|jvr-||z }t||? t||d}||k(sT|j|f|S#tt f$rYywxYw#t$rYywxYw)Nz..Tr)rr,r@ splitlinesr*rMsetrr8r#rrrrradd) r2rrlines referencesrr candidaters r_read_valid_manifest_referencesrs1*=M M *'')446 EJRWW__Y/0I &T  $ $ &$-2E2E*E  - y) 4 <  (IQUVF Z  NN9 % & % \ "  s#C+C)C&%C&) C65C6blob_paths_to_deletect||}||r|j|sy t|r|jjSdS#t $rYywxYw)zHPredict bytes freed if all known references are part of a deletion plan.r)rissubsetr,r(r5r*)r2rrrs rexpected_shared_blob_freed_sizersa1YGJ:3F3FG[3\-=j-Iz!))PqP s$A A AAc<t|}t|s tSt} t|j }|D]}t jd|j t|s/ t|j }|D]e}tj|j#|jj|jsIt|sU|j|g|S#t $r|cYSwxYw#t $rYwxYw)zBReturn all valid canonical payload paths in a marked shared store.z [0-9a-f]{2})rrDrrHrIr*rerrKr1rrLr,r)rrBpaths prefix_dirsrirQrRs rshared_store_blob_pathsrs +I y )u uE9,,./ " ! <<  8 @ V`Ha  :--/0G !E))%**5AJJ))*//:$U+ %  ! ! L!     s#C>7D> D  D  DDci}t|D]2}t||}||r |jj||<4|S#t$rYBwxYw)zReturn store payloads without any valid manifest reference, with their sizes. Payloads with a missing or unreadable manifest are not listed, consistent with `sweep_shared_blob`. )rrr(r5r*)r unreferencedr2rs runreferenced_shared_blobsr*sk %'L-i8 4ZK  !* +5+;+;+=+E+E Z(    sA AArcZ|j|jdtjdd}t t j j djfdt|D}|j||jd|jd5}t j|jdddt j|||j!d y#1swY2xYw#|j!d wxYw) NrWrXrYc3bK|]&}|jjd(yw)r>N)rr).0r#rs r z$_rewrite_manifest..>s-gDT--i8AACDBGgs,/rmrbTrF)r rKr^r_rr8r#rjoinsortedr`rarrfilenorcrN)rrrtmp_pathcontentfiles ` r_rewrite_manifestr:s&&-*<*<)=Qw?P?PQR?S>TTX'YZHRWW__Y/0I)''gTZ[eTfggG$u ]]4  $D HHT[[] # $ 8]+4(  $ $ 4(s%AD5$D D DDD*cttjj|}ttjj|}ttjjt |} |j |}t|r]t|jdk7sEtj|j&|jj|jddk7ry t|5t||}| dddyt!|}|rt#||| dddyt%|s dddy|j'j(}|j+|j+d|cdddS#t $rYywxYw#1swYyxYw#t,$r%}t.j1d|d|Yd}~yd}~wwxYw)zCollect one shared blob if its manifest has no remaining valid references. Missing, invalid, or unreadable metadata is never grounds for deletion. Returns the number of payload bytes removed. rrNTrFzCould not collect shared blob 'rZ)rr8r#rrrrrDrrrrrKrhrrrrr,r(r5rNr*r6r\)r2rexpected_store_dirrelative_store_pathrrsizeres rsweep_shared_blobrHs bggooj12JRWW__Y/0Ibggoo.>y.IJK(445GH 2 3 "(( )Q .  $ $Z__ 5 =    ! !Z__Ra%8 8 z * 8YOJ!  :*EM!-YG  $J/  ##%--D      D 1       6zl#aSIJslF"> F= F1F="F1=F= F1F==F1 F=" F.-F.1F:6F=:F== G+G&&G+)I__doc__r8rr^rr& contextlibrpathlibrtypingrfilelockrrrr r get_logger__name__r6compilerrrREPO_TYPES_MAPPINGrrr?rAr!rrescaperJstrrrr"boolr,r1intr<rDrSrfrjrbrxr]rrrrrrrrrrrrrrrrrrdictrrrrrrrs   %+   H %"**_-"**CHHVI4P4P-Q$R#SSXYZ6! BJJ9299-E#F"GG[\]3d 3t3 Ad AcAdA\t\s\t\ 4DtC$J4. cDj T (t(((d t4#*t ( mt m m>cDj>S>""t"" $SVY]S]bf"+sTz+d+R3:R#*RtR&PU 9Tz 9&)Dj 9HL 9 D[ 9FO#*OtOt OHdHtHD$D4D $ 95E+F  FdPT,t$ZdZ  !$ 14t LORVJ   FXXtXCRVJX[_X"# 444Tz 4 : 4  4 4ntPSTXPY\`P`6  /24y EH4Z  sTzc$i6 t T3Y  )T )s4y )SSWZ )\` )&$&cDj&S&r