/jdZddlZddlmZmZmZddlmZddlm Z ddl m Z e dZ gdZ d d gZdd ee zd ed efdZdddddee deeezdzdeeezdzdee gefdzd ee ddff dZdee zd efdZded efdZy)z6Contains utilities to handle paths in Huggingface Hub.N)Callable GeneratorIterable fnmatchcase)Path)TypeVarT).gitz.git/*z*/.gitz **/.git/**z.cache/huggingfacez.cache/huggingface/*z*/.cache/huggingfacez**/.cache/huggingface/**r z.cachepath max_lengthreturnct|}tjdk7r|Stjj |}t ||ks|j dr|S|j drd|ddzSd|zS)aReturn `path` in its Windows extended-length form if it is too long, unchanged otherwise. Some Windows versions do not allow for paths longer than 255 characters (247 for directories, i.e. MAX_PATH minus room for an 8.3 file name). In this case, we must specify them as extended paths by using the `\\?\` prefix, which only works on absolute paths. Network shares take the `\\?\UNC\server\share\...` form: prefixing them verbatim would produce an invalid `\\?\\\server\...` path. Args: path (`str` or `Path`): The path to convert. Returned unchanged on non-Windows platforms, if it is short enough, or if it is already an extended path. max_length (`int`, *optional*): Length above which the path must be converted. Defaults to 255, the limit for files. Pass 247 for paths whose parent directories are created by the caller. ntz\\?\z\\z\\?\UNC\N)strosnamer abspathlen startswith)r r absolute_paths _/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_paths.pyas_extended_pathr(sy t9D ww$ GGOOD)M =Z'=+C+CI+N ' ab 111 } $$)allow_patternsignore_patternskeyitemsrrrc#Kt|tr|g}t|tr|g}|!|Dcgc]}tt|}}|!|Dcgc]}tt|}}|dtdtfd}|}|D]E}t|||t fd|Ds+|t fd|DrB|Gycc}wcc}ww)a Filter repo objects based on an allowlist and a denylist. Input must be a list of paths (`str` or `Path`) or a list of arbitrary objects. In the later case, `key` must be provided and specifies a function of one argument that is used to extract a path from each element in iterable. Patterns are Standard Wildcards (globbing patterns), NOT regular expressions. The pattern matching is based on Python's `fnmatch.fnmatchcase`, so it is case-sensitive on every platform. Backslashes are treated as path separators and normalized to forward slashes in both patterns and paths before matching, so patterns built with `os.path.join` on Windows work as expected. Note that it matches `*` across path boundaries, unlike traditional Unix shell globbing. For example, `"data/*.json"` will match both `data/file.json` and `data/subdir/file.json`. See https://docs.python.org/3/library/fnmatch.html for more details. Args: items (`Iterable`): List of items to filter. allow_patterns (`str` or `list[str]`, *optional*): Patterns constituting the allowlist. If provided, item paths must match at least one pattern from the allowlist. ignore_patterns (`str` or `list[str]`, *optional*): Patterns constituting the denylist. If provided, item paths must not match any patterns from the denylist. key (`Callable[[T], str]`, *optional*): Single-argument function to extract a path from each item. If not provided, the `items` must already be `str` or `Path`. Returns: Filtered list of objects, as a generator. Raises: :class:`ValueError`: If `key` is not provided and items are not `str` or `Path`. Example usage with paths: ```python >>> # Filter only PDFs that are not hidden. >>> list(filter_repo_objects( ... ["aaa.pdf", "bbb.jpg", ".ccc.pdf", ".ddd.png"], ... allow_patterns=["*.pdf"], ... ignore_patterns=[".*"], ... )) ["aaa.pdf"] ``` Example usage with objects: ```python >>> list(filter_repo_objects( ... [ ... CommitOperationAdd(path_or_fileobj="/tmp/aaa.pdf", path_in_repo="aaa.pdf"), ... CommitOperationAdd(path_or_fileobj="/tmp/bbb.jpg", path_in_repo="bbb.jpg"), ... CommitOperationAdd(path_or_fileobj="/tmp/.ccc.pdf", path_in_repo=".ccc.pdf"), ... CommitOperationAdd(path_or_fileobj="/tmp/.ddd.png", path_in_repo=".ddd.png"), ... ], ... allow_patterns=["*.pdf"], ... ignore_patterns=[".*"], ... key=lambda x: x.path_in_repo ... )) [CommitOperationAdd(path_or_fileobj="/tmp/aaa.pdf", path_in_repo="aaa.pdf")] ``` Nitemrczt|tr|St|tr t|Std|d)Nz9Please provide `key` argument in `filter_repo_objects`: `z` is not a string.) isinstancerr ValueError)r!s r _identityz&filter_repo_objects.._identitys=$$ $%4y XY]X^^pqr rrc36K|]}t|ywNr.0rr s r z&filter_repo_objects..s1_1+dA2F1_c36K|]}t|ywr'rr(s rr+z&filter_repo_objects..s.]{4/C.]r,)r#r_add_wildcard_to_directories_normalize_separatorsr any)rrrrpr%r!r s @rfilter_repo_objectsr2CsL.#&()/3'*+!ZhiUV67LQ7OPii"[jkVW78Ma8PQkk { sA s# s $SY/  %c1_P^1_._   &3.]_.]+]   jks-CC CC.A+Cvaluec8t|jddS)N\/)rreplace)r3s rr/r/s u:  dC ((rpatternc2|jdr|dzS|S)Nr6*)endswith)r8s rr.r.s} Nr))__doc__rcollections.abcrrrfnmatchrpathlibrtypingr r DEFAULT_IGNORE_PATTERNSFORBIDDEN_FOLDERSrintrlistr2r/r.rrrGs= 99 CL X&%3:%3%%<.2.2%) g A;gIOd*g#Y_t+ g 1#s( d " g q$} gT)t))##r