O/jdZddlmZddlmZmZddlmZddlmZddl Z ddl m Z ddl mZd e jd efd Zd e jd e fd ZeGddZeGddZeGddZeGddZy)z' Basic configuration for Dora is here. ) Namespace) dataclassfield)fnmatch)PathN) DictConfig) OmegaConfdataargsc|jD]=}t|tsJt||s"t ||}|1t |||?y)z>Update the given dataclass from the argument parser args. N)__dict__ isinstancestrhasattrgetattrsetattr)r r keyvalues L/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/dora/conf.pyupdate_from_argsrsO}}*#s### 4 D#&E c5) *cfgctj|d}t|tsJ|j D]K\}}t|t sJt ||rt|||2td|jd|y)z6Update the given dataclass from the hydra config. T)resolvezObject of type z does not have an attribute N) r to_containerrdictitemsrrrAttributeError __class__)r rdctrrs rupdate_from_hydrar!s  d 3C c4  iikF U#s### 4  D#u % ?4>>2BC??Be"EF F FrcJeZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z e jeed <d Zeed <d Ze jeed<eeZe j(eed<dZeed<dZeed<dZeed<dZeed<d Ze jeed<d Ze jeed<d Ze jeed<dZeed<dZeed<d Ze jeed<eeZ e j(eed <d Z!e jeed!<d Z"eed zed"<y )# SlurmConfiga Configuration when scheduling a job. This differs slightly from Slurm/Submitit params because this will automatically scale some values based on the number of GPUs. Args: gpus (int): number of total GPUs to schedule. Number of nodes and tasks per nodes will be automatically inferred. mem_per_gpu (float): amount of memory in GB to schedule per gpus. time (int): maximum duration for the job in minutes. cpus_per_gpu (int): number of cpus per gpu, this will set the `cpus_per_task` automatically, based on the number of gpus and `one_task_per_node`, unless `cpus_per_task` is explicitely provided. cpus_per_task (int or None): number of cpus per task. partition (str): partition name comment (str): comment for the job. setup (List[str]): list of shell commands to execute before the actual command. Use it for `module load`. max_num_timeout (int): maximum number of requeue. one_task_per_node (bool): if True, schedules a single task per node, otherwise, will schedule one task per gpu (default is False). array_parallelism (int): when using job arrays, how many tasks can run in parallel. qos (str or None): qos param for slurm. account (str or None): account param for slurm. dependents (int): if > 0, start a number of dependent jobs. Requeuing will be deactivated and rely on dependent jobs instead. container_chdir (bool): set to True when using a container image to chdir. force_chdir (bool or None): same as container_chdir, kept for compat. srun_args (list[str]): extra srun args, useful for using a container image. python (str or None): indicate to submitit to use an alternative python. ..warning:: this assumes one task per GPU. Set `one_task_per_node` if you do not want that. Tasks without any gpus are not really supported at the moment. gpus( mem_per_gpuitime cpus_per_gpuN cpus_per_tasklearnlab partitioncommentdefault_factorysetupmax_num_timeout constraintFone_task_per_nodearray_parallelismexcludeqosaccountr dependentscontainer_chdir force_chdir srun_argspythonnodelist)#__name__ __module__ __qualname____doc__r%int__annotations__r'floatr(r*r+tpOptionalr-rr.rlistr1Listr3r5r6boolr8r9r:r;r<r=r>r?r@rArrr#r#.s?%LD#MKD#L#&*M2;;s#*Is $GR[[ $5E2773<5OSJ#t# s $GR[[ $ CS  $GR[[ $J!OT!%)KT")#D9Irwws|9#FBKK #!%Hd3i$%rr#cJeZdZUdZdZeed<dZeed<dZeed<dZ eed<y) SubmitRulesa: Submit rules describe in which case Shepherd will schedule new jobs. Args: retry (bool): if true, all failed or canceled jobs will be rescheduled. update_pending (bool): if true, all pending jobs whose Slurm parameters have changed will be replaced. update (bool): if true, all pending or running jobs whose Slurm parameters have changed will be replaced. replace (bool): if true, all running jobs will be replaced by new jobs. replace_done (bool): if true, all done jobs will be relaunched. Fretryupdatereplace replace_doneN) rBrCrDrErQrMrGrRrSrTrNrrrPrPms1 E4FDGTL$rrPcteZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <y ) ShepConfigz Configuration for Shepherd. Mostly naming conventions for folders and files. There should be little reasons to change that. zjob.pkljob_filezjob.json json_job_fileby_idorphanssubmititsubmitit_folderlatestlatest_submititarraysN) rBrCrDrErWrrGrXrYrZr\r^r_rNrrrVrVsOHc#M3#E3GS%OS%#OS#FCrrVceZdZUdZedZeed<eeZ e je ed<dZ eed<eeZe je ed<dZeed <d Ze j$eed <d Ze j$e ed <d Ze ed<dZe ed<eeZeed<dZe ed<dZeed<dZe ed<dZe ed<de defdZfdZxZS) DoraConfiga Main Dora configuration. The main parameters to change are the following. Args: dir (Path or str): path where Dora will save all useful informations, logs. This is also where you should store your checkpoints (see `dora.xp.XP`). exclude (List[str]): list of patterns of argument names to ignore when computing the XP signature and doing deduplication. For instance 'num_workers', etc. git_save (bool): when True, experiments can only be scheduled from a clean repo. A shallow clone of the repo will be made and execution will happen from there. This does not impact `dora run` unless you pass the `--git_save` flag. post_git_save_commands (list[str]): sequence of commands to run on a fresh git clone. local_code (bool): if True, copies the git clone into /tmp before running. shared (Path or None): if provided, the path to a central repository of XPs. For the moment, this only supports sharing hyper-params, logs etc. will stay in the per user folder. grid_package (str or None): if provided, package to look for grids. Default to the package with the `train.py` module followed by `.grids`. z ./outputsdirr/r9Fgit_savepost_git_save_commands local_codeNshared grid_packagez history.jsonhistoryxpsshepzrendezvous.txtrendezvous_fileuse_rendezvousgrids_gridscodes_codesarg_namereturncB|jD]}t||syy)zUReturn True if the given argument name should be excluded from the signature.TF)r9r)selfrqpatterns r is_excludedzDoraConfig.is_excludeds)|| Gx) rc`|dvrddlm}|t||}t|||y)N)rbrfr$)to_absolute_path)rcrxrsuper __setattr__)rtnamerrxrs rrzzDoraConfig.__setattr__s4 $ $ 2 -e45 D%(r)rBrCrDrErrbrGrrKr9rIrLrrcrMrdrerfrJrgrhrirVrjrkrlrnrprvrz __classcell__)rs@rraras([!C!!$7GRWWS\7Hd+0+FBGGCLFJ $FBKK $%)L"++c")"GS!CZ8D*8+OS+ ND FCFCCD))rra)rEargparser dataclassesrrrpathlibrtypingrIomegaconf.dictconfigr omegaconfr Anyrr!r#rPrVrarNrrrs(+*266** FBFF F F ;&;& ;&|  (      5)5) 5)r