O/j9UdZddlmZddlmZmZddlZddlmZddl Z ddl m Z ddl Z ddlZddlZddlZddlmZmZmZdd lmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%ddl&Z'eedZe jPe)gdfe*d<dZ+eGddZ,dZ-de j\defdZ/ede,fdedede)dede j`ede,de jbe fdZ2dZ3de jbe)dede jbe de jbe fd Z4de j\deded!e jbe d"e jPde5f d#Z6y)$z This defines the `dora grid` action, and provides a `run_grid` API that can be used from a notebook or any other script. When using the API, you can provide the equivalent of the command line flags with the `RunGridArgs` dataclass. )ProcessPoolExecutor) dataclassfieldN)partial)Path) SlurmConfig SubmitRulesupdate_from_args)ExplorerLauncherHerd) DecoratedMain)colorize simple_logfatal)SheepShepherd)import_or_fatalreliable_rmtreetry_loadzGrid:logcyN)argskwargss L/root/repos/pocs/video-evals/.venv/lib/python3.12/site-packages/dora/grid.pyno_printr%sceZdZUdZeeZeje e d<dZ e e d<dZ ee d<dZej ee d <d Ze e d <d Ze e d <d Ze e d <d Ze e d<d Ze e d<d Zej e e d<d Ze e d<dZej ee d<dZej ee d<dZej ee d<d Ze e d<y) RunGridArgsay Arguments to tune the behavior of the `run_grid` function. Args: patterns (list[str]): List of patterns used to filter by name the XPs. monitor (bool): if True, will monitor the advances of the XPs every `interval` minutes, stopping only when all runs completed or failed. interval (float): interval in minutes to wait between updates. trim (int or None): if provided, will trim all XP logs to the epoch of the XP with the provided index. Useful to compare XP started at different times. trim_last (bool): if True, will trim all XP to the least advanced XP. silent (bool): if True, do not print anything (e.g. API usage). dry_run (bool): if True, Dora will simulate the run of the grid, without scheduling or canceling any XP. cancel (bool): if True, will cancel all XPs in the grid. If `patterns` is provided, only XP matching the patterns will be canceled. clear (bool): This will cancel any previous job, clear the XP folder, and reschedule a new experiment. )default_factorypatternsTmonitorintervalNtrimF trim_lastsilentdry_runcancelclearinitjupyterfolderrtail_from_commandline)__name__ __module__ __qualname____doc__rlistr$tpListstr__annotations__r%boolr'floatr(Optionalintr)r*r+r,r-r.r/r0rr1r2rr rr"r")s.#48Hbggcl8GTHe!D"++c !ItFDGTFDE4#D"++d #GT $FBKK # CS  !D"++c !#t#r r"c|jj}||jdz}t|}|jA|jj dddz}t |jj|z }|jjs>g}t |jj}tj|D]\}} } | D]} t || z j|} | jjds<| jjdrX| j| j z } |j#t%| j dd|j"jst'dd|dt'd d j)|t+j,d |dz|jz} t| } |j.}t5t6st3|d |S#t0$rt3| d Y9wxYw)Nz.grids./z.py_z No grid file z in package z. Maybe you made a typo?zPotential grids are: z, rz( has no exploration function `explorer`.z' must be an instance of `dora.Explorer`)dora grid_packagepackagergridreplacer__file__parentexistsoswalk relative_tonameendswith startswithstemappendr:rjoinsysexitexplorerAttributeErrorr isinstancer )rmainrEgrids grid_filename grid_file candidatespkg_rootrootfoldersfilesfilefullpath grid_namerGrWs r _get_explorerf[s99))L||h. L )E yy ))#s3e; (//-?  yy 0 0 2 '..$&GGH$5 G D'5 G J-::8D==))%09Q9QRU9V'>H%%c(m&;&;C&EF  G G 99 )9)9); - l<.I)) * #DIIj$9#: ;<  s"TYY.I 9 %DF== h )  ABC O F  CDEFs9 H%%H?>H?rrZct||}|j}t||t}t||t }d|_t||t |||j|||y)NT)rfget_slurm_configr r r"r2run_gridrG)rrZrWslurmrules grid_argss r grid_actionrmsfD$'H  ! ! #EUD! MEUD! I"&IY% T8TYYui@r rWrerkrjreturnc t|tsJ||j}|jj|jj z |z }|j ddt}t|t}|jrjArtC| j<j>d| _Ng}g}|jED]]}|jF|j vs|jI| |jK|jF}|J|jI|_|j|jVs| D cic]} | j<jX| }} |jZD]}|Dcgc] }||vs||}}|s|d}|j\|j<jX}t/|dk(r|j_|||g|ja|5|D]} |j_| || ddd|D]}|jcr|j6J|j9||je|j<}td|j6jTd|j<jXd||jVr| D]} | jcr| j6J|je| j<}td| j6jTd| j<jXd||j9| |j*s| D]} || j<jXz }|jAs|jgrH|jgr6|ji| j<j>jik(rJ|jk| j<j>|j;|D]}|jm|jnr"| D]} |jq| j<|jVr| S| s td| S|j>|j|jrfDcgc]}|| }}|rC|j0s tudt/|dk(sJ|d} | | } |je j<}|j>!ty| j<j>| S|jrZ| jjAst-d | jd!t{j|d"d"d#d$d%| jg| S| jjAst-d&|d tjt| jtj| S|jrt}!ntx}!|!d'| |jr$|jsdd(lHmI}"|"jd)|jt|||| |!r | S|js | St|jd*z}#|!t|#D]?}$d+|#|$z d,d-}%|#|$z d.kr t|%d/}%|!|%d01tjdA|!d2#1swYxYw#tL$r}td|jFd|||jjNjPz }tS|}|P|d}t/|dk(r|d}td|jTd|jFd|j9|Yd}~d}~wwxYwcc} wcc}w#1swY:xYwcc}w#tv$rt-d|j>Y wxYw#t$rY| SwxYw)3a Run a grid search, this is the API underlying the `dora grid` command, so that it can be used from a notebook. You can also provide patterns to filter out XPs to be displayed. Args: main (DecoratedMain): main training function, decorated with Dora. explorer (Explorer): explorer instance that will define the XPs to launch. grid_name (str): this must be a unique name for the grid. rules (SubmitRules): see `dora.conf.SubmitRules`, those defines the rules for rescheduling failed XP etc. slurm (SlurmConfig or None): if provided, this will override the default Slurm config defined my the `main` argument. Returns: A list of `dora.shep.Sheep`. NT)exist_okparents)r)poolz'--dry_run is incompatible with --clear.zYou are about to restart z experiments from the grid z' from scratch. This cannot be reverted.zConfirm [yN]: yzAbort...zCanceling all current jobs...)sheepzDeleting XP folders...z$Error when trying to load old sheep z: rzCanceling job z from unloadable sheep rArz for no longer required sheep rBz for sheep zNo sheep to handle.zIThe folder, log, and tail flags are only supported from the command line.zInvalid index zLog z does not existr1z-n200z-fz"Log file does not exist for sheep zMonitoring Grid )display)wait<zNext update in z.0fz seconds 31 )endz< )RrYr rhrDdir_gridsmkdirrrr_slowrr completeupdater7sheepsvalues_filter_grid_sheepsr$r-r+rlenr2inputlowerjob cancel_lazycommitxpr0rKriterdirrOrSget_sheep_from_sig Exceptionshepjob_filerjob_idr,sig job_arrays slurm_configsmaybe_submit_lazy job_arrayis_doneget_name is_symlinkresolve symlink_tounlinkr.init_xpr1 RuntimeError IndexErrorprintrLexecvpshutil copyfileobjopenrUstdoutBrokenPipeErrorr*rr/IPythonry clear_outputr%r?r'rangertimesleep)&rZrWrerkrjr grid_folderherdshepherdrslauncherrreplru to_unlink old_sheepschild old_sheeperrorrjobsrdependent_jobs sheep_maprr array_sheepsfirstrOlinkactionactionsindex maybe_printryrelaouts& rriris* h )) ) }%%'))--$))"2"22Y>KtT2 6D#&H zz  # t%DAH X  MMO   HeT2 OO $++$$& 'F f =F zz << ; < 'F }4OPY{[5 5 6  ! !)*Dzz|s"j! +, 2Eyy$$$5$1 2  $% Exx%%'0EI  IJ$$&- ::T[[ (   U # -$77 C !,,,!!),%-( OO ;;6<=UUXX\\5(= = HI6?Ts3)CSIcNTLT OE&&uxx||4E< A%**5%?''.H!-H 225%GHHH H 4   "==, ,,  y 1==.D .!5!5 67"))*!D63 4 4 {{ 2E==?yy,,,}}UXX.nUYY%5%5$6k%((,,qQUPVWX$$5$1  2 << 1E%((,,.D{{} 1(T\\^uxx?V?V?X-XXX0  1  E LLN  yy #E LL " # {{  !" %)[[$((DII$F]&&J\v]G]%% QR R7|q     25ME}}UXX& ;; " %((// " YY "99##%UYYK78 IIfvtUD%))D E 99##%:4&BC ""4 ?CJJ?  {{  "9+./  << '  d + 4x =  M||  MDMMB&' < C#ECK#4ODCs{RsD)  & JJqM   H% a  N >:5::,bPQ 499>>#:#::)#q'C4yA~)-a. 4KEJJ$>THHV^ 2 N4;;-0 1 2#   ss(a"=a/d! d& d&3d+d8d8d=.7e%"a,/ d8Bdd+d5 =!e"!e"% e21e2c|sy|D]:}d}|dddk(r|dd}d}tj|d|zdz}|r|s5y|r:yy)NTFr!*)fnmatch)rOr$patternnegresults r _match_namerRsg   2A;# abkGCsW}s':;   r r$rcVg}t|D].} |jt||j|0g}|D];}|j |j }t||s+|j|=|r|Dcgc]}|| }}|S#t$rYwxYwcc}wr)r7rSr?remove ValueErrorrrr) r$rZrindexesprrurOidxs rrrcsG (^  NN3q6 " OOA   C}}UXX& tX & JJu #*+Cs3x++ J   ,sB B& B#"B#rrc2|j|Dcgc]}|jc}\}}|Dcgc]}|j|j}}d} |jt ||j} n|j rt d|D} | |D cgc]} | d|  }} g} d} tt|||D]\} \}}}|j}|jsd} |d}n|dd}|| |jxsd|jj|d}i}||d < |j||}|j!|| j#||r |d |t%j&dt%j(d |j+g|j-z }|t%j.| ||j1 | Scc}wcc}wcc} w#t$r|j|}YwxYw) zSingle iteration of monitoring of the jobs in a Grid. Returns `True` if all jobs are done or failed, and `False` otherwise. Nc32K|]}t|ywr)r).0metricss r zmonitor..s9G3w<9sTFzN/Arv)rOrsidrstateMetaz Base name: )shortengroups)colors) get_namesrget_xp_historyr(rr)min enumerateziprrcurrent_job_idr process_sheepNotImplementedErrorprocess_historyrrStttablegroup get_grid_metaget_grid_metrics treetable get_colors)rrZrWrrrunames base_name historiesr(rlinesfinishedrhistoryrOrmetalineotherrs rr%r%ws  ~~T&BEuxx&BCE9<@A5$$UXX.AIA D yy9TYY'( 9y99 3<=WUd^= = EH)23tY3N)O%%w }}H =E"1IE''-288<<  V  6**5':E E T-0M9- HH HHVX335 6  % % '( E  UE(2E2E2GHI O_'CA>.# 6,,W5E 6s"G*"G/ G40G99HH)7r6concurrent.futuresr dataclassesrrr functoolsrrLpathlibrtypingr8rrUrconfr r r explorer r rrZrrrrrrrrutilsrrrrrCallabler:r;rr"rfAnyrmr>r9rirrr<r%rr rrs3(  <<--,,!== ' G <R[[#<  .$.$ .$b#L Abff AM A#.-SW!,B=BHBBB8: K8PBB35775>BJ""''#,m "35775>(4"&&4 44QV44)-4r