Command line interface¶
All rjm_batch_* commands take -f <localjobdirfile>: a text file
listing local job directories, one per line. Logging is controlled with
-l (logfile), -ll (loglevel), and -le (per-logger overrides).
--force ignores any prior state stored in remote_job.json.
rjm_config¶
This is an interactive script to set up Globus Transfer and Globus Compute on NeSI for use with RJM. It will create a new Globus guest collection (shared directory) on NeSI and will configure RJM to use the Globus Compute endpoint run by NeSI. Finally, all configuration values will be written to the RJM config file on the local machine and the authentication steps will be run.
While running this script, you will need to enter your NeSI username and your NeSI project code and will need to use a web browser to carry out NeSI and Globus authentication as required.
By default this configures the Globus stack. Pass -s / --ssh to
configure the experimental Paramiko (SSH/SFTP) backend instead, in which
case the Globus interactive flow and Globus authentication are skipped.
usage: rjm_config [-h] [-l LOGFILE] [-ll {debug,info,warn,error,critical}]
[-s] [-w] [-v]
Named Arguments¶
- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (setting the level here overrides the config file)
- -s, --ssh
Generate an SSH key pair (stored under ~/.rjm) for use with the Paramiko runner. When this option is chosen, Globus Transfer/Compute setup is skipped.
Default:
False- -w, --where-config
Print location of the config file and exit
Default:
False- -v, --version
show program's version number and exit
rjm_authenticate¶
Refresh or replace the cached Globus tokens at ~/.rjm/rjm_tokens.json.
Only relevant for the Globus backend.
Perform required authentication (if any)
usage: rjm_authenticate [-h] [--force] [--verbose] [-l LOGFILE]
[-ll {debug,info,warn,error,critical}] [-v]
Named Arguments¶
- --force
Delete any stored tokens and force reauthentication
Default:
False- --verbose
Enable print statements
Default:
False- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (default: 'critical')
Default:
'critical'- -v, --version
show program's version number and exit
rjm_health_check¶
Round-trip check against the configured remote: uploads a small file, runs a simple command, and cleans up. Works against both backends.
Perform basic checks of the interface to the remote machine
usage: rjm_health_check [-h] [-l LOGFILE]
[-ll {debug,info,warn,error,critical}] [-le] [-k] [-r]
[-v]
Named Arguments¶
- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (default: 'critical')
Default:
'critical'- -le, --logextra
Also log globus and paramiko at the chosen loglevel
Default:
False- -k, --keep
Keep health check files on remote system, i.e. do not delete them after completing the check (default=False)
Default:
False- -r, --retries
Allow retries on function failures
Default:
False- -v, --version
show program's version number and exit
rjm_batch_submit¶
Upload files listed in each rjm_uploads.txt and start the remote job
for every local job directory listed in -f.
Upload files and start jobs
usage: rjm_batch_submit [-h] -f LOCALJOBDIRFILE [-l LOGFILE]
[-ll {debug,info,warn,error,critical}] [-le] [--force]
[-n] [-c CMD] [-d REMOTEDIR] [-j JOBTYPE] [-m MEM]
[-p PROJECTCODE] [-w WALLTIME] [-v]
Named Arguments¶
- -f, --localjobdirfile
file that contains the names of the local job directories, one name per line
- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (setting the level here overrides the config file)
- -le, --logextra
Also log funcx and globus at the chosen loglevel
Default:
False- --force
ignore progress from previous runs stored in job directory, i.e. start from scratch
Default:
False- -n, --defaultlogname
Use default log name instead of "batch_submit"
Default:
False- -c, --cmd
not used; provided for compatibility with previous versions
- -d, --remotedir
not used; provided for compatibility with previous versions
- -j, --jobtype
not used; provided for compatibility with previous versions
- -m, --mem
not used; provided for compatibility with previous versions
- -p, --projectcode
not used; provided for compatibility with previous versions
- -w, --walltime
not used; provided for compatibility with previous versions
- -v, --version
show program's version number and exit
rjm_batch_wait¶
Wait for the remote jobs to finish and download the files listed in
rjm_downloads.txt.
Wait for the jobs to complete and download files
usage: rjm_batch_wait [-h] -f LOCALJOBDIRFILE [-l LOGFILE]
[-ll {debug,info,warn,error,critical}] [-le]
[-z POLLINGINTERVALSEC] [-w WARMUPPOLLINGINTERVALSEC]
[-d WARMUPDURATIONSEC] [-n] [-v]
Named Arguments¶
- -f, --localjobdirfile
file that contains the names of the local job directories, one name per line
- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (setting the level here overrides the config file)
- -le, --logextra
Also log funcx and globus at the chosen loglevel
Default:
False- -z, --pollingintervalsec
job status polling interval in seconds (minimum is 60)
- -w, --warmuppollingintervalsec
job status polling interval in seconds during the warmup period (minimum is 10)
- -d, --warmupdurationsec
Warmup period duration for job status polling (maximum is 300)
- -n, --defaultlogname
Use default log name instead of "batch_wait"
Default:
False- -v, --version
show program's version number and exit
rjm_batch_run¶
Convenience entry point that performs rjm_batch_submit followed by
rjm_batch_wait in a single invocation.
Upload files, run the job and download results
usage: rjm_batch_run [-h] -f LOCALJOBDIRFILE [-l LOGFILE]
[-ll {debug,info,warn,error,critical}] [--force]
[-z POLLINGINTERVALSEC] [-w WARMUPPOLLINGINTERVALSEC]
[-d WARMUPDURATIONSEC] [-v]
Named Arguments¶
- -f, --localjobdirfile
file that contains the names of the local job directories, one name per line
- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (setting the level here overrides the config file)
- --force
ignore progress from previous runs stored in job directory, i.e. start from scratch
Default:
False- -z, --pollingintervalsec
job status polling interval in seconds (minimum is 60)
- -w, --warmuppollingintervalsec
job status polling interval in seconds during the warmup period (minimum is 10)
- -d, --warmupdurationsec
Warmup period duration for job status polling (maximum is 300)
- -v, --version
show program's version number and exit
rjm_batch_cancel¶
Cancel running jobs for the listed local job directories.
Cancel any running jobs
usage: rjm_batch_cancel [-h] -f LOCALJOBDIRFILE [-l LOGFILE]
[-ll {debug,info,warn,error,critical}]
[-z POLLINGINTERVALSEC] [-v]
Named Arguments¶
- -f, --localjobdirfile
file that contains the names of the local job directories, one name per line
- -l, --logfile
logfile. if not specified, all messages will be printed to the terminal.
- -ll, --loglevel
Possible choices: debug, info, warn, error, critical
level of log verbosity (setting the level here overrides the config file)
- -z, --pollingintervalsec
not used; provided for compatibility with previous versions
- -v, --version
show program's version number and exit