Last modified: Feb 07, 2012 by Lemrani

Translation of BQS to GE commands

1. Main commands

	BQS	GE
submit and monitor a job
submit job	qsub, qrun	qsub -P P_myproject
monitor job	qjob	qstat, qacct
	qcat	-
	qselect	-
Modify status and options of a job
delete job	qdel	qdel
hold job	qhold	qhold
release job	qrls	qrls
alter job	qalter	qalter
send signal to job	qsig	qmod
resubmit job	qrerun	qresub + qdel
global commands
accounting	qaccount	qacct
farm configuration	qshow, qvar	qhost, qconf, qquota
other commands
advanced reservation		qrsub, qrstat, qrdel
		qlogin
		qsh
		qrsh
		qtcsh
		qmake
		qmod
		qmon
		qping

2. qsub options

	BQS	GE
help	qsub -help man qsub	qsub -help man qsub
resources	-l ressource=[value]	-l ressource=[value]
virtual memory	-l M=256MB	-l vmem=256M -l h_vmem=256M (hard) -l s_vmem=256M (soft)
disk space	-l scratch=4096MB	-l fsize=4096M -l h_fsize=4096M (hard) -l s_fsize=4096M (soft)
cpu time	-l T=300000	-l ct=6000 (real time is seconds) -l ct=01:40:00 -l h_cpu=01:40:00 (hard) -l s_cpu=01:40:00 (soft)
elapsed time		-l h_rt=6000 (real time is seconds) -l h_rt=01:40:00 (hard) -l s_rt=01:40:00 (soft)
standard output size	-l spool=512KB
resources	-l hpss,u_hpss_{groupname}	-l hpss=1
platform	-l platform=LINUX+oldLINUX	-l os=sl5 (optional)
other options
job name	-N job_name	-N job_name
export environment variables	default	default (-V by default)
prefix for directives in script	-C prefix	-C prefix (prefix = 2 ascci characters, default = «#$».
option files		-@ optionfile (command lines start with #)
submit binary file		-b job
send email at address name	-mu user[@host]	-M user1[@host1],user2[@host2]
emails sent when:	-mb, -me	-m b|e|a|s|n : beginning/end/abortion/suspended/no email
submit at specified time	-a time (-a 02120430)	-a time MMDDhhmm.[SS]
standard output	-o stdout.txt -e stderr.txt	-o stdout.txt -e stderr.txt
standard input		-i stdin.txt
verify job (verify as with qstat -j but don’t submit job)		-verify
merge error into output	-o /path/stdout.txt -eo	-o /path/stdout.txt -j y[es]
run on current working directory		-cwd
run on specified directory		-wd working_dir
define queue	-q <class>	-q <queue>
no rerun	-nr	-r y[es]|n[o]
execute at end of job	-af /path/filename.sh
arguments	#!/usr/local/bin/tcsh qsub <<eof $HOME/test.sh arg1 arg2 arg3 eof	qsub [command [args]] qsub myjob arg1 arg2 arg3
Local scratch space	$TMPBATCH	$TMPDIR

3. BQS classes: definition / SGE queues: definition

BQS	GE
A	short
G	medium
T	long
restricted access
W	verylong
J	huge
V	demon

4. Significant differences BQS/Grid Engine

For more details "how to submit a job"

- You should specify the units of vmem and fsize and they should be at least 64M
- Your group has to be specified with the option -P P_myproject for instance (-P P_atlas for the atlas group).
This is typically a default option that can be put in your $HOME/.sge_request file.
- There is no resource per group to specify (like u_hpss_atlas) but only global resources (like hpss=1). The limitations per group are done by Grid Engine configuration based on the specified project.
- The CPU specified in Grid Engine is real time (for instance in seconds) and is a factor 40 to 50 lower
than for BQS.
- The local scratch space is $TMPDIR instead of $TMPBATCH.
- Environmental variables :
BATCH_SYSTEM=GE
ENVIRONMENT = ACCESS for the ccage and ccige host machines.
ENVIRONMENT = SEQUENTIAL_BATCH | PARALLEL_BATCH | INTERACTIVE_BATCH
for submitted the jobs
- qalter : You need to specify again all resources used in the qsub otherwise the resources not specified will take the default value.

Some novelties

- Job arrays : submit an array of many tasks with one qsub
- Interactive jobs : Use qlogin command with same options as qsub command (access on ccige.in2p3.fr)
- Muticore jobs : use several cores on a single machine
- Job dependencies : A job will not be executed before a list of other jobs, tasks is finished.

Not available in Grid Engine
- The qcat command : you can use instead interactive jobs with qlogin command.
- The possibility to release a resource (set it to 0) during a job
- When a job is submitted Grid Engine doesn't test that the resources (memory, scratch, cpu)
match the caracteristics of a queue. If there is a mismatch, the job will stay in queue indefinitely.
It's better to specify the needed resources without specifying the queue. Grid Engine automatically
assigns the matching queue to the job.

Faq
- My outputs are not in the submission directory:
By default the outputs (stdout and stderr) are put on the $HOME directory for Grid Engine instead of the submission directory. You can use -o $PWD and -e $PWD to redirect the outputs to the submission directory.
- I get the error message : "unable to set bqs configuration" :
Something in your own profil or the profil of a group supersedes the default profile of cc-in2p3 users. The command ge_env (or source /usr/local/shared/bin/ge_env.sh or .csh) cures this problem.
- In the default configuration #$ in a script precedes options for Grid Engine and this might induce errors if you have #$ in your old scripts.
- Note that from Feb. 7th 2012 the cpu obtained with qacct is directly expressed in the normalized units : HS06 . secondes