Batch Queue List
Batch Queue List
List of LSF queues for each work group
| Options | Description | s | l | h | p 1 | z |
|---|---|---|---|---|---|---|
| DESCRIPTION | Queue descriptions | Jobs with an execution time under 6 hours | Jobs with an execution time under 48 hours | Big jobs of users | MPI/OpenMP jobs of users | Theoretical calculations, registered users only |
| PRIORITY | The queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues. | 120 | 100 | 100 | 100 | 100 |
| NICE | Adjusts the UNIX scheduling priority at which jobs from this queue execute. | 5 | 5 | 5 | 5 | 5 |
| QJOB_LIMIT | Total number of job slots that this queue can use. | 4000 | - | 1500 | 1500 | - |
| HJOB_LIMIT | Per-host running jobs. | - | - | - | - | - |
| UJOB_LIMIT | Per-user job slot limit. | 1000 | 1500 | 300 | 300 | 100 |
| TASKLIMIT | Per-task limit. | 1 | 1 | 1 | 24 | - |
| RES_REQ | Resource requirements used to determine eligible hosts. | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] |
| CPULIMIT | Maximum normalized CPU time and optionally. | 3:00 | 24:00:00 | 168:00:00 | 576:00:00 | - |
| RUNLIMIT | Maximum normalized execution time. | 6:00 | 48:00:00 | 240:00:00 | 48:00:00 | - |
| MEMLIMIT | Maximum resident set size (RSS) of a process in this queue.（MB） | 4096 | 4096 | 4096 | 4096 5 | 4096 |
| SWAPLIMIT | Maximum swap size of a process in this queue.（MB） | - | - | - | - | - |
| PROCESSLIMIT | Maximum processes of a job in this queue. 7 | 20 | 20 | 20 | - | 20 |
| FAIRSHARE | Enables queue-level user-based fairshare and specifies share assignments. | USER_SHARES [default~1] |
- | - | - | - |
| FAIRSHARE_QUEUES | Defines cross-queue fairshare. | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 |
| RERUNNABLE | If yes, enables automatic job rerun (restart). | YES | YES | YES | YES | YES |
| PRE_EXEC | A command run on the execution host before the job. | - | - | - | - | - |
| USERS | A space-separated list of user names or user groups that can submit jobs to the queue. | all | all | all | all | permitted users |
| HOSTS | A space-separated list of hosts on which jobs from this queue can be run. | calccomputeT1,T2 | calccomputeT1,T2 | calccomputeT1 | calccomputeT1_nonGRA | calccomputeT1 |
| REQUEUE_EXIT_VALUES | Enables automatic job requeue. | 101 | 101 | 101 | 101 | 101 |
| SLOT_RESERVE | Enables processor reservation for the queue and specifies the reservation time. | - | - | - | MAX_RESERVE_TIME [4320] |
- |
| BACKFILL | If Y, enables backfill scheduling for the queue. | - | - | - | y | - |
| CORELIMIT | The per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - |
| STACKLIMIT | The per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - |
| JOB_STARTER | Creates a specific environment for submitted jobs prior to execution. | 2 | 2 | 2 | export PSM_SHAREDCONTEXTS_MAX=4 | 2 |
| Options | Descriptions | sx | lx | hx | px 1 | zx |
|---|---|---|---|---|---|---|
| DESCRIPTION | Queue descriptions | Jobs with an execution time under 6 hours (Large Memory) | Jobs with an execution time under 48 hours (Large Memory) | Big jobs of users (Large Memory) | MPI/OpenMP jobs of users (Large Memory) | Theoretical calculations, registered users only (Large Memory) |
| PRIORITY | The queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues. | 100 | 100 | 100 | 100 | 100 |
| NICE | Adjusts the UNIX scheduling priority at which jobs from this queue execute. | 5 | 5 | 5 | 5 | 5 |
| QJOB_LIMIT | Total number of job slots that this queue can use. | - | - | 300 | - | - |
| HJOB_LIMIT | Per-host running jobs. | - | - | - | - | - |
| UJOB_LIMIT | Per-user job slot limit. | 200 | 200 | 60 | 100 | 20 |
| TASKLIMIT | Per-task limit. | 6 | 6 | 6 | 24 | 6 |
| RES_REQ | Resource requirements used to determine eligible hosts. | affinity[core(1):cpubind=core] span[hosts=1] | affinity[core(1):cpubind=core] span[hosts=1] | affinity[core(1):cpubind=core] span[hosts=1] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] span[hosts=1] |
| CPULIMIT | Maximum normalized CPU time and optionally. | 3:00 | 24:00:00 | 168:00:00 | 576:00:00 | - |
| RUNLIMIT | Maximum normalized execution time. | 6:00 | 48:00:00 | 240:00:00 | 48:00:00 | - |
| MEMLIMIT | Maximum resident set size (RSS) of a process in this queue.（MB） | 8192 | 8192 | 8192 | 8192 | 8192 |
| SWAPLIMIT | Maximum swap size of a process in this queue.（MB） | - | - | - | - | - |
| PROCESSLIMIT | Maximum processes of a job in this queue. 7 | 20 | 20 | 20 | - | 20 |
| FAIRSHARE | Enables queue-level user-based fairshare and specifies share assignments. | - | - | - | - | - |
| FAIRSHARE_QUEUES | Defines cross-queue fairshare. | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 |
| RERUNNABLE | If yes, enables automatic job rerun (restart). | YES | YES | YES | YES | YES |
| PRE_EXEC | A command run on the execution host before the job. | - | - | - | - | - |
| USERS | A space-separated list of user names or user groups that can submit jobs to the queue. | all | all | all | all | permitted users |
| HOSTS | A space-separated list of hosts on which jobs from this queue can be run. | calccomputeT2 | calccomputeT2 | calccomputeT2 | calccomputeT2 | calccomputeT2 |
| REQUEUE_EXIT_VALUES | Enables automatic job requeue. | 101 | 101 | 101 | 101 | 101 |
| SLOT_RESERVE | Enables processor reservation for the queue and specifies the reservation time. | - | - | - | MAX_RESERVE_TIME [4320] |
- |
| BACKFILL | If Y, enables backfill scheduling for the queue. | - | - | - | y | - |
| CORELIMIT | The per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - |
| STACKLIMIT | The per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - |
| JOB_STARTER | Creates a specific environment for submitted jobs prior to execution. | 2 | 2 | 2 | export PSM_SHAREDCONTEXTS_MAX=4 | 2 |
| Options | Description | P1 | Pmpi | b_b 1 | b_index | b_nagoya |
|---|---|---|---|---|---|---|
| DESCRIPTION | Queue descriptions | For parallel servers | For parallel servers(MPI) | Analysis jobs of Belle users (72 hours) | Analysis jobs of Belle users (for read index files.) | Analysis jobs of Belle users (for connect to Nagoya-Univ.) |
| PRIORITY | The queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues. | 100 | 100 | 100 | 110 | 110 |
| NICE | Adjusts the UNIX scheduling priority at which jobs from this queue execute. | 5 | 5 | 5 | 5 | 5 |
| QJOB_LIMIT | Total number of job slots that this queue can use. | - | - | - | 600 | 600 |
| HJOB_LIMIT | Per-host running jobs. | - | - | - | - | - |
| UJOB_LIMIT | Per-user job slot limit. | - | - | 1000 | 100 | 100 |
| TASKLIMIT | Per-task slot limit. | 40 40 407 | 240 | 1 1 44 | 1 | 1 |
| RES_REQ | Resource requirements used to determine eligible hosts. | span[hosts=1] affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | span[hosts=1] affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] |
| CPULIMIT | Maximum normalized CPU time and optionally. | - | - | 96:00 | 48:00 | 48:00 |
| RUNLIMIT | Maximum normalized execution time. | - | - | 48:00 | 48:00 | 48:00 |
| MEMLIMIT | Maximum resident set size (RSS) of a process in this queue.（MB） | - | - | 4096 | 4096 | 4096 |
| SWAPLIMIT | Maximum swap size of a process in this queue.（MB） | - | - | - | - | - |
| PROCESSLIMIT | Maximum processes of a job in this queue. 7 | - | - | 30 | 30 | 30 |
| FAIRSHARE | Enables queue-level user-based fairshare and specifies share assignments. | - | - | - | USER_SHARES [default~1] |
- |
| FAIRSHARE_QUEUES | Defines cross-queue fairshare. | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 | b_nagoya | - |
| RERUNNABLE | If yes, enables automatic job rerun (restart). | NO | NO | YES | YES | YES |
| PRE_EXEC | A command run on the execution host before the job. | - | - | - | - | - |
| USERS | A space-separated list of user names or user groups that can submit jobs to the queue. | parallelcompute users only | parallelcompute users only | belleall | belleall | belleall |
| HOSTS | A space-separated list of hosts on which jobs from this queue can be run. | parallelcompute | parallelcompute | calccomputeT1,T2 | calccomputeT1 | calccomputeT1 |
| REQUEUE_EXIT_VALUES | Enables automatic job requeue. | 101 | 101 | - | - | - |
| SLOT_RESERVE | Enables processor reservation for the queue and specifies the reservation time. | - | - | MAX_RESERVE_TIME [4320] |
- | - |
| BACKFILL | If Y, enables backfill scheduling for the queue. | - | - | y | - | - |
| CORELIMIT | The per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue. | 0 | 0 | - | - | - |
| STACKLIMIT | The per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | 4096 | 4096 | 4096 |
| JOB_STARTER | Creates a specific environment for submitted jobs prior to execution. | - | - | 2 | 2 | 2 |
| Options | Description | koto_prod | th 1 | cmb_p | cmb_px | a | p400 |
|---|---|---|---|---|---|---|---|
| DESCRIPTION | Queue descriptions | Analysis jobs of had_koto users | Analysis jobs of theory users | For CMB | For CMB (Large Memory) | Guaranteed Resource Allocation (GRA) | MPI/OpenMP jobs of users（400 parallel) |
| PRIORITY | The queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues. | 100 | 100 | 100 | 100 | 100 | 100 |
| NICE | Adjusts the UNIX scheduling priority at which jobs from this queue execute. | 5 | 5 | 5 | 5 | 5 | 5 |
| QJOB_LIMIT | Total number of job slots that this queue can use. | 300 | - | - | - | - | - |
| HJOB_LIMIT | Per-host running jobs. | - | - | - | 10 | - | - |
| UJOB_LIMIT | Per-user job slot limit. | - | 300 | 300 | 100 | 4 | 800 |
| TASKLIMIT | Per-task slot limit. | 1 | 1 1 244 | 48 | 24 | 1 | 400 |
| RES_REQ | Resource requirements used to determine eligible hosts. | - | span[hosts=1] affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] |
| CPULIMIT | Maximum normalized CPU time and optionally. | 1:00 | 1152:00 | 2304:00 | 1152:00 | 24:00 | 19224:00 |
| RUNLIMIT | Maximum normalized execution time. | 2:00 | 480:00 | 48:00 | 48:00 | 48:00 | 48:00 |
| MEMLIMIT | Maximum resident set size (RSS) of a process in this queue.（MB） | 4096 | 4096 | 4096 | 24567 | 4096 | 4096 |
| SWAPLIMIT | Maximum swap size of a process in this queue.（MB） | - | - | - | - | - | - |
| PROCESSLIMIT | Maximum processes of a job in this queue. 7 | 20 | 20 | - | - | 20 | - |
| FAIRSHARE | Enables queue-level user-based fairshare and specifies share assignments. | - | - | - | - | - | - |
| FAIRSHARE_QUEUES | Defines cross-queue fairshare. | - | fairshare_grp1 3 | fairshare_grp1 3 | fairshare_grp1 3 | - | - |
| RERUNNABLE | If yes, enables automatic job rerun (restart). | YES | YES | YES | YES | YES | YES |
| PRE_EXEC | A command run on the execution host before the job. | - | - | - | - | - | - |
| USERS | A space-separated list of user names or user groups that can submit jobs to the queue. | had_koto | permitted users | cmb | cmb | all | permitted users |
| HOSTS | A space-separated list of hosts on which jobs from this queue can be run. | calccomputeT1,T2 | calccomputeT1 | calccomputeT1 | calccomputeT1_GRA | calccomputeT1_nonGRA | |
| REQUEUE_EXIT_VALUES | Enables automatic job requeue. | 101 | - | - | - | 101 | 101 |
| SLOT_RESERVE | Enables processor reservation for the queue and specifies the reservation time. | - | MAX_RESERVE_TIME [4320] |
- | - | - | MAX_RESERVE_TIME [4320] |
| BACKFILL | If Y, enables backfill scheduling for the queue. | - | y | - | - | - | y |
| CORELIMIT | The per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - | - |
| STACKLIMIT | The per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - | - |
| JOB_STARTER | Creates a specific environment for submitted jobs prior to execution. | 2 | 2 | - | - | - | - |
| Options | Description | b2_a | b2_beast | b2_fei | b2_prod | b_prod | dc_generic | ph | t2k |
|---|---|---|---|---|---|---|---|---|---|
| DESCRIPTION | Queue descriptions | Analysis jobs of Belle2 users | Analysis jobs of Belle2 users | Analysis jobs of Belle2 users | Analysis jobs of Belle2 users | Analysis jobs of Belle users | Jobs of OpenStack | MPI/OpenMP jobs of users (Long jobs) | Analysis jobs of t2k users |
| PRIORITY | The queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues. | 100 | 130 | 110 | 110 | 110 | 100 | 100 | 100 |
| NICE | Adjusts the UNIX scheduling priority at which jobs from this queue execute. | 5 | 5 | 5 | 5 | 5 | - | 5 | 5 |
| QJOB_LIMIT | Total number of job slots that this queue can use. | - | 200 | 100 | 400 | 1000 | - | - | 500 |
| HJOB_LIMIT | Per-host running jobs. | - | - | - | - | - | - | - | - |
| UJOB_LIMIT | Per-user job slot limit. | - | - | - | - | - | - | 100 | - |
| TASKLIMIT | Per-task slot limit. | 1 4 44 | 1 | - | 1 | 1 | - | 48 | 1 |
| RES_REQ | Resource requirements used to determine eligible hosts. | span[hosts=1] affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] | span[hosts=1] affinity[core(1):cpubind=core] | - | affinity[core(1):cpubind=core] | affinity[core(1):cpubind=core] |
| CPULIMIT | Maximum normalized CPU time and optionally. | 96:00 | 3:00 | 48:00 | 24:00 | 168:00 | - | 23040:00 | 3:00 |
| RUNLIMIT | Maximum normalized execution time. | 48:00 | 6:00 | 96:00 | 48:00 | 168:00 | - | 480:00 | 6:00 |
| MEMLIMIT | Maximum resident set size (RSS) of a process in this queue.（MB） | 4096 | 4096 | 4096 | 4096 | 4096 | - | 4096 | 4096 |
| SWAPLIMIT | Maximum swap size of a process in this queue.（MB） | - | - | - | - | - | - | - | - |
| PROCESSLIMIT | Maximum processes of a job in this queue. 7 | 30 | 20 | 20 | 100 | 30 | - | - | 20 |
| FAIRSHARE | Enables queue-level user-based fairshare and specifies share assignments. | - | - | - | - | - | - | - | - |
| FAIRSHARE_QUEUES | Defines cross-queue fairshare. | - | - | - | - | - | - | - | - |
| RERUNNABLE | If yes, enables automatic job rerun (restart). | YES | YES | YES | YES | NO | - | YES | YES |
| PRE_EXEC | A command run on the execution host before the job. | - | - | - | - | - | - | - | - |
| USERS | A space-separated list of user names or user groups that can submit jobs to the queue. | permitted users | permitted users | permitted users | permitted users | permitted users | - | permitted users | t2k |
| HOSTS | A space-separated list of hosts on which jobs from this queue can be run. | calccomputeT1,T2 | calccomputeT1,T2 | calccomputeT1,T2 | calccomputeT1,T2 | calccomputeT1,T2 | - | calccomputeT1_nonGRA | calccomputeT1,T2 |
| SLOT_RESERVE | Enables processor reservation for the queue and specifies the reservation time. | - | - | - | - | MAX_RESERVE_TIME [4320] |
- | MAX_RESERVE_TIME [4320] |
- |
| BACKFILL | If Y, enables backfill scheduling for the queue. | y | - | - | - | y | - | y | - |
| CORELIMIT | The per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue. | - | - | - | - | - | - | - | - |
| STACKLIMIT | The per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue. | 4096 | - | - | - | 4096 | - | - | - |
| JOB_STARTER | Creates a specific environment for submitted jobs prior to execution. | 2 | 2 | 2 | 2 | 2 | - | export PSM_SHAREDCONTEXTS_MAX=4 | 2 |
Host Groups for Job Execution
| Host Group | Nodes |
|---|---|
| calccomputeT1 | ccb001 - ccb292 |
| calccomputeIO | ccb293 - ccb294 |
| calccomputeT2 | ccb501 - ccb572 |
| calccomputeT1_GRA | 32 nodes which match the first half of calccomputeT1 with the second half |
| calccomputeT1_nonGRA | calccomputeT1 other than calccomputeT1_nonGRA |
| parallelcompute | ccp01 - ccp06 |
Crossed-queue Fairshare Group
This system defines the faireshare policy which is applied simultaneously to multiple queues (crossed-queue fairshare). When a user submits a job to more than one queue, the priority will be affected by other jobs that the user has submitted to the queues defined in the following table.
| Group | Queue |
|---|---|
| fairshare_grp1 | sx, l, h, p, z, lx, hx, px, zx, P1, Pmpi, b_b, b2_a, cmb_p, cmb_px, t2k, koto_prod, th, gridshort, gridbelle_short, gridilc_short, gridg4med_short, gridmiddle, gridlong, gridbelle_middle, gridbelle_long, gridbelle_heavy, gridbelle_mergejob, gridilc_middle, gridilc_long, gridilc_heavy, gridg4med_middle, gridg4med_long, gridg4med_heavy, gridops, p400, ph |
Additional Warnings
In addition to the queues listed above, there are also the following queues. They execute only jobs from the Grid system, therefore normal users cannot submit jobs in these queues.
- gridshort
- gridmiddle
- gridlong
- gridbelle_short
- gridbelle_middle
- gridbelle_long
- gridbelle_heavy
- gridilc_short
- gridilc_middle
- gridilc_long
- gridilc_heavy
- gridg4med_short
- gridg4med_middle
- gridg4med_long
- gridg4med_heavy
- gridops
-
When they are pending, jobs in queues p, px, p400, ph and b_b, th reserve the job slots available when they are submitted for their own execution. Reserved job slots cannot execute any other jobs as long as they are reserved. The reservation is kept either until all the job slots requested by the parallel job have been found or 24 hours after the reservation started. ↩↩↩↩
-
Run this script for submitted jobs prior to execution. LSB_DJOB_NUMPROC is set a numeric value specified bsub -n option. It is set to [Default value] at PROCLIMIT if it no specify.
if [ -z "${OMP_NUM_THREADS}" ]; then if [ -z "${LSB_DJOB_NUMPROC}" ]; then export OMP_NUM_THREADS=1 else export OMP_NUM_THREADS=${LSB_DJOB_NUMPROC} fi fi if [[ -z "${LSB_DJOB_NUMPROC}" ]] || [[ ${LSB_DJOB_NUMPROC} -eq 1 ]]; then export BASF_NPROCESS=0 else export BASF_NPROCESS=${LSB_DJOB_NUMPROC} fi↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩ -
For information on grouping of crossed-queues, see Crossed-queue Faireshare Group ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
-
It can specify the number of CPUs assigning your jobs in queues b_b and th. This three values at PROCLIMIT mean [Minimum value], [Default value] and [Maximum value] from the left. The number of CPUs is able to specify between [Minimum value] and [Maximum value] by bsub -n option. It is set to [Default value] if it no specify. ↩↩↩
-
MEMLIMIT of queue p is 96GB when you run 24 processes MPI job. ↩
-
HJOB_LIMIT of queue b_index is set bellow values.
cb001 - cb267 ... HJOB_LIMIT = 3 cb501 - cb568 ... HJOB_LIMIT = 5↩ -
OS limit 1500 processes/user apart from LSF limitation. ↩↩↩↩↩↩
-
The number of CPUs assigning your jobs in queue P1 is fixed to 24, for occupying 1 node by 1 job. ↩