Batch Queue List

List of LSF queues for each work group

OptionsDescriptionslhp (*1)z
DESCRIPTIONQueue descriptionsJobs with an execution time under 6 hoursJobs with an execution time under 48 hoursBig jobs of usersMPI/OpenMP jobs of usersTheoretical calculations, registered users only
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.120100100100100
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.55555
QJOB_LIMITTotal number of job slots that this queue can use.-----
HJOB_LIMITPer-host running jobs.10285283
UJOB_LIMITPer-user job slot limit.6001000---
PROCLIMITPer-job job slot limit.11124-
RES_REQResource requirements used to determine eligible hosts.-----
CPULIMITMaximum normalized CPU time and optionally.3:0024:00168:001152:00-
RUNLIMITMaximum normalized execution time.6:0048:00240:0072:00-
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)4096409640964096 (*5)4096
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)202020-20
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)
RERUNNABLEIf yes, enables automatic job rerun (restart).YESYESYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.allallallpermitted usersall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.calccomputeT1,T2calccomputeT1,T2calccomputeT1calccomputeT1calccomputeT1
REQUEUE_EXIT_VALUESEnables automatic job requeue.101101101101101
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.---MAX_RESERVE_TIME
[4320]
-
BACKFILLIf Y, enables backfill scheduling for the queue.---y-
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.-----
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.-----
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.(*2)(*2)(*2)export PSM_SHAREDCONTEXTS_MAX=4(*2)


OptionsDescriptionssxlxhxpx (*1)zx
DESCRIPTIONQueue descriptionsJobs with an execution time under 6 hours (Large Memory)Jobs with an execution time under 48 hours (Large Memory)Big jobs of users (Large Memory)MPI/OpenMP jobs of users (Large Memory)Theoretical calculations, registered users only (Large Memory)
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.100100100100100
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.55555
QJOB_LIMITTotal number of job slots that this queue can use.-----
HJOB_LIMITPer-host running jobs.Don't use *x queues without large memory jobs

For allowing using large memory, *x queues have the following limitation of running jobs.
- 6 jobs per-host over all *x queues
- only 408 jobs can run on whole the system
Please note that it may cause a long delay to dispatch of your jobs.
UJOB_LIMITPer-user job slot limit.-----
PROCLIMITPer-job job slot limit.11124-
RES_REQResource requirements used to determine eligible hosts.-----
CPULIMITMaximum normalized CPU time and optionally.3:0024:00168:001152:00-
RUNLIMITMaximum normalized execution time.6:0048:00240:0072:00-
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)2560025600256002560025600
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)202020-20
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)
RERUNNABLEIf yes, enables automatic job rerun (restart).YESYESYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.allallallpermitted usersall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.calccomputeT2calccomputeT2calccomputeT2calccomputeT2calccomputeT2
REQUEUE_EXIT_VALUESEnables automatic job requeue.101101101101101
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.---MAX_RESERVE_TIME
[4320]
-
BACKFILLIf Y, enables backfill scheduling for the queue.---y-
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.-----
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.-----
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.(*2)(*2)(*2)export PSM_SHAREDCONTEXTS_MAX=4(*2)


OptionsDescriptionP1Pmpib_b (*1)b_indexb_nagoya
DESCRIPTIONQueue descriptionsFor parallel serversFor parallel servers(MPI)Analysis jobs of Belle users (72 hours)Analysis jobs of Belle users (for read index files.)Analysis jobs of Belle users (for connect to Nagoya-Univ.)
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.100100100110110
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.5551515
QJOB_LIMITTotal number of job slots that this queue can use.---320-
HJOB_LIMITPer-host running jobs.2424143-5 (*6)3
UJOB_LIMITPer-user job slot limit.---10020
PROCLIMITPer-job job slot limit.24 24 24 (*8)1441 1 4 (*4)11
RES_REQResource requirements used to determine eligible hosts.span[hosts=1]-span[hosts=1]--
CPULIMITMaximum normalized CPU time and optionally.--72:00144:00144:00
RUNLIMITMaximum normalized execution time.--72:0048:0048:00
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)--409640964096
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)--303030
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)b_index b_nagoya
RERUNNABLEIf yes, enables automatic job rerun (restart).NONOYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.allallbelleallbelleallbelleall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.parallelcomputeparallelcomputecalccomputeT1,T2calccomputeT1,T2calccomputeT1,T2
REQUEUE_EXIT_VALUESEnables automatic job requeue.101101---
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.--MAX_RESERVE_TIME
[4320]
--
BACKFILLIf Y, enables backfill scheduling for the queue.--y--
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.00---
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.--409640964096
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.--(*2)(*2)(*2)


OptionsDescriptionkoto_prodth (*1)cmb_pcmb_pxa
DESCRIPTIONQueue descriptionsAnalysis jobs of had_koto usersAnalysis jobs of theory usersFor CMBFor CMB (Large Memory)Guaranteed Resource Allocation (GRA)
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.100100100100100
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.55555
QJOB_LIMITTotal number of job slots that this queue can use.100----
HJOB_LIMITPer-host running jobs.1428282828
UJOB_LIMITPer-user job slot limit.-50144-2
PROCLIMITPer-job job slot limit.11 1 12 (*4)48241
RES_REQResource requirements used to determine eligible hosts.-span[hosts=1]---
CPULIMITMaximum normalized CPU time and optionally.1:00480:001152:001152:0024:00
RUNLIMITMaximum normalized execution time.2:00480:0072:0072:0048:00
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)409640964096256004096
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)2020--20
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
-
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)-
RERUNNABLEIf yes, enables automatic job rerun (restart).YESYESYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.had_kotopermitted usersallallall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.calccomputeT1,T2calccomputeT1calccomputeT1calccomputeT2calccomputeT1,T2
REQUEUE_EXIT_VALUESEnables automatic job requeue.101---101
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.-MAX_RESERVE_TIME
[4320]
---
BACKFILLIf Y, enables backfill scheduling for the queue.-y---
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.-----
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.-----
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.(*2)(*2)---

(*1) When they are pending, jobs in queues p, px and b_b, th reserve the job slots available when they are submitted for their own execution. Reserved job slots cannot execute any other jobs as long as they are reserved. The reservation is kept either until all the job slots requested by the parallel job have been found or 24 hours after the reservation started.

(*2) Run this script for submitted jobs prior to execution. LSB_DJOB_NUMPROC is set a numeric value specified bsub -n option. It is set to [Default value] at PROCLIMIT if it no specify.

if [ -z "${OMP_NUM_THREADS}" ]; then
	if [ -z "${LSB_DJOB_NUMPROC}" ]; then
		export OMP_NUM_THREADS=1
	else
		export OMP_NUM_THREADS=${LSB_DJOB_NUMPROC}
	fi
fi
if [[ -z "${LSB_DJOB_NUMPROC}" ]] || [[ ${LSB_DJOB_NUMPROC} -eq 1 ]]; then
	export BASF_NPROCESS=0
else
	export BASF_NPROCESS=${LSB_DJOB_NUMPROC}
fi

(*3) For information on grouping of crossed-queues, see Crossed-queue Faireshare Group

(*4) It can specify the number of CPUs assigning your jobs in queues b_b and th. This three values at PROCLIMIT mean [Minimum value], [Default value] and [Maximum value] from the left. The number of CPUs is able to specify between [Minimum value] and [Maximum value] by bsub -n option. It is set to [Default value] if it no specify.

(*5) MEMLIMIT of queue p is 96GB when you run 24 processes MPI job.

(*6) HJOB_LIMIT of queue b_index is set bellow values.

cb001 - cb270 ... HJOB_LIMIT = 3
cb501 - cb568 ... HJOB_LIMIT = 5

(*7) OS limit 1500 processes/user apart from LSF limitation.

(*8) The number of CPUs assigning your jobs in queue P1 is fixed to 12, for occupying 1 node by 1 job.

Host Groups for Job Execution

Host GroupNodes 
calccomputeT1cb001 - cb270
calccomputeT2cb501 - cb568
parallelcomputecp01 - cp06

Crossed-queue Fairshare Group

This system defines the faireshare policy which is applied simultaneously to multiple queues (crossed-queue fairshare). When a user submits a job to more than one queue, the priority will be affected by other jobs that the user has submitted to the queues defined in the following table.

GroupQueue
fairshare_grp1s, sx, l, h, p, z, lx, hx, px, zx, P1, Pmpi, b_b, cmb_p, cmb_px, th, koto_prod, gridshort, gridbelle_short, gridilc_short, gridg4med_short, gridmiddle, gridlong, gridbelle_middle, gridbelle_long gridbelle_heavy, gridilc_middle gridilc_long, gridilc_heavy, gridg4med_middle, gridg4med_long, gridg4med_heavy, gridops

Additional Warnings

In addition to the queues listed above, there are also the following queues. They execute only jobs from the Grid system, therefore normal users cannot submit jobs in these queues.

  • gridshort
  • gridmiddle
  • gridlong
  • gridbelle_short
  • gridbelle_middle
  • gridbelle_long
  • gridbelle_heavy
  • gridilc_short
  • gridilc_middle
  • gridilc_long
  • gridilc_heavy
  • gridg4med_short
  • gridg4med_middle
  • gridg4med_long
  • gridg4med_heavy
  • gridops

Last-modified: 2018-12-10 (月) 13:35:02 (0m)