Batch Queue List

List of LSF queues for each work group

OptionsDescriptionslhp (*1)z
DESCRIPTIONQueue descriptionsJobs with an execution time under 6 hoursJobs with an execution time under 48 hoursBig jobs of usersMPI/OpenMP jobs of usersTheoretical calculations, registered users only
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.120100100100100
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.55555
QJOB_LIMITTotal number of job slots that this queue can use.-----
HJOB_LIMITPer-host running jobs.10285283
UJOB_LIMITPer-user job slot limit.10001000400--
PROCLIMITPer-job job slot limit.11124-
RES_REQResource requirements used to determine eligible hosts.-----
CPULIMITMaximum normalized CPU time and optionally.3:0024:00168:001152:00-
RUNLIMITMaximum normalized execution time.6:0048:00240:0072:00-
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)4096409640964096 (*5)4096
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)202020-20
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)
RERUNNABLEIf yes, enables automatic job rerun (restart).YESYESYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.allallallpermitted usersall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.calccomputeT1,T2calccomputeT1,T2calccomputeT1calccomputeT1_evencalccomputeT1
REQUEUE_EXIT_VALUESEnables automatic job requeue.101101101101101
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.---MAX_RESERVE_TIME
[4320]
-
BACKFILLIf Y, enables backfill scheduling for the queue.---y-
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.-----
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.-----
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.(*2)(*2)(*2)-(*2)


OptionsDescriptionssxlxhxpx (*1)zx
DESCRIPTIONQueue descriptionsJobs with an execution time under 6 hours (Large Memory)Jobs with an execution time under 48 hours (Large Memory)Big jobs of users (Large Memory)MPI/OpenMP jobs of users (Large Memory)Theoretical calculations, registered users only (Large Memory)
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.100100100100100
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.55555
QJOB_LIMITTotal number of job slots that this queue can use.-----
HJOB_LIMITPer-host running jobs.10285283
UJOB_LIMITPer-user job slot limit.4001000400--
PROCLIMITPer-job job slot limit.11124-
RES_REQResource requirements used to determine eligible hosts.-----
CPULIMITMaximum normalized CPU time and optionally.3:0024:00168:001152:00-
RUNLIMITMaximum normalized execution time.6:0048:00240:0072:00-
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)2560025600256002560025600
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)202020-20
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)
RERUNNABLEIf yes, enables automatic job rerun (restart).YESYESYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.allallallpermitted usersall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.calccomputeT2calccomputeT2calccomputeT2calccomputeT2_evencalccomputeT2
REQUEUE_EXIT_VALUESEnables automatic job requeue.101101101101101
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.---MAX_RESERVE_TIME
[4320]
-
BACKFILLIf Y, enables backfill scheduling for the queue.---y-
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.-----
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.-----
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.(*2)(*2)(*2)-(*2)


OptionsDescriptionP1Pmpib_b (*1)b_indexb_nagoya
DESCRIPTIONQueue descriptionsFor parallel serversFor parallel servers(MPI)Analysis jobs of Belle users (72 hours)Analysis jobs of Belle users (for read index files.)Analysis jobs of Belle users (for connect to Nagoya-Univ.)
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.100100100110110
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.5551515
QJOB_LIMITTotal number of job slots that this queue can use.---320-
HJOB_LIMITPer-host running jobs.2424143-5 (*6)3
UJOB_LIMITPer-user job slot limit.---10020
PROCLIMITPer-job job slot limit.24 24 24 (*8)1441 1 4 (*4)11
RES_REQResource requirements used to determine eligible hosts.span[hosts=1]-span[hosts=1]--
CPULIMITMaximum normalized CPU time and optionally.--72:00144:00144:00
RUNLIMITMaximum normalized execution time.--72:0048:0048:00
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)--409640964096
SWAPLIMITMaximum swap size of a process in this queue.(MB)-----
PROCESSLIMITMaximum processes of a job in this queue. (*7)--303030
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)b_index b_nagoya
RERUNNABLEIf yes, enables automatic job rerun (restart).NONOYESYESYES
PRE_EXECA command run on the execution host before the job.-----
USERSA space-separated list of user names or user groups that can submit jobs to the queue.allallbelleallbelleallbelleall
HOSTSA space-separated list of hosts on which jobs from this queue can be run.parallelcomputeparallelcomputecalccomputeT1,T2calccomputeT1,T2calccomputeT1,T2
REQUEUE_EXIT_VALUESEnables automatic job requeue.101101---
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.--MAX_RESERVE_TIME
[4320]
--
BACKFILLIf Y, enables backfill scheduling for the queue.--y--
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.00---
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.--409640964096
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.--(*2)(*2)(*2)


OptionsDescriptionkoto_prodth (*1)cmb_pcmb_pxap400
DESCRIPTIONQueue descriptionsAnalysis jobs of had_koto usersAnalysis jobs of theory usersFor CMBFor CMB (Large Memory)Guaranteed Resource Allocation (GRA)MPI/OpenMP jobs of users(400 parallel)
PRIORITYThe queue priority. A higher value indicates a higher LSF dispatching priority, relative to other queues.100100100100100100
NICEAdjusts the UNIX scheduling priority at which jobs from this queue execute.555555
QJOB_LIMITTotal number of job slots that this queue can use.100-----
HJOB_LIMITPer-host running jobs.142828282828
UJOB_LIMITPer-user job slot limit.-501441442-
PROCLIMITPer-job job slot limit.11 1 12 (*4)48241400
RES_REQResource requirements used to determine eligible hosts.-span[hosts=1]----
CPULIMITMaximum normalized CPU time and optionally.1:00480:001152:001152:0024:0019224:00
RUNLIMITMaximum normalized execution time.2:00480:0072:0072:0048:0072:00
MEMLIMITMaximum resident set size (RSS) of a process in this queue.(MB)4096409640962560040964096
SWAPLIMITMaximum swap size of a process in this queue.(MB)------
PROCESSLIMITMaximum processes of a job in this queue. (*7)2020--20-
FAIRSHAREEnables queue-level user-based fairshare and specifies share assignments.USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
USER_SHARES
[default, 1]
--
FAIRSHARE_QUEUESDefines cross-queue fairshare.fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)fairshare_grp1 (*3)--
RERUNNABLEIf yes, enables automatic job rerun (restart).YESYESYESYESYESYES
PRE_EXECA command run on the execution host before the job.------
USERSA space-separated list of user names or user groups that can submit jobs to the queue.had_kotopermitted usersallallallpermitted users
HOSTSA space-separated list of hosts on which jobs from this queue can be run.calccomputeT1,T2calccomputeT1calccomputeT1calccomputeT2calccomputeT1_odd,T2_oddcalccomputeT1_even
REQUEUE_EXIT_VALUESEnables automatic job requeue.101---101101
SLOT_RESERVEEnables processor reservation for the queue and specifies the reservation time.-MAX_RESERVE_TIME
[4320]
---MAX_RESERVE_TIME
[4320]
BACKFILLIf Y, enables backfill scheduling for the queue.-y---y
CORELIMITThe per-process (hard) core file size limit (in KB) for all of the processes belonging to a job from this queue.------
STACKLIMITThe per-process (hard) stack segment size limit (in KB) for all of the processes belonging to a job from this queue.------
JOB_STARTERCreates a specific environment for submitted jobs prior to execution.(*2)(*2)----

(*1) When they are pending, jobs in queues p, px and b_b, th reserve the job slots available when they are submitted for their own execution. Reserved job slots cannot execute any other jobs as long as they are reserved. The reservation is kept either until all the job slots requested by the parallel job have been found or 24 hours after the reservation started.

(*2) Run this script for submitted jobs prior to execution. LSB_DJOB_NUMPROC is set a numeric value specified bsub -n option. It is set to [Default value] at PROCLIMIT if it no specify.

if [ -z "${OMP_NUM_THREADS}" ]; then
	if [ -z "${LSB_DJOB_NUMPROC}" ]; then
		export OMP_NUM_THREADS=1
	else
		export OMP_NUM_THREADS=${LSB_DJOB_NUMPROC}
	fi
fi
if [[ -z "${LSB_DJOB_NUMPROC}" ]] || [[ ${LSB_DJOB_NUMPROC} -eq 1 ]]; then
	export BASF_NPROCESS=0
else
	export BASF_NPROCESS=${LSB_DJOB_NUMPROC}
fi

(*3) For information on grouping of crossed-queues, see Crossed-queue Faireshare Group

(*4) It can specify the number of CPUs assigning your jobs in queues b_b and th. This three values at PROCLIMIT mean [Minimum value], [Default value] and [Maximum value] from the left. The number of CPUs is able to specify between [Minimum value] and [Maximum value] by bsub -n option. It is set to [Default value] if it no specify.

(*5) MEMLIMIT of queue p is 96GB when you run 24 processes MPI job.

(*6) HJOB_LIMIT of queue b_index is set bellow values.

cb001 - cb267 ... HJOB_LIMIT = 3
cb501 - cb568 ... HJOB_LIMIT = 5

(*7) OS limit 1500 processes/user apart from LSF limitation.

(*8) The number of CPUs assigning your jobs in queue P1 is fixed to 12, for occupying 1 node by 1 job.

Host Groups for Job Execution

Host GroupNodes 
calccomputeT1cb001 - cb267
calccomputeT2cb501 - cb568
parallelcomputecp01 - cp06
calccomputeT1_oddodd number server of calccomputeT1
calccomputeT1_eveneven number server of calccomputeT1
calccomputeT2_oddodd number server of calctomputeT2
calccomputeT2_eveneven number server of calccomputeT2

Crossed-queue Fairshare Group

This system defines the faireshare policy which is applied simultaneously to multiple queues (crossed-queue fairshare). When a user submits a job to more than one queue, the priority will be affected by other jobs that the user has submitted to the queues defined in the following table.

GroupQueue
fairshare_grp1s, sx, l, h, p, z, lx, hx, px, zx, P1, Pmpi, b_b, cmb_p, cmb_px, th, koto_prod, gridshort, gridbelle_short, gridilc_short, gridg4med_short, gridmiddle, gridlong, gridbelle_middle, gridbelle_long gridbelle_heavy, gridilc_middle gridilc_long, gridilc_heavy, gridg4med_middle, gridg4med_long, gridg4med_heavy, gridops, p400

Additional Warnings

In addition to the queues listed above, there are also the following queues. They execute only jobs from the Grid system, therefore normal users cannot submit jobs in these queues.

  • gridshort
  • gridmiddle
  • gridlong
  • gridbelle_short
  • gridbelle_middle
  • gridbelle_long
  • gridbelle_heavy
  • gridilc_short
  • gridilc_middle
  • gridilc_long
  • gridilc_heavy
  • gridg4med_short
  • gridg4med_middle
  • gridg4med_long
  • gridg4med_heavy
  • gridops

Last-modified: 2019-10-29 (火) 11:14:12 (2m)