Work Server, Parallel Server

Using Work Servers / Parallel Servers

System Overview

The Data Analysis system offers total of eleven work servers and six parallel servers.

NB) Only granted users can login to Parallel servers.


Work servers T1	7
Work servers T2	4
Parallel servers	6

Main Features of Work Servers

SSH-login service
File access via General Parallel File System (GPFS) and HSM System (GHI)
Calculations with batch job scheduling using Platform LSF
Various softwares

Main Features of Parallel Servers

SSH-login service
File access via General Parallel File System (GPFS)
Calculations with batch job scheduling using Platform LSF
Various softwares (Mainly Gaussian)

NB) Parallel servers process calculation jobs themselves, unlike work servers which hand them to the batch servers. For this reason, when a job is being executed the response time of parallel servers to other requests (such as logins) might be unusually long.

Environment Requirements

SSH Version 2 must be used to access these servers remotely.

See below for more information on connections with SSH.

When using UNIX terminals such as Linux, simply use the ssh command.
If you are connecting from Windows, you can either install cygwin and execute ssh from its terminal or use a terminal emulator such as TeraTerm or PuTTY. If you use TeraTerm, choose UTF-8 TeraTerm Pro.

In addition, direct remote access is permitted on this system. See here for more details.

Warning

Be sure to set the passphrase to the authentication keys for ssh. The keys which passphrase isn't set to are deleted by administrator. Also, do not place unnecessary keys.

Log In / Log Out

Work Servers

When logging in to work servers, use a delegate host name listed in the table below. If you use an abbreviation (e.g. login.cc) for the delegate host name it is sometimes not possible to connect. In that case, use the FQDN (e.g. login.cc.kek.jp).

Delegate Host Name		Host Name	OS
login.cc.kek.jp	ccw.cc.kek.jp	ccw01.cc.kek.jp	CentOS7.7
		ccw02.cc.kek.jp	CentOS7.7
		ccw03.cc.kek.jp	CentOS7.7
		ccw04.cc.kek.jp	CentOS7.7
		ccw05.cc.kek.jp	CentOS7.7
		ccw06.cc.kek.jp	CentOS7.7
		ccw07.cc.kek.jp	CentOS7.7
-	ccx.cc.kek.jp	ccx01.cc.kek.jp	CentOS7.7
		ccx02.cc.kek.jp	CentOS7.7
		ccx03.cc.kek.jp	CentOS7.7
		ccx04.cc.kek.jp	CentOS7.7

[Example]

$> ssh  ccw.cc.kek.jp

Once logged in with a delegate host name, work servers will be assigned by round robin. For this reason, you log in to a different host every time. Bear in mind that local settings on a particular node (e.g. crontab) are not supported for any of the work servers.
All user accounts and passwords are managed by LDAP. Thus, the same account name and password can be used to log in to any hosts.
In case a work server had a failure, it is taken out of the round robin selection. After a problem occurred, it takes time between the reconfiguration of the round robin and the moment it disappears from the selection. If you are not able to access a delegate host name, try to log in again. It is possible to log in to a different work server.

CPU/Elapsed time limitation of processes on Work Servers

Work servers ( ccw.cc.kek.jp and ccx.cc.kek.jp ) limit CPU/Elapsed time of processes.

Over 12hours of CPU time ... Kill the process
Over 2weeks of Elapsed time ... Kill the process
Occupy CPU usage (>300%.H/proc, >8x(100%.0.5H)/node) ... Limit the process

Memory limitations on Work Servers

The amount of memory usage for all general users is limited on each work server as mentioned below. In case the memory utilization reaches the limit, a series of processes which use large amount of memory may be forced to terminate for system protection. We appreciate your understandings.

ccw.cc.kek.jp : 176GB
ccx.cc.kek.jp : 368GB
Use >350GB.D (50GB-7days) ... Kill the process

Warning

When we find an abuse that affects system operation, we might perform killing a process, or limiting usage of CPU and/or memory.

Parallel Servers

When logging in to parallel servers, normally you should use ccp01.cc.kek.jp. The other parallel servers (ccp02～ccp06) are used by submitting jobs from ccp01.

Changing Password

The first time you log in, the passwd command is automatically executed. Once you have changed your password, the session will terminate once, so you have to log in again. For password policies, please see the paragraph Password Rules. SSH ver2 is the required protocol to log in.

WARNING: Your password has expired.
You must change your password now and login again!
Changing password for user a0019.
Enter login(LDAP) password:  -   -   -   - ← Input your current password
New password:  -   -   -   -   -   -   -   ← Input your new password
Retype new password:  -   -   -   -   -    ← Input your new password again
LDAP password information changed for a0019
passwd: all authentication tokens updated successfully.

Changing Password (Any Time)

To change password, use passwd command;

[Example]

$ passwd
Changing password for user a0019.
Enter login(LDAP) password:  -   -   -   - ← Input your current password
New password:  -   -   -   -   -   -   -   ← Input your new password
Retype new password:  -   -   -   -   -    ← Input your new password again
LDAP password information changed for a0019
passwd: all authentication tokens updated successfully.

Warning

If you make a mistake when you input the current password, the passwd command will terminate without changing anything.

Password Rules

Avoid specifying a password based on a Dictionary Word and follow the rules below.

the password must have at least 9 characters,
it must include a least 1 non-alphanumeric symbole (ex: $ % *),
it must include a least 1 digit,
it must include at least 1 lowercase alphabet letter,
it must contain at least 5 different characters ("aa" is counted as 1, "ab" is counted as 2),
the last 4 passwords used cannot be re-use.

To change your login shell, execute the command chsh with the option "-s" and the full path of your preferred shell as arguments. You can check the list of available shells by using the "-l" option. The change of shell will take effect at the next login.

Note

After chsh run, it can take up to 5 minutes.

[Example]

$ chsh -l
/bin/bash
/bin/csh
/bin/sh
/bin/tcsh
/bin/zsh
$ chsh -s /bin/tcsh
 Shell was changed.
$

Environment Settings

Tcsh/csh

.cshrc This file is invoked when a user l - ogs in with ssh by csh or tcsh. Add or modify the file if neccesary. tcsh users should be aware that if you create a .tcshrc file in your home directory, the .cshrc file will be invalid.

Note

This file is located in the home directory.

Note

The originals of the files above are located under /sw/share/etc/skel on the work/parallel servers, you can refer to them if necessary.

Bash/sh

.bash_profile This file is invoked when a user logs in with ssh on bash. .bashrc This file is invoked from .bash_profile. Add and modify this file if neccesary.

Note

These files are located in the home directory.

Note

The originals of the files above are located under /sw/share/etc/skel on the work / parallel servers, you can refer to them if necessary.

Tools Configuration Files

Each tool has environment configuration files located under /etc/profile.d.

The main tools added to this system are listed below along with their environment configuration files.

	For csh/tcsh	For ksh/bash/zsh
LSF	lsf.csh	lsf.sh
Tex	texlive.csh	texlive.sh
Environment Modules	modules.csh	modules.sh
Mathematica	Mathematica.csh	Mathematica.sh

Warning

These files cannot be modified by normal users. Nevertheless, it is possible to overwrite the environment variables set in these files. If you change one of these variables to a wrong value, the tools will no longer work properly. Please be careful when modifying the value of an environment variable.

Environment Modules

In this system, "Environment Modules" is available to switch environment variables for using a software including several versions of compiler, MPI Library, Python, JAVA.

The Environment Modules allows users to set environment variables dynamically required for using each software. It provides easily way to changing environment variables when using different category or versions of each software.

Warning

In the initial configuration, only OS standard version of each software is available. It is recommended to put "module load" command (refer to instructions below) in your shell configuration files which automatically invoked when logging in (.bashrc, .cshrc, etc.).

Warning

When using each software in LSF jobs, you also need to load environment settings. Execute "module load" command to load environment settings in your shell configuration files which invoked when logging in or in your job scripts before using each software,

Warning

If you want to use the module command in an LSF job when the $SHELL environment variable is not /bin/bash, include "source /etc/profile.d/modules.sh;" in the job.

Available Modules

With the command below, all the available modules are listed.

$> module avail↓

Note

"avail" can be abbreviated as "av".

[Example]

$> module av↓

--------------------------- /opt/Modules/modulefiles ---------------------------
aocl/aocl-2.1         intel32/2020          openmpi/2.1.6-gcc750
binutil/234           openmpi/1.10.7-gcc    openmpi/2.1.6-gcc830
gcc/750               openmpi/1.10.7-gcc750 openmpi/2.1.6-gcc930
gcc/830               openmpi/1.10.7-gcc830 openmpi/2.1.6-intel
gcc/930               openmpi/1.10.7-gcc930 openmpi/2.1.6-pgi
git/2260              openmpi/1.10.7-intel  pgi/20.1(default)
gsl/26                openmpi/1.10.7-pgi    pgi/2020
intel/2020            openmpi/2.1.6-gcc     python/3.7

Loading an environment setting

With the command below, the environment setting ls loaded.

$> module load <module name>↓

By executing "module load" command, environment variables such as PATH, etc. are set in the current executing shell environment.

[Example]

The syntax of loading the environment setting required for OpenMPI build with Intel compiler.

$> module load openmpi/1.10.7-intel↓

$> which mpirun
/opt/openmpi/intel/openmpi-1.10.7/bin/mpirun

$> which icc
/opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc

Note

By loading the OpenMPI module, corresponding compiler module is loaded automatically.

Display environment variables that will be set

With the command below, you can confirm environment variables that will be set.

$> module show <module name>↓

[Example]

$> module show openmpi/1.10.7-intel
-------------------------------------------------------------------
/opt/Modules/modulefiles/openmpi/1.10.7-intel:

prepend-path  -  PATH /opt/openmpi/intel/openmpi-1.10.7/bin
prepend-path  -  LD_LIBRARY_PATH /opt/openmpi/intel/openmpi-1.10.7/lib
prepend-path  -  MANPATH /opt/openmpi/intel/openmpi-1.10.7/share/man
-------------------------------------------------------------------

Warning

You can check the environment variables of the compiler module that is automatically loaded when you load the OpenMPI module, by executing "modules show " command.

The syntax of checking the environment setting of Intel compiler.

$> module show intel/2020

Display module names that are currently loaded

With the command below, you can confirm module names that currently loaded.

$> module list↓

Note

"list" can be abbreviated as "li".

[Example]

$> module li
Currently Loaded Modulefiles:
  1) intel/2020  -   -   -  2) openmpi/1.10.7-intel

Additional loading modules and loading multiple modules

You can load additional modules by executing more than once "modules load" commands. You can also load Multiple modules at once by executing a "modules load" command.

[Example]

$> module load openmpi/1.10.7-intel python/3.7↓

 Please enable conda command :source /opt/python-3.7/etc/profile.d/conda.sh

$> module li
Currently Loaded Modulefiles:
  1) intel/2020  -   -   -  3) python/3.7
  2) openmpi/1.10.7-intel

Unloading modules

With the command below, you can unload a module that currently loaded.

$> module unload <module name that currently loaded>↓

[Example]

$> module unload openmpi/1.10.7-intel

Cleaning up modules that currently loaded

With the command below, you can unload all modules that currently loaded.

$> module purge↓

Note

There are module command options "switch/swap" for swhiching modules, but these options somtimes does not work correctry, so we don't recommend to use these options. "purge" or "unload" options are recommened.

Languages

The standard language environment of work/parallel servers is UTF-8（en_US.UTF-8） which is an English environment. Other available codes include POSIX C, EUC(Japanese) and UTF-8. To change language, set the value of an environment variable LANG as follows. If not specified, the language environment will be set to UTF-8.

Environment	Setting
English	en_US.UTF-8
POSIX C	C
Japanese UTF-8	ja_JP.UTF-8
Japanese EUC	ja_JP.eucJP
Shift JIS	ja_JP.SJIS

Japanese EUC and SJIS are not officially supported but it is possible to use them.
However, file names with Shift JIS characters are not supported.

［Example］ With csh

%> setenv LANG ja_JP.UTF-8↓

With bsh

$> export LANG=ja_JP.UTF-8↓

When using Japanese UTF-8 or languages other than English, the terminal emulator setting on the client and the LANG variable on the server must match. In case of TeraTerm, the right correspondence between the two terminal settings is shown below.

TeraTerm Setting	LANG Environment Variable
UTF-8	ja_JP.UTF-8
Japanese EUC	ja_JP.eucJP

Using Japanese

Japanese input method is mostly dependent on terminal emulators used on client machines. For example, if you use TeraTerm on Windows, it is possible to use Japanese Input Methods available on Windows (MS-IME or ATOK...).

For more details, please refer to the manual of each method.

Emails

Receiving emails on work/parallel servers is not supported. Ountbound emails are not supported either, except for LSF job result notification emails.

Do not use the mail command.

Printing

List of Printers and Queues

Refer to Printer Information for the list of printers and printer queues.

Printing Output

Use the lpr command to print out.

$> lpr -P Priter Queue Name  filename.ps↓

Example) Printing out on one side with printer DocuPrint CP400ps (Queue：xprt41ps1）in the Computing Research center warehouse.

$> lpr -P xprt41ps1 test.ps↓

If the output file is not a ps file, you can convert it to ps format using the paps command.

$> a2ps -o - filename | lpr -P xprt41ps1↓

or

$> a2ps -P xprt41ps1  filename↓

Default printers are not configured, but if necessary you can set the environment variable PRINTER to the printer queue you usually use.

%> setenv PRINTER xprt41ps1↓  -    ← for csh
$> export PRINTER=xprt41ps1↓  -    ← for bsh

If the command above is added in .cshrc or .profile file, after log in again, the command "lpr filename.ps" will print the file with the default printer.

$> export PRINTER=xprt41ps1↓  -    ← for bsh

If the command above is added in .cshrc or .profile file, after log in again, the command "lpr filename.ps" will print the file with the default printer xprt41ps1.

Note

The paps command currently installed enables printing files written in UTF-8 character encoding. (Japanese only)

Example) tcsh is used in this example.

%> setenv PRINTER xprt41ps1↓
%> setenv LANG ja_JP.UTF-8↓
%> echo  'これは日本語のテストです。' > test.txt↓
%> a2ps test.txt↓

In addition, the Perl version of this command, a2ps.pl has been installed in /usr/local/bin. It does not require to set up LANG, but it does require Japanese encoding to be EUC.

Example） tcsh is used in this example.

%> setenv PRINTER xprt41ps1↓
%> /usr/local/bin/a2ps.pl test.txt | lpr↓

Cancellation

To view a job waiting atop of printer queue, or a job that is being printed;

$> lpq↓

[Example]

$> lpq↓
 xprt41ps1 is ready and printing
 Rank  - Owner   Job  -  File(s)  -   -   -   -   -   -  Total Size
 active  t0001   3677  - test_c  -   -   -   -   -   -   1024 bytes

To cancel a job waiting or printing;

$> lprm↓

Note

You cannot cancel print jobs that have been already sent to the printer with this command. To cancel them, use the cancel button on the printer.

To cancel a waiting job in printer queue;

$> lprm Job ID↓

[Example]

$> lpq↓
 xprt41ps1 is ready and printing
 Rank  - Owner   Job  -  File(s)  -   -   -   -   -   -  Total Size
 active  t0001   3677  - test_c  -   -   -   -   -   -   1024 bytes
$> lprm 3677↓

Check the print job ID with the lpq command, and then cancel it by using this number and the lprm command.

You can only cancel your own print jobs.

Other Software

It is also possible to use enscript (No Japanese) or paps+lpr to print files.

[Example with enscript]

$> enscript  -P xprt41ps1 --media=A4  test.txt↓
 [ 1 pages * 1 copy ] sent to xprt41ps1

Note

In enscript, the default paper size is letter. It can be changed to A4 by specifying "DefaultMedia: A4" in the confiruation file (~/.enscriptrc).

[Example with paps+lpr]

$> paps test.txt  | lpr -P xprt41ps1↓

Important Point regarding Printers

If you wish to use any printer that is NOT listed in the Printer Information Page (including personal printers) from work servers, ask the administrator beforehand since network configuration changes might be required.

Online Manuals

Online manuals are available from command line on work/parallel servers.

man command

$> man command name↓

[Example]

$> man ls↓

info, pinfo command

$> info command name↓
$> pinfo command name↓

[Example]

$> info ls↓
$> pinfo ls↓

Additional Warnings

/tmp directory

/tmp shall be the location that makes basic system applications available to create temporary files. Please refrain from creating large size files or number of files in the /tmp by yourself. If the usage rate of the / tmp directory becomes high, large files are deleted. /tmp directory has been set quota, 1GB/users.

sshfs

Please mind the following things for using sshfs. It will cause a serious impact on the system operation if it is not used in the right way.

Unmount sshfs mount point before suspending your computer.
Use a direct hostname (ccw*.cc.kek.jp) as a connecting server instead of "login.cc.kek.jp".
DO NOT use an access server for sshfs. This will cause a serious impact on the system operation.

Work Server, Parallel Server

Using Work Servers / Parallel Servers

System Overview

Main Features of Work Servers

Main Features of Parallel Servers

Environment Requirements

Log In / Log Out

Work Servers

CPU/Elapsed time limitation of processes on Work Servers

Memory limitations on Work Servers

Parallel Servers

Changing Password

First Login

Changing Password (Any Time)

Password Rules

Changing Login Shell

Environment Settings

Tcsh/csh

Bash/sh

Tools Configuration Files

Environment Modules

Available Modules

Loading an environment setting

Display environment variables that will be set

Display module names that are currently loaded

Additional loading modules and loading multiple modules

Unloading modules

Cleaning up modules that currently loaded

Languages

Using Japanese

Emails

Printing

List of Printers and Queues

Printing Output

Cancellation

Other Software

Important Point regarding Printers

Online Manuals

man command

info, pinfo command

Additional Warnings

/tmp directory

sshfs