User Guide for EMI System

Introduction

EMI System Overview

The Data Analysis System has adopted UMD, a middleware developed mainly by EGI (European Grid Infrastructure) to enable WLCG.
UMD is a replacement middleware of gLite3.2, which has been used in former system.
In this document, we call this UMD-featured system the "EMI System".
Note: WLCG (Worldwide LHC Computing Grid) is a GRID computing infrastructure for LHC.

EMI system provides following functions:

  • Sharing computing resources between domestic and overseas research institutes
  • Managing, sharing and utilizing massive data
  • Monitoring of resource utilization
  • Efficient user management

Component overview of EMI System is shown below.

EMI_en.jpg

The function of each components is described below.

  • UI (User Interface)
    UI provides client functions such as user authentication, job submission and data transfer.
    In this system, the UI features contained within "/cvmfs/grid.cern.ch/" on work servers are available. 
  • CE (Computing Element)
    CE provides management of jobs, and job dispach to execution nodes.
    EMI system has CREAM (Computing Resource Execution And Management) as a CE.
    CREAM coordinates LRMS (Local Resource Management System) to dispatch jobs.
    EMI system uses LSF scheduler for LRMS.
  • WN (Worker Node)
    WN provides job execution function.
    In this system, WN is managed by LSF. 
  • LFC (Logical File Catalog)
    LFC manages storage resource of EMI system and catalog data called LFN (Logical File Name).
    By using LFN, users can access storage resources without regard to file location.
  • StoRM (STOrage Resource Manager)
    StoRM provides access to storage resources used in the EMI system.
    StoRM defines storage area and provides a function to access storage resources.
    GHI file system and GPFS Servers are used for assignment data.
  • BDII (Berkeley Database Information Index)
    BDII collects resource information of EMI environment which users can refer to.
    In EMI system, BDII top and BDII site components are provided.
    BDII site collects information of internal components, and BDII top collects that of external components. 
  • PX (my ProXy)
    PX updates a period of user proxy certificates.
  • APEL (Accounting Processor for Event Logs)
    APEL collects accounting information for jobs.
  • ARGUS (Authorization Framework)
    ARGUS provides policy-based authorization framework.
  • VOMS (Virtual Organization Membership Service)
    VOMS manages user information related VO (Virtual Organization).
    VOMS also proceeds user query for LCG sites which supports VO managed in this system.
  • CVMFS (CernVM File System)
    It provides file system for utilizing data processing software resources in high energy physics developed in research institutes.

Hostnames of components are listed below.

gLite system componentHostname
UIlogin.cc.kek.jp
ccw.cc.kek.jp
ccx.cc.kek.jp
CEkek2-ce01.cc.kek.jp
kek2-ce02.cc.kek.jp
WNcb001.cc.kek.jp ~ cb270.cc.kek.jp
cb501.cc.kek.jp ~ cb567.cc.kek.jp
LFCkek2-lfc.cc.kek.jp
kek2-lfcb1.cc.kek.jp
kek2-lfcb2.cc.kek.jp
kek2-lfc03.cc.kek.jp
StoRMkek2-se01.cc.kek.jp
kek2-se02.cc.kek.jp
kek2-se03.cc.kek.jp
BDIIkek2-bdii.cc.kek.jp (BDII_top)
kek2-sbdii.cc.kek.jp (BDII_site)
PXkek2-px.cc.kek.jp
APELkek2-apel.cc.kek.jp
ARGUSkek2-argus.cc.kek.jp
VOMSvoms.cc.kek.jp
CVMFS Stratum 0cvmfs-stratum-zero.cc.kek.jp
CVMFS Stratum 1cvmfs-stratum-one.cc.kek.jp

Requirement for using EMI system

  • Available CA (Certification Authority)

This system requires user certificate.
For Certification Authority which provides available certificates for EMI system, please refer to the following link.
http://wlcg.web.cern.ch/getting-started/certificates

KEK provides CA system called KEK GRID CA. For details, please refer to the following link.
http://gridca.kek.jp

  • Available VO's

This system requires registration to a VO this system supports.
Please refer to the following link for VO registration information.

Operation Portal (http://operations-portal.egi.eu/vo )
Attention : The above page requires your user certificate.

Available VOs of the EMI system
(EMI system manages by VOMS server)

belle
cdfj
g4med
ppj

(Other supported VOs)

calice
dteam
geant4
ilc
ops
t2k.org
kagra
vo.france-asia.org

For VO information managed by this system VOMS server, you can refer following link.
KEK VOMS Server
Attention : The above page requires your user certificate.

User Interface(UI)

Login to UI

In this system, UI component is integrated with work servers of Central Computing System.
An user account of Central Computing System is needed to use the UI.
For registration, please see the following link.
http://ccwww.kek.jp/kek/cc/oper/index.html

When you access to a work server, you should use SSH vertion2.
The hostname of work server is either "login.cc.kek.jp", "ccw.cc.kek.jp" or "ccx.cc.kek.jp".

 #> ssh -l username ccw.cc.kek.jpusername : user account

You are asked to input password. If you enter correct password, you can log in to a work server.

Set up in order to use the UI

In the home directory of the work server to create the environment variable setup file for the UI.

 #> cp /cvmfs/grid.cern.ch/etc/profile.d/setup-cvmfs-ui.sh . ↓
 #> vi setup-cvmfs-ui.sh ↓
 
Edit as in the table below.(Examples utilizing EMI systems in KEK)
export ENVIRONMENT VARIABLE NAME=VALUE
export X509_CERT_DIR=/etc/grid-security/certificates
export X509_VOMS_DIR=/etc/grid-security/vomsdir
export VOMS_USERCONF=/etc/grid-security/vomses
export MYPROXY_SERVER=kek2-px.cc.kek.jp
export LCG_GFAL_INFOSYS=kek2-bdii01.cc.kek.jp:2170,kek2-bdii02.cc.kek.jp:2170,bdii.grid.sinica.edu.tw:2170
export BDII_LIST=Comment out
export LD_LIBRARY_PATH=${base}/lib64:${base}/lib:${base}/usr/lib64:${base}/usr/lib:/usr/lib64:/usr/lib
export PERL5LIB=${base}/usr/lib64/perl5/vendor_perl:${base}/usr/lib/perl5/vendor_perl:${base}/usr/share/perl5

Add the following environment variable. This is all or VO to use.
You can determine the VO name available at <VO name> of "VO_ <VO name> _DEFAULT_SE" of the environment variable name.

export ENVIRONMENT VARIABLE NAME=VALUE
export VO_VO_FRANCE_ASIA_ORG_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_T2K_ORG_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_PPJ_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_OPS_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_KAGRA_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_ILC_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_GEANT4_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_G4MED_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_FKPPL_KISTI_RE_KR_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_DTEAM_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_CDFJ_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_CALICE_DEFAULT_SE=kek2-se01.cc.kek.jp
export VO_BELLE_DEFAULT_SE=kek2-se02.cc.kek.jp or kek2-se03.cc.kek.jp

And, add the following environment variable.

export ENVIRONMENT VARIABLE NAME=VALUE
export GLOBUS_TCP_PORT_RANGE=20000,25000

Apply the environment variable.

 #> . setup-cvmfs-ui.sh

If you add a command to the home directory in the ".bash_profile" or ".bashrc" to apply the environment variable, would apply the environment variables automatically the next login.

Example that you want to add to the ".bash_profile".
 #> vi $HOME/.bash_profile ↓
 
 Add the following to the last line, and then save the file.
 . setup-cvmfs-ui.sh ↓

Set up glite_cream.conf

For using glite-ce-job-output, you need to create this file.

~/.glite/[VO name]/glite_cream.conf or ~/.glite/glite_cream.conf

Contents of glite_cream.conf:

[
	UBERFTP_CLIENT="/cvmfs/grid.cern.ch/emi3ui-latest/usr/bin/uberftp"
]

User Authentication

Preparation

For preparation, put a certificate file to your home directory.
How to get it, see this link
( The command path is /opt/kek/caclt/bin/certreq on KEKCC work servers )

After log in to a work server, make a new directory in your home directory.

 #> mkdir $HOME/.globus ↓ 
 #> chmod 700 $HOME/.globus

Put your certificate pair (usercert.pem and userkey.pem) in .globus directory.

(Attention) usercert.pem and userkey.pem are important files for authentication.
You must not release userkey.pem for other person. You should set the correct permission to certificate files.

 #> cd $HOME/.globus ↓
 #> chmod 644 usercert.pem ↓
 #> chmod 400 userkey.pem

Proxy certificate

A proxy certificate is used for user authentication.
To obtain a proxy certificate, run the following command.

 #> voms-proxy-init -voms vonamevoname : VO name
 
 Sample) voms-proxy-init -voms ppj ↓

When you execute this command, a passphrase of certificate is required.
After you enter the correct passphrase, proxy certificates is issued.
You can now be authenticated with this proxy certificate.

Also, you can set group and role when you enter voms-proxy-init as shown below.

Setting a group

 #> voms-prox-init -voms voname:/voname/GROUP ↓

 Sample) voms-proxy-init -voms ppj:/ppj/test ↓

Setting a role

 #> voms-proxy-init -voms voname:/voname/Role=ROLE ↓
 
 Sample) voms-proxy-init -voms ppj:/ppj/Role=production

You can set both group and role.

 #> voms-proxy-init -voms voname:/voname/GROUP/Role=ROLE ↓ 
 
 Sample) voms-proxy-init -voms ppj:/ppj/test/Role=production

Confirm the proxy certificate

You can confirm the proxy information by using following command.

 #> voms-proxy-info -all

Please confirm that the attribute line includes VO name and that the value of timeleft is not 0.

$ voms-proxy-info -all
subject   : /C=JP/O=KEK/OU=CRC/CN=TEST USER/CN=proxy
issuer    : /C=JP/O=KEK/OU=CRC/CN=TEST USER
identity  : /C=JP/O=KEK/OU=CRC/CN=TEST USER
type      : full legacy globus proxy
strength  : 1024 bits
path      : /tmp/x509up_u12345
timeleft  : 12:00:00
key usage : Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
=== VO ilc extension information ===
VO        : ilc
subject   : /C=JP/O=KEK/OU=CRC/CN=TEST USER
issuer    : /C=JP/O=KEK/OU=CRC/CN=host/voms.cc.kek.jp
attribute : /ppj/Role=production/Capability=NULL
timeleft  : 12:00:00
uri       : voms.cc.kek.jp:15023

Discard proxy certificates

You can discard your proxy certificates using following command.

 #> voms-proxy-destroy

Get resource information

You need the information of resource location for submitting jobs and transferring data.
lcg-infosites command provides you the VO resource information.

 #> lcg-infosites --vo voname componentvoname : VO name
 component :target component

Available target of components is listed below.

alldlilfctag
bdii_sitedliLocallfcLocalvobox
bdii_topftsmyproxyvoms
cegridicesevoms-admin
closeSElbsitenamesvoview
creamlcg-cespacewms

Example1:List the available CEId for ppj VO

 #> lcg-infosites --vo ppj ce

Example2 : List the available SE and its vacant and used amounts for ppj VO

 #> lcg-infosites --vo ppj se

Example3:List the available LFC for ppj VO

 #> lcg-infosites --vo ppj lfc

Submitting jobs

Available queues

Job scheduler LSF is used in this system.
LSF provides multiple queues for EMI job submission.
The available queue depends on your VO and role.

The available queues for EMI system are listed in EMI Queue List pages.

How to submit a job

To submit a job, the following steps are needed.

Prepare the job file
Gather the CEId information
Submit a job
Manage a job
Get the output of job

The following section explains job submission methods.

Prepare a job file

A job for EMI is defined by a job description file written in JDL(job description language).
You need to create a JDL file for job submisson.
You should also prepare script file to execute on WN.

Below shows a sample JDL file and a script file.

Sample JDL file;

file name: test.jdl

[
Executable = "/bin/sh";  # Execution shell
Arguments = "test.sh";   # Execution script
StdOutput = "std.out";   # Standard Output file
StdError = "std.err";    # Error Output file
InputSandbox = {"test.sh"};   # Input files for job
OutputSandbox = {"std.out","std.err"};   # Output files for job
OutputSandboxBaseDestURI = "gsiftp://localhost"; # Base URI for Output files
]

This sample executes the test.sh script and gets std.out and std.err files.

Sample script;

file name : test.sh

#!/bin/sh

echo "Hello World!"
echo "I am `hostname`"

In this sample, echo command is used to output a simple information.

Gather CEId information

To submit a job, you can search for available CEId.
To search CEId, you can execute following command.

 #> lcg-infosites --vo ppj ce

The output of this command is as follows:

#   CPU    Free Total Jobs      Running Waiting ComputingElement
----------------------------------------------------------------
  3000    2823          0            0       0 kek2-ce01.cc.kek.jp:8443/cream-lsf-gridmiddle
  3000    2822          0            0       0 kek2-ce02.cc.kek.jp:8443/cream-lsf-gridmiddle

The CEId is displayed in the ComputingElement column.
The CEId of middle queue is shown as a typical CEId of the VO's CEIds.

A host has one or more CEId.
For example, kek2-ce01.cc.kek.jp of ppj VO has three CEId as follows.

  • kek2-ce01.cc.kek.jp:8443/cream-lsf-gridshort
  • kek2-ce01.cc.kek.jp:8443/cream-lsf-gridmiddle
  • kek2-ce01.cc.kek.jp:8443/cream-lsf-gridlong

These CEId's depend on what LSF queue type is available on the ppj VO.
kek2-ce01.cc.kek.jp:8443/cream-lsf-gridlong is a CEId of kek2-ce01.cc.kek.jp which uses gridlong queue.
Please choose CEId depending on the queue you want to use.

The available queues for EMI system are listed in EMI Queue List pages.

Submit a job

After searching the appropriate CEId, you can submit a job.

 #> glite-ce-job-submit -r CEId -o file_name -a jdl_fileCEId : CEId
 file_name : the file JOBID is written
 jdl_file : JDL file
 
 Sample) glite-ce-job-submit -r kek2-ce01.cc.kek.jp:8443/cream-lsf-gridmiddle -o jobid.txt -a test.jdl

If you submit a job successfully, the jobid of the job is written to a file specified by -o option.
This jobid is unique, and you can use it to monitor job status, to cancel, and to get job output.
The -r option is not mandatory. If you do not use this option, your job is submitted to any CE.
Depending on your ROLE, some CEId's of KEK are not available.

Check job status

You can check the job status by using following command.

 #> glite-ce-job-status -i jobid_file ↓

Sample) glite-ce-job-status -i jobid.txt ↓ 

This command displays jobid, job status and its reason.
Also, you can use the -d option to display the details if glite-ce-job-submit is used.
The -d option has four levels. The following example is set to 2, which shows the detail of this job.

 #> glite-ce-job-status -i jobid.txt -d 2 ↓ 

The status parameters of job are;

STATUSSubscription
IDLEPending on cream node
RUNNINGAccepting job request by CREAM
REALLY-RUNNINGJob is running on WN
DONEJob is successfully done
DONE_FAILEDJob has failed
ABORTEDJob is aborted
CANCELEDJob is canceled

Get an output of job

(*) At first, you need to Set up glite_cream.conf.

You can get an output of your job by using following command.

 #> glite-ce-job-output -i jobid_filejobid_file : jobid file
 
 Sample) : glite-ce-job-output -i jobid.txt ↓

If you use glite-ce-job-output, <CE_HOSTNAME>_PORT_JOBID directory is created on current directory
and output files of your job is stored in this directory.
glite-ce-job-output command gives an output of all jobs listed in jobid.txt (glite-ce-job-output).

Cancel a job

You can cancel your job by using following command.

 #> glite-ce-job-cancel -i jobid_file ↓
  
 Sample) : glite-ce-job-cancel -i jobid.txt ↓

You can choose which jobs in jobid.txt you want to cancel.
To confirm that the job has been canceled, run glite-ce-job-status command.
The output shows the job status 'CANCELED.'

Using LFC

Environment Settings

LFC provides metadata (replica catalogue) for files created in the EMI system, which enables internal and external file reference.
These catalog data has unique GUID and LFN. User can access the data by using an LFN.
LFN has a directory structure which is based /grid/<VO_NAME>.

To use LFC, you should set environmental variables. For KEK LFC (kek-lfc.cc.kek.jp), type the following commands.

 #> export LCG_CATALOG_TYPE=lfc ↓
 #> export LFC_HOST=lfc_host ↓ 
 
 Sample) export LCG_HOST=kek2-lfc.cc.kek.jp ↓

The hostnames of supported LFC are listed using lcg-infosites command.

Create a directory

You can create a directory by using following command.

 #> lfc-mkdir dir_namedir_name : directory name you want create
 
 Sample) lfc-mkdir /grid/ppj/testdir ↓

Confirm metadata

To list metadata, run the following command.

 #> lfc-ls pathpath : file or directory path
 
 Sample) lfc-ls /grid/ppj/testdir
  • The -l option shows the details of file and directory.
  • The -R option shows the recursive output.

LFC_HOME variable can be set for your convenience. For example, to set your main directory, type

 #> export LFC_HOME=/grid/ppj/testdir↓ 

After setting this valiables, your home will be set to $LFC_HOME.
lfc-ls command without any argument shows $LFC_HOME.

Removing directory

To remove a file or a directory, type following command.

 #> lfc-rm -r dir_name ↓ (In case directory)
 #> lfc-rm file_name ↓ (In case file)
 dir_name : directory name you want remove
 file_name : file name you want remove

 Sample) lfc-rm -r /grid/ppj/testdir

To confirm file or directory has been removed, run lfc-ls command.

Data transfer

GridFTP TURL is the dynamic assigned address.
The path of TURL could be changed without notice.
If you continue to use TURL directly, your data access is not guaranteed in the future.


Check available SE

EMI system uses storage element (SE) for data transfer.
SE enables metadata access, data upload/download and data replication to other sites.
You can also refer SE data via LFC metadata.

You can list available SE per VO with lcg-infosites command.

 #> lcg-infosites --vo vo_name sevo_name : VO name

The output shows the hostname, used space and available space of SE.

 Avail Space(kB)  Used Space(kB)  Type  SE
------------------------------------------
     79999999998               2  SRM   kek2-se01.cc.kek.jp

EMI system provides two SE's.

kek2-se01.cc.kek.jp
kek2-se02.cc.kek.jp
kek2-se03.cc.kek.jp

Each SE uses SRM protocol for data management. Actual data is stored to GHI file-system or GPFS Servers.

You can list meta data of StoRM by using following command.

 #> gfal-ls srm_URIsrm_URI : URI of meta data
 
 Sample) gfal-ls srm://kek2-se01.cc.kek.jp:8444/ppj/testfile ↓

The -l option lists the detailed information. SRM URI is an unique SRM protocol file path.
Its structure is srm://<StoRM hostname:8444>/<Vo name>/<file path>

Upload a file

You can upload files to StoRM by using following command.

 #> gfal-copy file_name srm_URI lfn_namefile_name : target file to upload
 srm_URI : URI of meta data
 lfn_name : LFN meta data file path
 
 Sample) gfal-copy file:///tmp/testfile srm://kek2-se01.cc.kek.jp:8444/ppj/testfile lfn:/grid/ppj/testfile ↓

By executing this command, testfile is uploaded to kek2-se01.cc.kek.jp.
The -v option lists the detailed information.
In fact, the testfile is transferred to Gridftp server which kek2-se01.cc.kek.jp manages via gridftp protocol.
Also, SRM URI and LFN is assigned to uploaded file.
You can upload files to kek2-se.cc.kek.jp by the same command.

In addition, you can transfer files with SRM URI, not LFN.

 #> gfal-copy file:///tmp/testfile srm://kek2-se01.cc.kek.jp:8444/ppj/testfile ↓

By executing this command, the testfile is uploaded to srm://kek2-se01.cc.kek.jp/ppj/testfile.

Download a file

You can download a file from SE by using following command.

 #> gfal-copy lfn_name file_path ↓ (Specify the LFN)
 #> gfal-copy srm_uri file_path ↓ (Specify the SRM URI)
 lfn_name : LFN meta data file path
 srm_URI : SRM URI file path
 file_path : file path to download a file
 
 Sample) gfal-copy lfn:/grid/ppj/testfile file:///tmp/download ↓
     gfal-copy srm://kek2-se01.cc.kek.jp/ppj/testfile file:///tmp/download ↓

For above example, you can download a file with LFN or SRM URI.

Replica a file

EMI system provides a function to replicate a file to other SE.
The "replica" mean that it is the same LFN file but SRM URI is different.

You can replicate a file by using following command.

 #> gfal-copy src_name dst_name lfn_namesrc_name : source file(SRM URI, gsiftp URI)
 dst_name : destination file(SRM URI, gsiftp URI)
 lfn_name : LFN meta data file path
 
 Sample) gfal-copy srm://kek2-se01.cc.kek.jp:8444/ppj/testfile srm://sample.test.org/testfile lfn:/grid/ppj/testfile ↓

By executing this command, the file /gird/ppj/testfile is replicated to kek2-se01.cc.kek.jp.

Delete a file

You can delete a file from SE by using following command.

 #> gfal-rm lfn_name ↓ (LFN Base)
 #> gfal-rm srm_URI ↓ (SRM URI Base)
 lfn_name : LFN of source file
 srm_URI : SRM URI file path

 Sample) gfal-rm lfn:/grid/ppj/testfile↓ (LFN Base)
     gfal-rm srm://kek2-se01.cc.kek.jp/ppj/testfile ↓ (SRM URI Base)

To confirm target file is deleted, type lfc-ls, gfal-ls command.

Reference from LFN to SRM

This section describes how to confirm SRM URI path frm LFN.
To comfirm it, the following command is used.

 #> gfal-xattr  lfn_name user.replicaslfn_name : LFN of source file
 
 Sample) gfal-xattr lfn:/grid/ppj/testfile user.replicas

The sample result of this command is as follow; In case StoRM

 #> gfal-xattr lfn:/grid/ppj/testfile user.replicas ↓
   srm://kek2-se01.cc.kek.jp/ppj/testdata

WebDAV Interface

You can upload a file to StoRM WebDAV by using following command.

 #> curl -T file_name WebDAV_URL --cert proxycertificate --key proxycertificate -E proxycertificate --capath ca_pathfile_name : target file to upload
 WebDAV_URL : WebDAV URL
 proxycertificate : Proxy certificate file 
 ca_path : CA certificate path

 Sample) curl -T /tmp/testfile https://kek2-se01.cc.kek.jp:8443/webdav/ppj/testfile \
       --cert /tmp/x509up_12345 \
       --key /tmp/x509up_12345 \
       -E /tmp/x509up_12345 \
       --capath /etc/grid-security/certificates ↓

By executing this command, testfile is uploaded to kek2-se01.cc.kek.jp.

You can download a file from StoRM WebDAV by using following command.

 #> curl WebDAV_URL -o file_name --cert proxycertificate --key proxycertificate -E proxycertificate --capath ca_pathWebDAV_URL : WebDAV URL
 file_name : target file to download
 proxycertificate : Proxy certificate file 
 ca_path : CA certificate path

 Sample) curl https://kek2-se01.cc.kek.jp:8443/webdav/ppj/testfile \
       -o /tmp/testfile \
       --cert /tmp/x509up_12345 \
       --key /tmp/x509up_12345 \
       -E /tmp/x509up_12345 \
       --capath /etc/grid-security/certificates ↓

By executing this command, testfile is downloaded from kek2-se01.cc.kek.jp.

Software area

EMI system provides a software area for locating software and program per VO.
Total amount of this area is about 900GB. The path of this area is as follow.

/opt/exp_soft/<VO name>

All UI and WN have the same path to access the software area.
Only lcgadmin ROLE group is allowed to write to this area.
For other account group, read and execute permission is attached.

However, large amount of data is unsuitable for this area because this is sharing software area for VO group.
If you want to locate large data, please use GPFS or GHI file system.
In case of depletion this area, you may be asked to clean files individually by operation team.


Last-modified: 2017-03-02 (木) 11:06:48 (748d)