Using
HPSS from the KEKCC system
Types
and Features
HPSS client
interfaces available in KEKCC are; pftp,
Kerberos-ftp, Client API, HPSS tools, HPSS-VFS (HPSS through VFS), scp. Listed below are their characteristics.
|
Interface
|
Intended
usage
|
Advantage
|
Disadvantage
|
kinit needed in advance
|
loading
of /var/hpss/kek/hpss.{sh|csh}
needed
|
|
pftp
|
interactive data transfer
|
easy, high-speed data transfer. COS
specifiable
|
limited platform
|
Yes
*1
|
Yes
*2
|
|
Kerberos-ftp
|
interactive data transfer
|
easy, high-speed data transfer, COS specifiable
|
|
Yes
*1
|
Yes
|
|
Client API
|
data transfer from user program
|
high-speed direct data transfer from user
program
|
If you have source of HPSS 4.5, please
re-write it. If you have source of HPSS 6.2.1, re-compilation and re-link is recommended,
but not mandatory.
|
Yes
*1
|
Yes
*2
|
|
HPSS tools
|
interactive data transfer
|
easy, high-speed data transfer
|
limited platform
|
Yes
*1
|
Yes
*2
|
|
HPSS-VFS
|
file operation without data transfer
(file deletion, move, list aquisition etc)
This interface is slower than above
interfaces.
|
direct file operation from NFS
|
slow data tranfer
|
No
|
No
|
|
scp
|
Interactive data transferring
|
The command scp
is user friendly
|
Slow. When you create a new HPSS file,
COS cannot be specified.
|
No
|
No
|
*1: You need to issue ¡Èkinit¡É prior to use this
interface.
Example:
$ kinit
*2: Using clientAPI/HPSS tools by batch
job requires loading of /var/hpss/kek/hpss.{sh|csh} of job-running machine.
Example:
$ . /etc/profile.d/hpss.sh
Exception: You need to load /var/hpss/kek/hpss32.sh or hpss32.csh if
you run 32-bit application on 64bit-OS.
Example:
$ source /var/hpss/kek/hpss32.csh
HPSS Size, COS,
Family ID, and FileSet
File size stored
in HPSS
The minimum file size of HPSS is 0 and
maximum file size is 160GB (or 32GB, or 128MB, or 1MB, based on the COS).
Since HPSS is designed to store large
files > 256M bytes. Please store
the files > 2GB if the performance is important. Please store smaller-size
files in GPFS.
HPSS file size and
COS
HPSS can
improve performance by storing files in different COS according to file size.
In HPSS, there are three COS's, Small, Medium,
Large, in order to high-performance file environment. For recommended COS size, see the following
table.
Please refer
to the following URL to see how to specify the appropriate COS.
http://kekcc.kek.jp/service/hpss/uguide/cos73.html (It is
written in Japanese).
Please also
refer to the following URL for more details about COS and Fileset.
http://kekcc.kek.jp/service/hpss/uguide/hpss/cos_table.html
COS Table
|
Fileset
|
FamilyID
|
Mount
Point
|
Created
Year
|
COS
|
Hier
|
disk
SC
|
tape
SC
|
tapeMedia
|
max
file size
|
Comment
|
|
ps_klea
|
1002
|
/hpss/ps/klea_g
|
2006
|
9
|
9
|
30
|
99
|
JA
|
160GB
|
COS
is specified
|
|
None (since it is symbolic link)
|
1002
|
/hpss/ps/klea_g <- /hpss/had/koto
|
2009
|
9
|
9
|
30
|
99
|
JA
|
samve as above
|
Same as above
|
|
ilc
|
2001
|
/hpss/ilc_g
|
2006
|
10,20,30
|
10,20,30
|
20¡¡30
|
29
|
JA
|
Each COS has recommended
file size.
0MB < 10 <1MB
1MB < 20 <128MB
128MB < 30 < 32GB
|
|
acc
|
4001
|
/hpss/acc_g
|
2006
|
10,20,30
|
10,20,30
|
|
ce
|
6001
|
/hpss/ce_g
|
2006
|
10,20,30
|
10,20,30
|
|
test_g
|
10001
|
/hpss/test_g
|
2006
|
10,20,30
|
10,20,30
|
|
atls
|
3001
|
/hpss/atls
|
2009
|
10,20,30
|
10,20,30
|
|
ce_lcg
|
0
|
/hpss/ce/lcg
|
2006
|
80
|
80
|
30
|
89
|
JA
|
32GB
|
COS
is specified
|
|
ce_srb
|
0
|
/hpss/ce/srb
|
2003
|
80
|
80
|
32GB
|
COS
is specified
|
|
ce_bess
|
6004
|
/hpss/ce_g/bess
|
2008
|
80
|
80
|
32GB
|
COS
is specified
|
|
had
|
5100
|
/hpss/had
|
2009
|
51
|
51
|
30
|
51
|
JB
|
160GB
|
COS is specified
|
|
t2k
|
5200
|
/hpss/t2k
/hpss/t2k/beam
/hpss/t2k/nd280
|
2009
|
52
|
52
|
30
|
52
|
JB
|
160GB
|
COS is specified
|
|
mlf
|
5300
|
/hpss/mlf
|
2009
|
53
|
53
|
30
|
53
|
JB
|
160GB
|
COS is specified
|
|
cmb
|
5400
|
/hpss/cmb
|
2009
|
54
|
54
|
30
|
54
|
JB
|
160GB
|
COS is specified
|
|
vo_ilc
|
5500
|
/hpss/vo/ilc
|
2010
|
55
|
55
|
30
|
55
|
JB
|
160GB
|
COS is specified
|
|
vo_belle
|
5600
|
/hpss/vo/belle
|
2010
|
56
|
56
|
30
|
56
|
JB
|
160GB
|
COS is specified
|
From now on, /hpss/ps/klea_g
can be accessed by symbolic link of /hpss/had/koto.
*** NOTICE ***
- HPSS files smaller than 256MB are not recommended, but if neccesary
create them in COS=10 or 20.
- The max size of the COS is different. When you try to write the file > 128MB to COS=20, it is impossible.
Please explicitly specify COS=30 when you want to create a file > 128MB.
Why COS is
required
When you write to the fileset
withoud COS, such as /hpss/acc_g,
/hpss/ilc_g, /hpss/atls, /hpss/ce_g,
/hpss/test_g, the new HPSS file is created with
default COS=20. Since its max size is 128MB, you cannot write the file >
128MB. The max size different based on the COS
- Please refer to above table to
see each COS¡Çs maz size.
o /hpss/ps/klea_g
has COS=9 and its max size is 160GB.
o /hpss/ce/lcg,
/hpss/ce/srb, /hpss/ce_g/bess
has COS=80 and its max size is 32GB.
- If your sub-group uses the filesets,
/hpss/acc_g, /hpss/ilc_g, /hpss/atls,
/hpss/ce_g, /hpss/test_g, then you need to
specify COS=30 when you want to create a file > 128MB.
Specifying COS
- Creating files by FTP
interfaces( pftp, Kerberos-ftp, Normal-ftp )
Specify a COS by subcommands such as "quote site setcos
30." In case of pftp, it is automatically
decided according to file size. For details, see User's
Guide.
Below is an example.
|
ftp> quote
site setcos 30
200 COS set to 30.
ftp> put BM.tar
200 Command Complete (952320, "BM.tar",
0, 1, 1048576, 0).
200 Command Complete.
150 Transfer starting.
226 Transfer Complete.(moved = 952320).
952320 bytes sent in 0.1122 seconds (8.098 MBytes/sec)
200 Command Complete.
ftp>
|
- Creating files by
HPSS client API
Specify a COS by the argument of hpss_Open(). See Programmer's
Reference or sample codes in /opt/hpss/kek/examples
on the work server for more information.
Below is a coding example.
|
hpss_cos_hints_t hints_in, hints_out;
hpss_cos_priorities_t hints_pri;
hints_in.COSId = cosid;
hints_pri.COSIdPriority = REQUIRED_PRIORITY;
hfd = hpss_Open(hpss_file, O_WRONLY | O_CREAT | (aflag?O_APPEND:0),
perms, &hints_in, &hints_pri,
&hints_out);
|
*** NOTICE
***
If you wish to port the HPSS source code in HPSS 4.5 into HPSS 6.2, please
refer to the following.
http://kekcc.kek.jp/service/hpss/uguide/hpss/HPSSAPI_source_diff.html
If you wish to port the HPSS source code in HPSS 6.2.1 into HPSS
6.2.2.2p, then re-compilation and re-link is recommended, but not mandatory.
If you wish to port the HPSS source code in
HPSS 6.2 into HPSS 7.3, please refer to the following.
http://kekcc.kek.jp/service/hpss/uguide/HPSS73_changes.html (It is written in Japanese.)
If you use posix-like APIs, please refer
to the following.
http://kekcc.kek.jp/service/hpss/uguide/HPSS73_warpper.html
- COS cannot be specified when
creating files by HPSS-VFS, and the COS of the new files is determined by the following rule.
o If the fileset has the COS, ( for example, /hpss/ps/klea_g has COS=9), that COS is specified.
o If the fileset does not have the COS (for example, /hpss/ce_g has nos COS), COS=30 is
specified. .
*** NOTICE
***
It is possible but not recommended to create files with COS=9 for FileSets
without COS, such as /hpss/ce_g.
If your group is assigned the fileset without the COS, please use COS=10, 20, 30 based on the
file size.
HPSS and Family ID
A FamilyID is specified to a FileSet.
All files created in the fileset have the specified FamilyID. A file with a FamilyID
will be stored in the tape of the FamilyID. Files in
the same FileSet have the same FamilyID,
and therefore will be stored in the same tape. The FileSet
and FamilyID currently used are listed here.
Fileset test_g ( Mountpoint: /hpss/test_g ) is
test area.
The files in this area will be removed when not accessed 1 month.
HPSS Tape Usage
For the
available tape cartridges numbers, Please refer to the following URL.
http://www.cc.kek.jp/service/hpss/tapeusg.html¡¡(Available
from KEK intra-net only, update every 1 hour)
Or "hquota" command provide this information at
login.cc.kek.jp ( KEKCC work-server ).
*** NOTICE ***
It
is possible to use more tape cartridges than assigned, but not recommended.
Please use the
appropriate numbers of the tape cartridges within assigned.
Tape media 3590-K, 3592-JA, 3592-JB
Please refer to
the following ¡ÈCOS and Fileset Table ¡È for available tape cartridges types for each
COS.
http://kekcc.kek.jp/service/hpss/uguide/hpss/cos_table.html
Here are the
tape cartridge specification.
|
¡¡
|
3590-K
(Not
Available)
|
3592-JA
(J1A
format)
|
3592-JA
(E06
format)
|
3592-JB
(E06
format)
|
|
Used
period in thie site
|
2001¡Á2008
|
2006¡Á
from
now on , read-only
|
2009¡Áfrom now on
|
2009¡Áfrom
now on
|
|
Uncompressed
amount¡¡[GB/volume]
|
40
|
300
|
660
|
1000
|
|
Compressed
amount¡¡[GB/vol]
|
50*
|
400*
|
No actual data
|
Same
as left
|
|
Maximum
speed[MB/sec]
|
14
|
40
|
100
|
160
|
* Experiment results at this site
3590-K Tape Cartridge
- The data in the filesets created
during 2001 - 2005 was originally stored in the 3590-K tape cartridges.
o The 3590-K¡Çs uncompressed capacity is 40GB.
- Since the old tape drive 3590 was withdrawn, the data were
migrated into the 3592-JA (J1A format) tape cartridges.
o Now its data is available in read-only mode, but can be deleted.
- The part of the 3590-K data is not migrated since the subgroup
leader told it is unnecessary.
o Please refer to the following URL for more details. http://kekcc.kek.jp/service/hpss/uguide/hpss/cos_table.html
3592-JA Tape Cartridge
- The data in the filesets created
during 2006 - 2008 was originally stored in the 3592-JA (J1A format).
o The 3592-JA (J1A format) ¡Æs uncompressed capacity is 300G.
- In the new system starting in 2009, the tape is readable and
writable.
o The 3592-JA (J1A format) is readable, but not writable.
o The tape is writable by re-formatting from 3592 JA (J1A format) into
3592-JA (E06 format).
o The 3592-JA (E06 format) ¡Æs uncompressed capacity is 660GB.
- The existing 3592-JA (J1A format, 300GB) tape cartridge with
full of data can be repacked into the new 3592-JA (E06 format, 660GB), but
it takes time.
o If you want above, please contact your sub-group leader.
3592-JB Tape Cartridge
- In
the new system starting in 2009, the brand new tape type, 3592-JB (E06
format) is available.
o The 3592-JB (E06 format) ¡Æs uncompressed capacity is 1TB
- The new COS is required to use this new tape cartridges.
o Please refer to the following URL to check if your group¡Çs fileset is using JB or JA.
o
http://kekcc.kek.jp/service/hpss/uguide/hpss/cos_table.html
*** NOTICE ***
Each sub-group has its own family
ID. Each tape also has family ID as same as before. You can check the numbers
of the tape cartridges assigned to your sub group by the following URL. Please not use more tape cartridges than
assigned. http://www.cc.kek.jp/service/hpss/tapeusg.html¡¡(Available
from KEK intra-net only)
How to use pftp and Kerberos-ftp
There are 2 kinds of ftp to access HPSS.
1.
pftp HPSS
unique command provided by HPSS package
2.
kftp Kerberos
ftp provided by OS
Note: nftp Normal ftp provided by OS - Withdrawn in HPSS 7.3
When you use nftp, your password is transferred as plain text via
network. It is too dangerous. Please use pftp or kftp because of the security reason. The support of
normal-ftp of HPSS is withdrawn in HPSS 7.3..
HPSS pftp
Please use the alias ¡Èpftp¡É
or command ¡È/opt/hpss/kek/bin/pftp_client hco01.cc.kek.jp 4021¡É. The alias is defined in the /etc/profile.d/hpss.{sh|csh}
#> pftp
[Enter]
¡ÎExample¡Ï
#> pftp [Enter]
Connected to hco01.cc.kek.jp.
220 hco01 FTP server (HPSS 6.2 PFTPD V1.0.0 Thu Sep 16 09:24:41 MST 2004)
ready.
334 Using authentication type GSSAPI; ADAT must follow
GSSAPI accepted as authentication type
GSSAPI authentication succeeded
Preauthenticated FTP to hco01.cc.kek.jp as ibm-yoko:
232 GSSAPI user test_usr@CC.KEK.JP is authorized as test_usr@CC.KEK.JP
230-No directory! Logging in with home=/
230 User test_usr@CC.KEK.JP logged in as test_usr@CC.KEK.JP
Remote system type is UNIX.
Using binary mode to transfer files.
Pwidth set to default(1).
Multinode is Disabled.
ftp>
Kerberos-ftp
The 2nd argument of kftp in HPSS
6.2 was 4020, but it was changed into 6021.
HPSS 6.2.2.2p:¡¡/usr/Kerberos/bin/ftp hco01.cc.kek.jp 4020
HPSS 7.3 /usr/Kerberos/bin/ftp hco01.cc.kek.jp 6021
Please use the alias ¡Èkftp¡É
or command ¡È/usr/kerberos/bin/ftp hco01.cc.kek.jp 6021¡É. The alias is defined in the latest /etc/profile.d/hpss.{sh|csh} , which
is different from the previous HPSS 6.2.1. Please download the latest HPSS conf
file or rewrite your alias. Please replace ¡È/usr/Kerberos/bin/ftp¡É
with appropriate Kerberos-ftp command based on you OS.
#> /usr/kerberos/bin/ftp
hco01.cc.kek.jp 6021 [Enter]
*Note* Please use appropriate Kerberos ftp in your OS.
¡ÎExample¡Ï
#> /usr/kerberos/ftp hco01.cc.kek.jp 6021 [Enter]
Connected to hco01.cc.kek.jp.
220 hco01 FTP server (HPSS 6.2 PFTPD V1.0.0 Thu Sep 16 09:24:41 MST 2004)
ready.
334 Using authentication type GSSAPI; ADAT must follow
GSSAPI accepted as authentication type
GSSAPI authentication succeeded
Name (hco01.cc.kek.jp:ibm-yoko):
[Enter]
ftp>
When you use nftp, your password is transferred as plain text via
network. It is too dangerous. Please use pftp or kftp because of the security reason.
ClientAPI
The purpose of the Client API is to provide an interface which
mirrors the POSIX.1 specification where possible to provide ease of use to the
POSIX application programmer. In addition, extensions to allow the programmer
to take advantage of the specific features provided by HPSS are
provided.
Please refer to the following URL for the spec of the APIs.
http://kekcc.kek.jp/service/hpss/uguide/progref73.pdf
How to Compile and
Link
- Please refer to /opt/hpss/kek/examples/Makefile to see how to compile and link.
- Please refer to the above URL for the spec of the APIs.
- Please refer to
/opt/hpss/kek/examples/hpssls.c ans the sample program.
- Please refer the following URL if you wish more performance.
o
http://kekcc.kek.jp/service/hpss/uguide/HPSS73_warpper.html
- The 32bit libraries of HPSS client API is available on 64bit
OS.
o The spec of the APIs is the same.
o But environment variable and path for the libraries is different.
o Please refer to the following URL for mode details.
o
http://kekcc.kek.jp/service/hpss/uguide_en/hpss73e_32bit_api.html
HPSS tools
HPSS tools are available to ls, mkdir, rmdir
, symlink, cat and put HPSS files. It is implemented
by HPSS client APIs. The path to
these tools ¡Ê /opt/hpss/kek/examples ¡Ë is defined by
/opt/etc/profile.d/hpss.{sh|csh}.
hpssls
It lists the HPSS files, just like UNIX ls command.
#> hpssls
¡ÉHPSS Directory Name or File Name¡É
[Enter]
#> hpssls
¡ÉIf the diretory is the HPSS mount point, please put ¡Æ/.¡Ç
after the directory name¡É [Enter]
Example¡§hpssls -la /hpss/ps/klea_g/.
(Above /hpss/ps/klea_g
is the HPSS Mount Point name so that you need to put ¡Æ/.¡Ç After the directory
name.)
hpsscat
It shows the file contens,
just like UNIX cat command. (Only one argument can be specified.)
#> hpsscat
¡ÉHPSS filename¡É [Enter]
hpssput
It can copy UNIX file stdin
to HPSS file. When you want to read from stdin,
please specify ¡È-¡È as second argument.
#> hpssput
¡¡-c ¡ÈCOSname¡É ¡Édst_HPSSfile_name¡É¡¡¡Î¡Ésrc_UNIXfile_name¡É¡Ï
[Enter]
hpsscp
It copies a HPSS flie
to other HPSS file, just like UNIX cp command.
#> hpsscp
-cosid ¡ÈCOSname¡É ¡Ésrc_HPSSfile_name¡É
¡È dst_HPSSfile_name¡É [Enter]
hpsslevel
It shows the nums
of the bytes in disk and tape so that you can see if it is migrated, purged or staged.¡ÊOnly
one argument can be specified¡Ë
#> hpsslevel
¡ÉHPSSfile_name¡É
[Enter]
hpssstage
Since HPSS is hierarchical storage system, sometimes the
file is in tape only, and does not exist on disk. When the file exists in tape
only, accessing of that file takes some time. You can stage the file (make the
file exist on the disk) by HPSS tool. For examples, if you plan to use lots of
the files in tape tomorrow, you can make them to exist on the disk today, then
tomorrow¡Çs access is going to be very fast.
#> hpssstage
¡ÉHPSSfile_name¡Êmultiple files can be specified¡Ë¡É
[Enter]
Hints and Tips of
the HPSS tools
This is the example of the HPSS tools
combination.
You can archive the UNIX files and save it
to the HPSS file.
#> tar -cvf
- ¡ÉUNIXpath¡É | hpssput ¡ÉHPSSfile_name¡É [Enter]
The archived HPSS file is extracted
into UNIX file area.
#> hpsscat
¡ÉHPSSfile_name¡É | tar -xvf
- [Enter]
HPSS-VFS
¡ÊVFS interface¡Ë
HPSS-VFS¡ÊVFSinterface¡Ë
is available for filesystem-like access to HPSS.
HPSS-VFS is available on the hosts of hpss.cc.kek.jp. The same userid and password is available with login.cc.kek.jp.
Please refer to the following URL for your
sub-groups¡Ç fileset¡Çs mount point .
http://kekcc.kek.jp/service/hpss/uguide/hpss/cos_table.html
This VFS interface is slower than the other
interface, such as pftp or client API.
Using HPSS from your Linux host connected to KEK LAN
You can access HPSS from your Linux host
connected to KEK-LAN/JPARC- LAN /HANDAI-LAN.
The OS in the following URL is tested with this HPSS 6.2.2.2p.
http://kekcc.kek.jp/service/hpss/uguide/hpss73_reg_test_p.html
You can use the following access interface
to HPSS.
|
Interface
|
Client OS
|
|
Linux
|
|
pftp
|
OK
*1
|
|
ClientAPI
|
OK
*1
|
|
HPSS¥Ä¡¼Ž¥¡¦/span>
|
OK
*1
|
|
Kerberos ftp
|
OK
*2
|
|
Normal ftp
|
NG
*3
|
|
scp
|
OK
*2
|
Note:*1: Please intall
the HPSS package and install it on your Linux host.
Note:*2: Please use your OS¡Çs
command.
Note:*3 Normal ftp withdrawn in
HPSS 7.3.
How to set up your Linux
Host in order to access HPSS
[ Prerequisite ]
o Your linux host can refer to KEK DNS and
it can resolve the name of hpss.cc.kek.jp , or cw103.cc.kek.jp, cw104.cc.kek.jp.
o The scp of your Linux is installed.
o Your linux host can refer to KEK DNS and
it can resolve the name of aua01.cc.ke.jp
and aua02.cc.kek.jp.
o Your linux host can communicate with the
Kerberos server aua01.cc.kek.jp
o The command ¡Èkinit username¡É on your Linux
host completed successfully.
o The Kerberos –ftp (Example¡§/usr/Kerberos/bin/ftp and so on) in installed on you r Linux
host.
- HPSS client AP,
HPSS tool or HPSS pftp
o Your linux host can refer to KEK DNS and
it can resolve the name of
hco01.cc.kek.jp , hms01.cc.kek.jp ¡Áhms05.cc.kek.jp and aua01.cc.ke.jp
and aua02.cc.kek.
o Your linux host can communicate with the
Kerberos server aua01.cc.kek.jp
o The command ¡Èkinit username¡É on your Linux
host completed successfully.
o Your Linux OS/level is written in the following URL. (Otherwise, please
use it on your own risk.)
o
http://kekcc.kek.jp/service/hpss/uguide/hpss73_reg_test_p.html
o Your linux
host can be resolved by KEK DNS and name resolution on your Linux host is the
same.
- Please register
your Linux hostname to the KEK DNS.
- Let's assume that
your linux host name is myhost
and IP address is 10.xx.yy.zz and your host has the following /etc/hosts.
127.0.0.1 myhost localhost.localdomain
localhost
10.xx.yy.zz myhost
- Above /etc/hosts is
not good to use HPSS. With this /etc/hosts, the reverse lookup of "myhost" is 127.0.0.1, which is different from the DNS
reverse lookup of "myhost" 10.xx.yy.zz.
Please delete "myhost" in the line of
127.0.0.1.
o HPSS installation
package is installed in your Linux host.(
You can use scp or Kerberos-ftp without
installing HPSS.)
o Your linux host can
resolve the short name of hco01
(short-name). For example, your linux
host has the the following line in the /etc/hosts.
- 202.13.197.60
hco01.cc.kek.jp hco01
Instlation (Installing of client API,
HPSS tool or HPSS pftp)
- Plesae
download the /etc/krb5.conf file from the following URL. If you already
have the /etc/krb5.conf, please take backup before download/install it.
o
http://kekcc.kek.jp/service/hpss/uguide/hpss/krb5.conf
- Please download the HPSS installation package in the following
URL.
o
http://kekcc.kek.jp/service/hpss/uguide/hpss73_reg_test_p.html
o First, extract the conf file, then extract the binary file.
- Please refer to the following URL for more details.
o
http://kekcc.kek.jp/service/hpss/uguide/hpss73_end_user_terminal_setup.html
Set Environment
Variables and kinit before using HPSS
Before using HPSS, you need to issue kint and set some HPSS environment variables appropriately.
- KerberosTGT
Please obtain Kerberos¤ÎTGT by
the following command.
#> kinit ¡Éusername¡É [Enter]
- Please set up the HPSS environment variables.
o For ksh,bash users: Please read ¡É/var/hpss/kek/hpss.sh¡É
o For csh, tcsh
users: Please source ¡È/var/hpss/kek/hpss.csh¡É
o ¡ÉHPSS_API_HOSTNAME¡É environment variable
¡ÉHPSS_API_HOSTNAME¡É environment variable is set to the hostname. If your linux hostname
is the same with the network interface name with which you communicate with
hco01, it is OK. Otherwise, please reset it to the appropriate network
interface name or IP-address with which you use to communicate with hco01.
¡ÎExample¡Ï
Let¡Çs assume that userid test_usr
uses bash.
#> kinit test_usr
[Enter]
Password for test_usr@cc.kek.jp:password [Enter]
#> . /var/hpss/kek/hpss.sh [Enter]
Hints
in using Hierarchical Storage System
Concept
of HPSS – Migration, Purge, Stage
Migration
Once
a new HPSS file is created, it is usually written onto disk space. After a while,
it is also written onto tape cartridges. The file transferring from the disk to
the tape is called ¡Èmigration¡É in HPSS.
Purge
If
the file is not used for a while, then the file on the disk is automatically
erased and only the file on the tape exist. This deletion of the file on the
disk is called ¡Èpurge¡Éin HPSS.
Stage
When
the purged file is accessed by the end users, then HPSS automatically retrieve
the file from the tape and copy it onto the disk. This file transferring from
the tape to the disk is called ¡Èstage¡É in HPSS.
HPSS
Staging takes some time.
When
the purged file is accessed by the end users, then HPSS automatically retrieve
the file and mount the tape cartridge on the tape drive, then read the file
contents and copy it onto the disk area. As you can imagine, it takes some
time.
For
example, there are 10 tape drives in the HPSS system. If all of the tape drives
are used, then the next ¡Èstage¡É job might have to wait for long time.
I
appreciate your understanding.
The
access of HPSS client API, HPSS tools and HPSS pftp
is rather fast compared with the slower interface, scp
or VFS. These kinds of slower access usually requires more time to stage.
In
addition, read-only fileset access is slower than the
r/w filesets.
The
HPSS tool hpsscat is
faster than VFS
The
HPSS tool hpsscat is faster than HPSS VFS, please use
hpsscat when the performance is important.
Example¡§
If
you use the following command in VFS, it is slow.
tar
–xvf hpssfile.tar
Please
use the following command instead of above. It is faster.
hpsscat hpssfile.tar | tar –xvf -
Staging before Using HPSS Files makes your access faster.
If you plan to use lots of the
files in tape tomorrow, you can stage them today, then tomorrow¡Çs access is
going to be very fast.
#>
hpssstage ¡ÉHPSSfile_name¡Êmultiple
files can be specified¡Ë¡É
HPSS
FAQ
Please
refer to the following FAQ for hints and tips.
http://kekcc.kek.jp/service/hpss/uguide/hpss/HPSS_FAQ.html
(It
is written in Japanese. )