CVMFS client setup for the domain of kek.jp

The CernVM File System (CVMFS) is an HTTP-based file distribution service used to provide data and software for the multi-domain jobs. The CVMFS provides configurations for the three major domains, i.e. cern.ch, opensciencegrid.org, and egi.eu by default. To get working with another domain like kek.jp other than big three domains, we need to place some configuration files in the right place.

This guide describes below how we get ready and started with a domain of kek.jp.

• CVMFS client setup for the domain of kek.jp
  • Prerequisites
  • Installation
    • Red Hat Enterprise Linux and its derivative distributions like Scientific Linux and CentOS (Installation)
    • Debian and Debian-based distributions like Ubuntu (Installation)
    • Other OSs like SLES and macOS (Installation)
  • Uninstallation
    • Red Hat Enterprise Linux and its derivative distributions like Scientific Linux and CentOS (Uninstallation)
    • Debian and Debian-based distributions like Ubuntu (Uninstallation)
    • Other OSs like SLES and macOS (Uninstallation)
  • Troubleshooting
    • Our firewall drops packets bound for the port 8000
  • Appendix: Contents of the Package
    • /etc/cvmfs/keys/kek.jp/kek.jp.pub
    • /etc/cvmfs/domain.d/kek.jp.conf
    • /etc/cvmfs/default.d/69-kek.conf

Prerequisites

Before setup for a domain of kek.jp, we all need to install the CVMFS and make sure it is working properly. If not yet installed, follow the original document at https://cvmfs.readthedocs.io/. It can be completed within a few minutes for most cases.

If the CVMFS is working correctly on the machine, we can probe one or more repositories like:

% sudo cvmfs_config probe
Probing /cvmfs/grid.cern.ch... OK

To simplify the setup procedure for an uncommon domain kek.jp, we provide the package, i.e., cvmfs-config-kek-latest.noarch.rpm and cvmfs-config-kek_latest_all.deb. Both packages contain only a few configurations files, i.e.:

• /etc/cvmfs/keys/kek.jp/kek.jp.pub
• /etc/cvmfs/domain.d/kek.jp.conf
• /etc/cvmfs/default.d/69-kek.conf

Therefore, we all need to install the CVMFS and make sure running correctly before installing this package.

The complete content of these files is at the most bottom of this page. N.B.: The content might be not the latest since the confluence wiki doesn't support to display attached files.

Installation

Red Hat Enterprise Linux and its derivative distributions like Scientific Linux and CentOS (Installation)

% sudo yum install https://kekcc.kek.jp/service/kekcc/download/cvmfs-config-kek-latest.noarch.rpm

# Just adding the repository belle.kek.jp for the environment variablele CVMFS_REPOSITORIES in the file /etc/cvmfs/default.local
# You may modify the file manually
% sudo bash -c "$(curl -fsSL https://kekcc.kek.jp/service/kekcc/download/add-repo-belle.kek.jp.sh)"

# You probe a new repository belle.kek.jp like:
% sudo cvmfs_config probe
#...snip
Probing /cvmfs/belle.kek.jp... OK

Debian and Debian-based distributions like Ubuntu (Installation)

% curl -O https://kekcc.kek.jp/service/kekcc/download/cvmfs-config-kek_latest_all.deb
% sudo dpkg --install cvmfs-config-kek_1.0-6_all.deb

# Just adding the repository belle.kek.jp for the environment variablele CVMFS_REPOSITORIES in the file /etc/cvmfs/default.local
# You may modify the file manually
% sudo bash -c "$(curl -fsSL https://kekcc.kek.jp/service/kekcc/download/add-repo-belle.kek.jp.sh)"

# You probe a new repository belle.kek.jp like:
% sudo cvmfs_config probe
#...snip
Probing /cvmfs/belle.kek.jp... OK

Other OSs like SLES and macOS (Installation)

# 1. Create a directory where the public key should be placed into.
% sudo mkdir /etc/cvmfs/keys/kek.jp
% cd /etc/cvmfs/keys/kek.jp
% sudo curl --remote-name https://kekcc.kek.jp/service/kekcc/download/kek.jp.pub
% sudo chmod a=r kek.jp.pub

# 2. Set the default stratum-1s and the location of the public keys for the domain of kek.jp
% cd /etc/cvmfs/domain.d
% sudo curl --remote-name https://kekcc.kek.jp/service/kekcc/download/kek.jp.conf
% sudo chmod a=r kek.jp.conf

# 3. Set the defaults: This is just for future evolution but not mandatory.
% cd /etc/cvmfs/default.d
% sudo curl --remote-name https://kekcc.kek.jp/service/kekcc/download/69-kek.conf
% sudo chmod u=rw,go=r 69-kek.conf

# 4. Just adding the repository belle.kek.jp for the environment variablele CVMFS_REPOSITORIES in the file /etc/cvmfs/default.local
# You may modify the file manually
% sudo bash -c "$(curl -fsSL https://kekcc.kek.jp/service/kekcc/download/add-repo-belle.kek.jp.sh)"

# 5. You probe a new repository belle.kek.jp like:
% sudo cvmfs_config probe
#...snip
Probing /cvmfs/belle.kek.jp... OK

Uninstallation

Red Hat Enterprise Linux and its derivative distributions like Scientific Linux and CentOS (Uninstallation)

To uninstall the package, you may run yum history undo with a right transaction ID. You can find the ID in the summary of the transaction history with yum history like:

% sudo yum history
Loaded plugins: security
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    18 |  <john>                  | 2020-04-27 11:54 | Install        |    1
    17 |  <john>                  | 2020-04-27 11:52 | Install        |   16
    16 |  <john>                  | 2020-04-27 11:41 | Erase          |   16 EE
#...snip

The right transaction looks like:

% sudo yum history info 18
Loaded plugins: security
Transaction ID : 18
Begin time     : Mon Apr 27 11:54:19 2020
Begin rpmdb    : 278:efafe27cfb324a4605c10274ce6933ca0be36d00
End time       :                           (0 seconds)
End rpmdb      : 279:564c69fb6933b069a9e9f46e7ed95887b9428a24
User           :  <iwai>
Return-Code    : Success
Command Line   : install cvmfs-config-kek-1.0-6.noarch.rpm
Transaction performed with:
    Installed     rpm-4.8.0-59.el6.x86_64  @sl6x/6.6
    Installed     yum-3.2.29-81.el6.noarch @sl-security/6.6
Packages Altered:
    Install cvmfs-config-kek-1.0-6.noarch @/cvmfs-config-kek-1.0-6.noarch
history info

Then, yum history undo with the transaction ID, which is 18 in this example.

% sudo yum history undo 18
Loaded plugins: security
Undoing transaction 18, from Mon Apr 27 11:54:19 2020
    Install cvmfs-config-kek-1.0-6.noarch @/cvmfs-config-kek-1.0-6.noarch
Resolving Dependencies
--> Running transaction check
---> Package cvmfs-config-kek.noarch 0:1.0-6 will be erased
--> Finished Dependency Resolution
#...snip
Removed:
  cvmfs-config-kek.noarch 0:1.0-6

Complete!

Debian and Debian-based distributions like Ubuntu (Uninstallation)

To uninstall the package, you may run apt remove cvmfs-config-kek like:

% sudo apt remove cvmfs-config-kek
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  cvmfs-config-kek
#...snip

The meta-information for the package cvmfs-config-kek is now removed from the system. However, the files still remain on the system. If you run apt list cvmfs-config-kek, it shows like:

% sudo apt list cvmfs-config-kek
Listing... Done
cvmfs-config-kek/now 1.0-6 all [residual-config]

This is because whole configuration files typically placed under the /etc leave by default behaviour in the Debian even after uninstallation. If you never refer theses files, you may run apt purge cvms-config-kek to delete files like:

% sudo apt purge cvmfs-config-kek
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  cvmfs-config-kek*
#...snip

Other OSs like SLES and macOS (Uninstallation)

# 1. Remove unnecessary files and directories any more.
% sudo rm -rf /etc/cvmfs/keys/kek.jp /etc/cvmfs/domain.d/kek.jp.conf /etc/cvmfs/default.d/69-kek.conf

# 2. Edit /etc/cvmfs/default.local and remove belle.kek.jp from the CVMFS_REPOSITORIES.
# ...or
% sudo bash -c "$(curl -fsSL https://kekcc.kek.jp/service/kekcc/download/remove-repo-belle.kek.jp.sh)"

Troubleshooting

Our firewall drops packets bound for the port 8000

Most Stratum 1 servers are serving at the port 8000, which might give some sites the operational issue. To avoid the case, most of Stratum1s accept and redirect the port 80 to the port 8000 to use as the alternative port. If you are facing this issue, you may modify the file /etc/cvmfs/domain.d/kek.jp.conf like:

CVMFS_SERVER_URL="http://cvmfs-stratum-one.cc.kek.jp/cvmfs/@fqrn@;http://grid-cvmfs-one.desy.de/cvmfs/@fqrn@;http://cvmfs.racf.bnl.gov/cvmfs/@fqrn@"

Appendix: Contents of the Package

/etc/cvmfs/keys/kek.jp/kek.jp.pub

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxgXzdibYqf9zMFVwi8L/
Ulaxb6xhOgkh3CQ92rKiDAmoWRxP3r0j3TY+2DhuOJtEUs4dTqda/mOuq48XvxSB
17RXEtM3NK0TRZCCb509Ov1auqOo+t9S7g/8i553/Q9WLqP6xiI6ppKhg+8Eli0Z
gs2+PkJ5fb89su9jT5MTfRVhjFXgjeRlk/XDvIACdQ94xx5+irYxonHZYKq7ubmx
Yy47ObYXBpgttmxD6sln5HlaQytBIszQJS44N8W7KrfvYR5xS3b2bAEOXQhl6TSw
7+rMJBtCvP0UqLpu1YdOuSr2/uz9mztWeP9H9djXqpkK7fxzQq4uHNcN7L3ATGlE
PwIDAQAB
-----END PUBLIC KEY-----

/etc/cvmfs/domain.d/kek.jp.conf

CVMFS_SERVER_URL="http://cvmfs-stratum-one.cc.kek.jp:8000/cvmfs/@fqrn@;http://grid-cvmfs-one.desy.de:8000/cvmfs/@fqrn@;http://cvmfs.racf.bnl.gov:8000/cvmfs/@fqrn@"
CVMFS_KEYS_DIR=/etc/cvmfs/keys/kek.jp
CVMFS_USE_GEOAPI=yes

/etc/cvmfs/default.d/69-kek.conf

# To override, edit /etc/cvmfs/default.local
#CVMFS_CONFIG_REPOSITORY=config-kek.kek.jp
#CVMFS_CONFIG_REPO_REQUIRED=yes
#CVMFS_USE_GEOAPI=yes