1 of 14

Federated GridPP XRootD

a unified GridPP storage space

​

Marcus Ebert

marcus.ebert@ed.ac.uk

(ebert@slac.stanford.edu)

​

1

  • Setup possibilities

​

  • Current config

​

  • To Do and management possibilities

2 of 14

2

Client

​

Server

xrootd

​

Local XRootD

​

3 of 14

3

Client

​

Server 1

xrootd

cmsd

​

​

Locally Federated XRootD

​

Server 2

xrootd

cmsd

​

Server 3

xrootd

cmsd

​

Redirector

xrootd

cmsd

​

4 of 14

4

Client

​

Server 1 Site 1

xrootd

cmsd

​

​

Federated XRootD

​

Server 2 Site 1

xrootd

cmsd

​

Server 1 Site 2

xrootd

cmsd

​

Redirector

xrootd

cmsd

​

5 of 14

5

Client

​

Server 1

xrootd

cmsd

​

​

Local and Federated XRootD

​

Server 2

xrootd

cmsd

​

Redirector Site 1

xrootd

cmsd

​

Metamanager

xrootd

cmsd

​

Server 1

xrootd

cmsd

​

​

Server 2

xrootd

cmsd

​

Redirector Site 2

xrootd

cmsd

​

6 of 14

6

Client

​

Server 1

xrootd

cmsd

​

​

Local and Federated XRootD

​

Server 2

xrootd

cmsd

​

Redirector Site 1

xrootd

cmsd

​

Metamanager

xrootd

cmsd

​

Server 1

xrootd

cmsd

​

​

Server 2

xrootd

cmsd

​

Redirector Site 2

xrootd

cmsd

​

7 of 14

7

Client

​

Server 1 Site 1

xrootd

cmsd

​

​

Mixed Federated XRootD

Server 2 Site 1

xrootd

cmsd

​

Redirector Site 1

xrootd

cmsd

​

Metamanager

xrootd

cmsd

​

Server 1 Site 2

xrootd

cmsd

​

​

Server 2 Site 2

xrootd

cmsd

​

Redirector Site 2

xrootd

cmsd

​

Server 1 Site 3

xrootd

cmsd

​

Server 1 Site 4

xrootd

cmsd

​

8 of 14

8

XRootD

​

​

  • clients can talk with any machine directly that have a xrootd process running
    • of course only local files can be accessed
    • could be useful for local groups

​

  • cmsd needed for communications between hierarchy levels

​

  • redirector and metamanager machines are very lightweight
    • can run in VM
    • only to find out which server has a requested file and redirect client to that server

​

  • actual file transfer is always happening directly between server and client
    • in case of firewalls, a proxy server can be used
      • gateway between local xrootd installation and outside world

​

  • one common config file can be used for all machines within the cluster
    • makes site install and on-site administration very easy and fast

​

  • through plugins other protocols, like http, supported - no need for a full apache install!

  • xrdfs can provide a POSIX file system over the whole cluster on any client machine

9 of 14

9

XRootD current install

​

​

  • federated xrootd from slide 4
    • redirector at ECDF
    • 1 server at ECDF
    • 1 server at Durham
    • 1 server at Glasgow

​

  • config in xrootd-clustered.cfg
    • if ... else statements used to separate server/redirector/metamanger parts
      • one config file for all machines

​

  • auth_file is used to allow specific groups specific access to different parts

  • mapping to users also possible
    • could be used to restrict write access to AtlasDataDisk to production role

​

​

​

10 of 14

10

XRootD install process

​

​

  • starting from minimal OS install
    • I used SL6
    • best not to use a machine previously setup for DPM

  • install and setup needed packages for general Grid infrastructure
    • edg-mkgridmap
    • ca_*
    • the new gridpp-voms packages
    • hostcert.pem, hostkey.pem in /etc/grid-security
    • needed local groups/users

​

  • install xroot packages
    • xrootd
    • vomsxrd

  • copy hostcert.pem & hostkey.pem to /etc/grid-security/xrd/xrdcert.pem (xrdkey.pem)

  • follow provided instructions
    • basically copy all config files and scripts out of xrootd to your machine and start xrootd

​

​

11 of 14

11

install instructions

​

​

​

​

​

To get the most recent version of the config file, scripts, and auth_file:

​

​

Have a look to the GridPP wiki ---> here

12 of 14

12

To do and other things

​

  • space reporting needed
    • whatever VOs need (json file for ATLAS)

​

  • test usage as storage endpoint for VOs
    • asked Atlas about it, waiting to get it enabled

  • think about if/how frm.xfr process could be useful
    • if a file requested is not in the cluster, error is usually reported
    • but there is also the possibility to copy such file into the cluster from somewhere else
      • that’s frm.xfr
      • line in xrootd-clustered.cfg: frm.xfr.copycmd in /path/to/copyscript.sh $SRC $DST
      • configured on servers
      • see: http://xrootd.org/doc/prod/frm_config.htm

  • think about a general management structure...

13 of 14

13

Possible management structure

​

  • all.export /etc readonly
    • exports the xrootd /etc from within xrootd as readonly for xrootd access
    • that’s NOT Linux /etc

​

  • /etc holds all config files and scripts

  • one script runs as xrootd cronjob: tests if all services are running and restarts if needed

​

  • one script runs in background for xrootd: tests in intervals if there are new config files or scripts available, downloads it, and restarts services

​

  • start/end scripts provided to be used by user xrootd

​

local admin effort once it is running: maintain general OS and hardware

⇒ perfect for sites with very limited manpower for storage administration

but usable on any site

​

  • think about additional tasks that could be done...

14 of 14

14

Other possible management tasks

​

  • cronjob on all servers to provide a list of all local files
    • using find under /gridstorage (oss.localroot)
    • copies list file under /gridstorage/etc/$HOSTNAME-file.lst
      • only writeable on local machine for local files
      • readonly through xrootd

​

  • on a dedicated machine read all the list files and decide what to do
    • statistics how often same file is in the cluster
    • duplicate files with only 1 copy in the cluster
    • duplicate files with large access rate
    • ....
    • (any machine that is allowed to connect to the cluster could do management tasks through cronjobs)

​

​

​

​

​