Manual Accessing with Globus

From HLRS Dgrid
Jump to: navigation, search


This document gives a short introduction on accessing the D-Grid resources and references relevant documents for a quick guide.

Virtually all common used middleware systems in grid abstract from the batch-job-system for different hardware and software architectures to offer a uniform as possible interface to users and application developers. With Globus Toolkit 4 this vital role is done by the webservice "Grid Resources Allocation and Management" - WS-GRAM.


In contrast to classic unix based network services webservices uses HTTP as transport protocol. The communication between client and server is done as XML embedded in SOAP messages. The interface for a webservice is described by WSDL documents in machine readable form, therefore any client can use the web service platform and programming language independent. Webservices are not WWW pages and no portals!

Requesting a DFN Grid certificate

Precondition for accessing D-Grid resources is to have a grid certificate issued by the DFN or similar institution. If you don't have such a certificate you can request one at the nearest registration agency (RA). The list of RAs and the corresponding contact person can be found at The RA for HLRS can be reached by the web page .

With the printed request and your personal id card you can contact your selected RA. After you identified yourself to the RA and the RA has signed your request, you can import the issued certificate into your browser. A step by step manual can be found in the manuals section on the main page. To use the certificate also in your Unix environment you have to export the certificate in PKCS#12 format from the browser. In Mozilla Firefox this is done via the menu "Edit > Preferences > Tab Advanced > Subtab Encryption > Button View Certificates" by selecting your certificate and export it with the "Backup" button in PKCS#12 form.

If you want to extend a expired grid certificate take care that the Distinguished Name (DN) of the new certificate is identical to the old one. This means that all fields which are contained in the DN the identical information must be given. Only in this case you will have access to the resources as bevor otherwise you will be treated as a new member.

Membership in a Virtual Organisation

to use the resources of D-Grid you have to become a member of a so called virtual organization (VO) within D-Grid. Register yourself with your certificate at the proper VO for you associated project. Therefore your valid grid certificate has to be imported in your browser. After your request you will receive an email to validate the given email address. You have to validate it within 10 days, otherwise your request will be deleted automatically. You only have to click the link in this email. A step by step manual is also available in the manuals section.

After the representative you selected has accepted your request, you can use your grid certificate to access the D-Grid resources with Globus, LCG/gLite and UNICORE

If you have any problems with the registration, feel free to contact directly your VO representative.

D-Grid Access

Used Software

  • gsi-ssh Client
If you only want to log-in with gsi-ssh to the resources provided by D-Grid and you don't want to use a full grid environment you can use one of the following gsi-ssh clients:
  • GSI-SSHTerm a Java based client, also available as web applet. Select the Link below the "Run Plain GSISSHTerm (No VOMS)". A detailed usage manual for GSI-SSHTerm in D-Grid is available at the LRZ.
  • From the full Globus Toolkit 4 sourcecode package you can also compile the parts needed for gsi-ssh. You have to extract the package and execute the commands "./configure" and afterwards "make gsi-openssh" in the unzipped directory.
  • Globus WS-Core
If you only want zu submit grid jobs and transfer data with GridFTP, you can install the Globus Java Webservice Clients. The package can be downloaded from the Globus project page. After extracting the package you have to insert the directoy ws-core-4.0.6/bin to your PATH environment variable.
  • Globus full installation
To set up a full globus installation please use the provided binaries. If there is no binary for your platform available you can also cimplie the sources on your own. For this download the sourcecode package provided on the Globus website and follow the installation instruction.

Converting the PKCS#12 Files from the browser

The PKCS#12 file exported from the browser needs to be split in a private key file userkey.pem and the user certificate file usercert.pem für further usage with the globus toolkit.

   Extract the private key (userkey.pem)
   user@host:~> openssl pkcs12 -in GermanGridCert.p12 -out userkey.pem -nocerts
   Enter Import Password: ********
   MAC verified OK
   Enter PEM pass phrase: ********
   Verifying - Enter PEM pass phrase: ********
   Extract the user certificate (usercert.pem)
   user@host:~> openssl pkcs12 -in GermanGridCert.p12 -out usercert.pem -clcerts -nokeys
   Enter Import Password: ********
   MAC verified OK

Integrating in the Globus environment

The before created files containing user certificate and private key have to be moved to the folder ~./globus for the corresponding user and given proper access rights.

   user@host:~> mkdir -p ~/.globus
   user@host:~> mv usercert.pem ~/.globus/
   user@host:~> mv userkey.pem ~/.globus/
   user@host:~> chmod 700 ~/.globus
   user@host:~> chmod 600 ~/.globus/*

When using Windows place the file in a subfolder ".globus" below your user directory (i.e. "C:\Documents and Settings\{username}\.globus"). A version of OpenSSL for Windows can be downloaded from

Integrating the CA certificates

The Globus installation should contain the Chain of Trust of the EUGridPMA (European Policy Management Authority for Grid Authentication) to be able to work within D-Grid (At least the german CA root certificates must be included in order to be abele to authenticate user and server withing D-GRid). For the whole EUGridPMA you can download a certificate tarball for installation in the system wide used /etc/grid-security/certificates or alternatively the certificates can be stored in user's home directory.

   user@host:~> mkdir -p ~/.globus/certificates
   user@host:~> cd ~/.globus/certificates
   user@host:~/.globus/certificates> wget
   user@host:~/.globus/certificates> tar xvzf igtf-preinstalled-bundle-classic.tar.gz

If you use windows extract the certificates to the subfolder of your home directory called ".globus\certificates" (i.e. "C:\Documents and Settings\{username}\.globus\certificates", "C:\Benutzer\{username}\.globus\certificates", "C:\Users\{username}\.globus\certificates" or similar, depending on your Windows version - and replace {username} by the name of the user under which you are logged in).

! If you use a CA certificate collection from , this is outdated (date of the included files is July, 2nd, 2007)! Download them again from the above link.

Programs to unpack tgz-Files under windows can be downloaded as freeware from

Usage of D-Grid resources

Test the installation (Linux)

In the following we asume full globus installation to be installed but the shown commands are also usable with a Globus WS core installation and an additional installed gsi-ssh client. For authentication of the user to grid resources you first have to create a grid-proxy certificate with limited validity. These certificates are unencrypted an per default 12 hours vali after creation. After expiration of the grid-proxy certificate the user can simply create a new one.

Setting up the Glbous environment and generating a grid-proxy certificate with grid-proxy-init:

Setting up the Globus environment (this can also be included in the configuration files for your shell)

 user@host:~> export GLOBUS_LOCATION=/usr/local/globus-4.0.6
 user@host:~> source $GLOBUS_LOCATION/etc/
 Creating the grid-proxy certificate
 user@host:~> grid-proxy-init -debug
 User Cert File: /home/user/.globus/usercert.pem
 User Key File: /home/user/.globus/userkey.pem
 Trusted CA Cert Dir: /home/user/.globus/certificates
 Output File: /tmp/x509up_u1000
 Your identity: /C=DE/O=GridGermany/OU=Organisation/CN=Vorname Name
 Enter GRID pass phrase for this identity: ********
 Creating proxy .++++++++++++
 Your proxy is valid until: Wed Feb 13 23:51:48 2008
 Display the created grid-proxy certificate
 user@host:~> grid-proxy-info
 subject    : /C=DE/O=GridGermany/OU=Organisation/CN=Vorname Name/CN=1234567890
 issuer     : /C=DE/O=GridGermany/OU=Organisation/CN=Vorname Name
 identity   : /C=DE/O=GridGermany/OU=Organisation/CN=Vorname Name
 type       : Proxy draft (pre-RFC) compliant impersonation proxy
 strength   : 512 bits
 path       : /tmp/x509up_u1000
 timeleft   : 11:59:55

From now it is possible to establish an connection to the Globus frontend for the first time. In the following the access with gsi-ssh and WS-GRAM as well as file transfer with GridFTP will be explained.

Login on the globus node

Wee now explain how you can establish a SSH connection. As example we use the login on the D-Grid Cluster of HLRS. After creating the proxy certificate a connection to the login node of the cluster can be established with gsissh. gsissh is a modified version of OpenSSH that can additionally deal with x.509 certificates instead of username/password for authenticaten.

 user@host:~> grid-proxy-init
 user@host:~> gsissh -p 2222
 user@gridway:~>uname -a
 Linux gridway 2.6.18-128.7.1.el5 #1 SMP Mon Aug 24 08:12:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

The gsissh daemon listens per default on port 2222. This has to be passed to to the gsissh command with the -p 2222 option otherwise gsissh tries the default ssh port 22. Also it is important that this port has to be opened in your local firewall for outgoing connections. A list of available Frontends in D-GRid is available on the wegpage of the FZ Jülich.

globusrun-ws - a interface to different schedulers

The Globus Toolkit offers a unique interface for distribution of compute jobs which is independant from the underlying existing scheduler on the system. It maps the basic functionality of different schedulers to a web service which can be used with the globusrun-ws or a C/C++ or Java API. The scheduling webserivice is able to use the following scheduling framworks:

  • PBS compatible Scheduler (OpenPBS / PBSPro / TORQUE)
  • Condor
  • LSF
  • Sun Grid Engine (SGE)

In principle it is also possible to use any other scheduler with the API provided py the Globus Toolkit - provided that it implements the GRAM interface. The API offers on a standardised interface to each scheduler additioanl interfaces for submitting multi-jobs and interfaces for file staging before and after the job. Examples The folowing examples can only be executed with a valid globus proxy certificate. The example describes the execution of the command "hostname" on a PBS cluster node with the Globus GRAM webservice. The arguments passed to globusrun-ws are:

   * -submit: submitting a job
   * -s: Streaming of stdout und stderr to the lokal terminal
   * -Ft PBS: Submitting the job with PBS
   * -c <command>: create a job script that contains the command <command>
 Login to the Globus frontend node if there is no lokal globus installation available.
 user@host:~> gsissh -p 2222
 Last login: Thu Feb 14 13:55:29 2008 from
 Have a lot of fun...
 Abschicken des Befehls "/bin/hostname"
 dgin0005@gridway:~> globusrun-ws -submit -s -Ft PBS -c /bin/hostname
 Submitting job...Done.
 Job ID: uuid:7e72e40a-db11-11dc-bd65-00163e3f4320
 Termination time: 02/15/2008 15:28 GMT
 Current job state: Pending
 Current job state: Active
 ...setting Basic HLRS environment...
 Machinetype is Woodcrest 64bit
 node01                                                                <-- Output from the command /bin/hostname
 Current job state: CleanUp-Hold
 Current job state: CleanUp
 Current job state: Done
 Destroying job...Done.
 Cleaning up any delegated credentials...Done.

To submit the job to an other than the local Globus machine you can use globusrun-ws with the option -F <address of the job factory on the remote machine>.

 user@host:~> globusrun-ws -submit -s \
              -F \
              -c /bin/hostname
  Delegating user credentials...Done.
  Submitting job...Done.
  Job ID: uuid:93391b7a-db16-11dc-8b90-00163e3f4320
  Termination time: 02/15/2008 16:04 GMT
  Current job state: Pending
  Current job state: Active
  Current job state: CleanUp-Hold
  Current job state: CleanUp
  Current job state: Done
  Destroying job...Done.
  Cleaning up any delegated credentials...Done.

Per default the job factory is available belov the path /wsrf/services/ManagedJobFactoryService. The default port for the lgobus container is 8443. Since not all jobs only consist of a simple command line the globus Toolkit offers the possibility to describe jobs with a XML document. A detailed description of the schema for die XML job description should be available in the GT4 documentation.

The next example executes a simple echo command and writes the result in the file result in the home directory of the user on the remote Glbous node.

The job description in the file example-job.xml:

 <?xml version="1.0" encoding="UTF-8"?>

Transfering the job description file to the remote host:

 user@host:~> globusrun-ws -submit \
              -F \
              -f example-job.xml

Data transfer

To exchange data between workstation and Glbous node or between two Globus nodes GridFTP is used. In a Globus environment the user can start a file transfer with the globus-url-copy command.

 Copying from a globus node to the lokal filesystem
 user@host:~> globus-url-copy gsi file:///tmp/result
 Copying from one Globus node to another Glbous node 
 user@host:~> globus-url-copy gsi gsiftp://medigrid-srv.gwdg.defile/~/result

A data transfer can also be directly included in the job description. Therefore the tags <fileStageIn> to copy data to the Globus node before the job is running and <fileStageOut> for tranfer after finishing the job are available. The following example shows a simulation on the D-Grid resources at HLRS where the input data is stored at another location and the calculated results are also transferred to this location.

 <?xml version="1.0" encoding="UTF-8"?>

Links to further literature