Certificate Security Protocol (CSP)

In order to make the Grid Engine internal communication more secure, Grid Engine has been enhanced by a certificate based security layer that sits on top between the GDI and comlib layer. The security layer is based on OpenSSLs libcrypto and uses RSA for secret key exchange and rc4 (keylength 128bit) for encryption. The implementation originated from a diploma thesis realised in 1996. Instead of transfering messages in clear text, the messages are encrypted with a secret key. The secret key is exchanged via a public/private key protocol. The user presents its certificate to SGE to prove its identity and receives the certificate from SGE to be sure he is communicating to the correct system. After this initial announcement phase the communication is transparently continued in encrypted form. The session is valid only for a certain period then session has to be reannounced.
Beginning with GE 60u4 the handshake mechanism for exchanging a secret key is based on the standard SSL handshake mechanism.

Below you can find the steps to setup a CSP secured system.

Installation and setup of CSP secured Grid Engine

The CSP mode is enabled in the product_mode file ($SGE_ROOT/{default | $SGE_CELL/common/product_mode) by adding the suffix -csp to sge or sgeee.

To setup the Certificate Security Protocol enhanced version of Grid Engine, the steps involved are very similar to the standard setup. Apart from the standard setup of Grid Engine the following additional steps are necessary:

Generation of the CA system keys and certificates on the master machine
This is done by calling the install procedures with the -csp flag
Distribution of the system keys and certificates to the execution and submit hosts
It is the task of the system administrator to do it in a secure way (the keys must be transmitted to the execution/submit hosts in a secure manner e.g via ssh.
Generation of User keys and certificates
This can be done automatically by the sysadmin after master installation.
Admittance of new users by the sysadmin

The security enhancement can be either added to a previously configured system or the system can be setup to support CSP security during the installation. The installation is performed as usual except that the install script is called with the additional parameter -csp.

To generate the CSP certificates and keys the following information must be supplied:

Country code State Location Organization Organizational unit CA email address	C=US ST=California L=San Francisco O=8Bits OU=Support emailAddress=admin@eightbits.com

The screen shots of an example installation outline the steps that are performed. If the system is already installed, the procedure looks very similar. Instead of using install_qmaster and install_execd, the script $SGE_ROOT/util/sgeCA/sge_ca -init is used. The screens below appear in the same manner.

First the Certificate Authority is created. The approach taken here, is to have a Grid Engine specific CA at the master host. The directory structure that contains security relevant information consists of two parts. Under $SGE_ROOT/{default | $SGE_CELL}/common/sgeCA the publicly accessible CA and daemon certificate are stored. The corresponding private keys are stored under /var/sgeCA/{sge_service | port$COMM_PORT}/{default | $SGE_CELL}/private. User keys and certificates go into /var/sgeCA/{sge_service | port$COMM_PORT}/{default | $SGE_CELL}/userkeys/$USER.

Initializing Certificate Authority (CA) for OpenSSL security framework
----------------------------------------------------------------------

Creating /scratch2/eddy/sge_sec/default/common/sgeCA
Creating /var/sgeCA/port6789/default
Creating /scratch2/eddy/sge_sec/default/common/sgeCA/certs
Creating /scratch2/eddy/sge_sec/default/common/sgeCA/crl
Creating /scratch2/eddy/sge_sec/default/common/sgeCA/newcerts
Creating /scratch2/eddy/sge_sec/default/common/sgeCA/serial
Creating /scratch2/eddy/sge_sec/default/common/sgeCA/index.txt
Creating /var/sgeCA/port6789/default/userkeys
Creating /var/sgeCA/port6789/default/private
Hit  to continue >>

After setting up the directory structure the CA specific certificate and private key are generated. We use either pseudo random data from a special file or if available /dev/random for seeding the PRNG (pseudo random number generator, see http://www.openssl.org/support/faq.html and http://www.cosy.sbg.ac.at/~andi for additional info on random numbers)

Creating CA certificate and private key
---------------------------------------

Please give some basic parameters to create the distinguished name (DN)
for the certificates.

We will ask for

   - the two letter country code
   - the state
   - the location, e.g city or your buildingcode
   - the organization (e.g. your company name)
   - the organizational unit, e.g. your department
   - the email address of the CA administrator (you!)

Hit  to continue >>

Please enter your two letter country code, e.g. >US< >> DE    
Please enter your state >> Bavaria
Please enter your location, e.g city or buildingcode >> Regensburg
Please enter the name of your organization >> Gridware
Please enter your organizational unit, e.g. your department >> Griders
Please enter the email address of the CA administrator >> admin@griders.org


You selected the following basic data for the distinguished name of
your certificates:

Country code:         C=DE
State:                ST=Bavaria
Location:             L=Regensburg
Organization:         O=Gridware
Organizational unit:  OU=Griders
CA email address:     emailAddress=admin@griders.org


Do you want to use these data (y/n) [y] >>

Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/private/rand.seed<

1513428 semi-random bytes loaded
Creating CA certificate and private key

Using configuration from /tmp/sge_ca14364.tmp
Generating a 1024 bit RSA private key
.....++++++
................++++++
writing new private key to '/var/sgeCA/port6789/default/private/cakey.pem'
-----
Hit  to continue >>

After the installation of the CA infrastructure application and user certificates and private keys are created and signed by the CA for the admin user, for the pseudo daemon user and for the user root. Currently we use the outdated uniqueIdentifier field for tieing the Unix user name to the certificate, this will change in a future implementation.

Creating Daemon certificate and key
-----------------------------------

Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/private/rand.seed<

1513428 semi-random bytes loaded
Using configuration from /tmp/sge_ca14364.tmp
Generating a 1024 bit RSA private key
...............++++++
................++++++
writing new private key to '/var/sgeCA/port6789/default/private/key.pem'
-----
Using configuration from /tmp/sge_ca14364.tmp
Check that the request matches the signature
Signature ok
The Subjects Distinguished Name is as follows
countryName           :PRINTABLE:'DE'
stateOrProvinceName   :PRINTABLE:'Bavaria'
localityName          :PRINTABLE:'Regensburg'
organizationName      :PRINTABLE:'Gridware'
organizationalUnitName:PRINTABLE:'Griders'
uniqueIdentifier      :PRINTABLE:'root'
commonName            :PRINTABLE:'SGE Daemon'
emailAddress          :IA5STRING:'none'
Certificate is to be certified until Mar  5 13:50:57 2003 GMT (365 days)

Write out database with 1 new entries
Data Base Updated
created and signed certificate for SGE daemons
Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/userkey
s/root/rand.seed<

1513428 semi-random bytes loaded
Using configuration from /tmp/sge_ca14364.tmp
Generating a 1024 bit RSA private key
............++++++
.................++++++
writing new private key to '/var/sgeCA/port6789/default/userkeys/root/key.pem'
-----
Using configuration from /tmp/sge_ca14364.tmp
Check that the request matches the signature
Signature ok
The Subjects Distinguished Name is as follows
countryName           :PRINTABLE:'DE'
stateOrProvinceName   :PRINTABLE:'Bavaria'
localityName          :PRINTABLE:'Regensburg'
organizationName      :PRINTABLE:'Gridware'
organizationalUnitName:PRINTABLE:'Griders'
uniqueIdentifier      :PRINTABLE:'root'
commonName            :PRINTABLE:'SGE install user'
emailAddress          :IA5STRING:'none'
Certificate is to be certified until Mar  5 13:50:59 2003 GMT (365 days)

Write out database with 1 new entries
Data Base Updated
created and signed certificate for user >root< in >/var/sgeCA/port6789/default/userkeys/root<
Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/userkeys/eddy/rand.seed<

1513428 semi-random bytes loaded
Using configuration from /tmp/sge_ca14364.tmp
Generating a 1024 bit RSA private key
.............++++++
.....................................................++++++
writing new private key to '/var/sgeCA/port6789/default/userkeys/eddy/key.pem'
-----
Using configuration from /tmp/sge_ca14364.tmp
Check that the request matches the signature
Signature ok
The Subjects Distinguished Name is as follows
countryName           :PRINTABLE:'DE'
stateOrProvinceName   :PRINTABLE:'Bavaria'
localityName          :PRINTABLE:'Regensburg'
organizationName      :PRINTABLE:'Gridware'
organizationalUnitName:PRINTABLE:'Griders'
uniqueIdentifier      :PRINTABLE:'eddy'
commonName            :PRINTABLE:'SGE admin user'
emailAddress          :IA5STRING:'none'
Certificate is to be certified until Mar  5 13:51:02 2003 GMT (365 days)

Write out database with 1 new entries
Data Base Updated
created and signed certificate for user >eddy< in >/var/sgeCA/port6789/default/userkeys/eddy<
Hit  to continue >>

The security related setup of sge_qmaster is now complete and the installation procedure continues as usual:

SGEEE startup script -------------------- Your system wide SGEEE startup script is installed as: "/scratch2/eddy/sge_sec/default/common/rcsge" Hit to continue >>

If the shared filesystem is not secure enough to hold the CSP security information in a place that can be accessed by the execution daemons as well, it is necessary to transfer the directory containing the daemon's private key and the random file to the execution host. Here it is installed locally before the setup of the execution daemon can be performed.

Copy the private keys to the execd host
On the master machine as user root:
# umask 077
# (cd /; tar cvpf /var/sgeCA/port6789.tar /var/sgeCA/port6789/default)
On the execution host machine as user root:
# scp /var/sgeCA/port6789.tar@<master host> /
# (cd /; tar xvpf /var/sgeCA/port6789.tar )
Verify the file permissions:
# ls -lR /var/sgeCA/port6789/
/var/sgeCA/port6789/:
total 2
drwxr-xr-x   4 eddy     other        512 Mar  6 10:52 default
/var/sgeCA/port6789/default:
total 4
drwx------   2 eddy     staff        512 Mar  6 10:53 private
drwxr-xr-x   4 eddy     staff        512 Mar  6 10:54 userkeys
/var/sgeCA/port6789/default/private:
total 8
-rw-------   1 eddy     staff        887 Mar  6 10:53 cakey.pem
-rw-------   1 eddy     staff        887 Mar  6 10:53 key.pem
-rw-------   1 eddy     staff       1024 Mar  6 10:54 rand.seed
-rw-------   1 eddy     staff        761 Mar  6 10:53 req.pem
/var/sgeCA/port6789/default/userkeys:
total 4
dr-x------   2 eddy     staff        512 Mar  6 10:54 eddy
dr-x------   2 root     staff        512 Mar  6 10:54 root
/var/sgeCA/port6789/default/userkeys/eddy:
total 16
-r--------   1 eddy     staff       3811 Mar  6 10:54 cert.pem
-r--------   1 eddy     staff        887 Mar  6 10:54 key.pem
-r--------   1 eddy     staff       2048 Mar  6 10:54 rand.seed
-r--------   1 eddy     staff        769 Mar  6 10:54 req.pem
/var/sgeCA/port6789/default/userkeys/root:
total 16
-r--------   1 root     staff       3805 Mar  6 10:54 cert.pem
-r--------   1 root     staff        887 Mar  6 10:54 key.pem
-r--------   1 root     staff       2048 Mar  6 10:53 rand.seed
-r--------   1 root     staff        769 Mar  6 10:54 req.pem

Continue with the installation of sge_execd
# cd $SGE_ROOT
# ./install_execd -csp

Generation of user keys and certificates

In order to let users use the CSP secured system, the user must have access to a user specific certificate and private key. They are generated by the administrator logged in as the user root on the master machine. To generate certificates and private keys for the user it is most convenient to create a file of the following format and to execute the commands in the next box.

Create a file containing the following information, e.g. myusers.txt

eddy:Eddy Smith:eddy@griders.org
sarah:Sarah Miller:sarah@griders.org
leo:Leo Lion:leo@griders.org

where the fields are:

Unix user:Gecos field:email address

As user root on the master machine do:

% $SGE_ROOT/util/sgeCA/sge_ca -usercert myusers.txt

% ls -l /var/sgeCA/port6789/default/userkeys
dr-x------   2 eddy  staff        512 Mar  5 16:13 eddy
dr-x------   2 sarah staff        512 Mar  5 16:13 sarah
dr-x------   2 leo   staff        512 Mar  5 16:13 leo

Every user can then install its security related files in $HOME/.sge by:

% source $SGE_ROOT/default/common/settings.csh
% $SGE_ROOT/util/sgeCA/sge_ca -copy
Certificate and private key for user eddy have been installed

For every Grid Engine installation a subdirectory for the corresponding
COMMD_PORT number is installed:

% ls -lR $HOME/.sge
/home/eddy/.sge:
total 2
drwxr-xr-x   3 eddy staff        512 Mar  5 16:20 port6789

/home/eddy/.sge/port6789:
total 2
drwxr-xr-x   4 eddy staff        512 Mar  5 16:20 default

/home/eddy/.sge/port6789/default:
total 4
drwxr-xr-x   2 eddy staff        512 Mar  5 16:20 certs
drwx------   2 eddy staff        512 Mar  5 16:20 private

/home/eddy/.sge/port6789/default/certs:
total 8
-r--r--r--   1 eddy staff       3859 Mar  5 16:20 cert.pem

/home/eddy/.sge/port6789/default/private:
total 6
-r--------   1 eddy staff        887 Mar  5 16:20 key.pem
-r--------   1 eddy staff       2048 Mar  5 16:20 rand.seed

For checking and looking at certificates the following commands might be helpful.

% setenv ARCH `$SGE_ROOT/util/arch`

Display a certificate:

% $SGE_ROOT/utilbin/$ARCH/openssl x509 -in ~/.sge/port6789/default/certs/cert.pem -text

Check issuer 

% $SGE_ROOT/utilbin/$ARCH/openssl x509 -issuer -in ~/.sge/port6789/default/certs/cert.pem -noout
issuer= /C=DE/ST=Bavaria/L=Regensburg/O=Griders Org/OU=Testsystem at port 6789/CN=SGE Certificate Authority/uniqueIdentifier=CA/uniqueIdentifier=eddy/Email=eddy@griders.org

Check subject
% $SGE_ROOT/utilbin/$ARCH/openssl x509 -subject -in ~/.sge/port6789/default/certs/cert.pem -noout
subject= /C=DE/ST=Bavaria/L=Regensburg/O=Griders Org/OU=Testsystem at port 6789/CN=eddy donetti/Email=eddy@gridders.org

Show email of cert
% $SGE_ROOT/utilbin/$ARCH/openssl x509 -email -in ~/.sge/port6789/default/certs/cert.pem -noout
eddy@griders.org

Show validity
% $SGE_ROOT/utilbin/$ARCH/openssl x509 -dates -in ~/.sge/port6789/default/certs/cert.pem -noout
notBefore=Sep 25 14:48:38 2001 GMT
notAfter=Sep 25 14:48:38 2002 GMT

Show fingerprint
% $SGE_ROOT/utilbin/$ARCH/openssl x509 -fingerprint -in ~/.sge/port6789/default/certs/cert.pem -noout
MD5 Fingerprint=F9:FA:AB:86:F3:71:E4:7F:18:82:78:7D:5E:51:7B:B5

For renewing certificates proceed as follows (this script has been introduced in GE 6.0u4 and GE 5.3p7):

Change to $SGE_ROOT and become root on the master host (we assume $SGE_CELL being 'default'):
# tcsh
# source $SGE_ROOT/default/settings.csh

edit $SGE_ROOT/util/sgeCA/renew_all_certs.csh, change the number of days the certificates are valid:
---
  # extend the validity of the CA certificate by
  set CADAYS = 365
  # extend the validity of the daemon certificate by
  set DAEMONDAYS = 365
  # extend the validity of the user certificate by
  set USERDAYS = 365
---

run the changed script (default for all extension times are 365 days from the day the script is run)

# util/sgeCA/renew_all_certs.csh

Then you have to replace the old certificates against the new ones on all hosts that installed them locally (i.e. under /var/sgeCA/..., see above under
execution daemon installation).
If users have copied certificates and keys to $HOME/.sge they have to repeat $SGE_ROOT/util/sgeCA/sge_ca -copy to have access to the renewed certificates.